12 things production taught me while building SpotBook
Twelve field notes from production work at SpotBook and collaborative work on EPFO: the failures, engineering decisions, and people each decision was meant to protect.
Production engineering is easiest to explain through the problem it had to protect people from. These are twelve field notes from SpotBook and collaborative work on EPFO. Each starts with an operational failure, follows the engineering decision, and ends with what changed for the person using the system.
1. The Vanishing Cart
Two salespeople booked the last piece simultaneously. One order vanished entirely.
Root cause
There was no concurrency control on stock checks. Both transactions read one available piece, both tried to decrement it, and the second hit a constraint violation that rolled back the whole multi-item order.
The fix
I used SELECT ... FOR UPDATE for pessimistic locking and version-based optimistic locking on Product as a second layer. When one item in a draft fails, the rest of the draft persists.
Result: no further order-loss incidents after Season 2, and staff could trust the app again.
2. Identity-based diffing for order edits
Editing a 60-item order originally meant replacing the whole thing. That was dangerous and slow.
The first implementation deleted every item and re-inserted them. That broke stock accounting and became fragile under concurrent edits.
The frontend now computes additions, removals, and quantity changes by product ID. The backend validates only the stock impact of deltaQuantity = new - old, rather than revalidating and replacing the entire order.
Result: edits became atomic, safe, and fast without phantom stock leaks.
3. The composite loading gate
React pages flickered between loading, empty, and loaded. Users assumed their data was missing.
RTK Query’s isLoading only covers the first fetch. I created a reusable LoadingGate around isUninitialized || (isFetching && !data) and used skeletons rather than spinners.
Result: the flash of empty state disappeared, and staff stopped refreshing pages because they thought the app had broken.
4. The iPhone HEIC image problem
Product photos from iPhones did not display reliably, leaving a large share of uploads broken.
The upload flow first attempts native image decoding for JPEG, PNG, and WebP. If that fails, it converts HEIC to JPEG on the client, compresses it to 80 percent quality, and limits the image to 1200px.
Result: dependable image uploads across devices, smaller files, and no broken thumbnails.
5. DNS debugging on the expo floor
The app worked on Wi-Fi and failed on mobile data two hours before the expo opened.
The custom-domain records had not propagated consistently and SSL was misconfigured between the bare domain and www. I moved to a verified domain with preconfigured SSL, added A and CNAME records, tested multiple resolvers, and lowered TTL for the cutover.
Result: the app was reachable from any network before the doors opened.
6. WhatsApp receipt automation
Distributors wanted an instant receipt after every order, but manual sending did not scale across busy sales teams.
A confirmed order triggers a formatted WhatsApp Business message containing the order number, items, quantities, and total, sent to the distributor’s registered number.
Result: instant confirmation without manual work, and a digital proof that users could trust.
7. Multi-tenant data isolation
Moving from a single-client deployment to multiple manufacturers meant every query, API call, and report had to respect the authenticated tenant.
Tenant context is extracted from the JWT in a Spring Security filter. Hibernate filters scope queries to the tenant, and integration tests deliberately attempt cross-tenant access. The backoffice bypass is limited to the explicit Zone 0 boundary.
Result: tenant isolation is a deliberate system rule, not an assumption every developer has to remember.
8. Size-matrix complexity
A single jeans style can have eight sizes and four washes. Salespeople browse products, while the warehouse needs to count inventory at the variant level.
The model separates catalog-level Products from inventory-level ProductVariants. Orders reference variants, the catalog exposes a size and wash matrix, stock checks remain variant-specific, and prepacks combine several variants into one orderable unit.
Result: salespeople can browse naturally while the warehouse keeps the precision it needs.
9. PWA offline resilience
Expo venues have patchy connectivity. A blank screen or lost cart during an order is a lost sale.
The app uses service-worker caching for static assets, persists cart state locally, retries requests with exponential backoff, shows online and offline status, and queues critical actions for retry when a connection returns.
Result: brief drops did not mean lost carts or a broken workflow.
10. The marketing failure
This was not a code problem. Lead-generation ads produced form submissions from people without an understood need, budget, or buying intent.
The lesson was to separate the product problem from the distribution problem. For high-touch B2B software in a traditional industry, relationship-based sales, expo visits, referrals, and word of mouth are a better fit than optimizing for cheap form fills.
Lesson: engineering solves a product problem; distribution solves a business problem. They need different skills.
11. Aadhaar eSign at government scale
At EPFO, claim processing needed legally valid signatures. Physical verification introduced a long manual bottleneck.
The claim pipeline integrated Aadhaar-based eSign with retry handling for OTP failure and service downtime, while respecting the security controls required for government financial data.
Result: advance claims could move through without physical-signature dependency for the majority of cases.
12. API performance at scale
When a dashboard query slows down at government scale, the consequence is not a spinner. It is a helpline lighting up.
The work focused on schema and index design for claims, audit logs, and transaction history; composite indexes for high-cardinality queries; and restructuring heavy joins into indexed lookup paths without weakening immutable audit trails.
Result: the system protected the person waiting on the other side of a query, not just a dashboard metric.
The common thread is simple: the difficult parts should live underneath the interface. The real job is not to show users how complicated the system is. It is to make the next correct action feel possible.