Monsoon Ops: How a Mumbai D2C Brand Survived 3 Vendor Outages in August — Zero Lost Orders

In one week of August, a Mumbai D2C skincare brand lost its payment gateway for 40 minutes, its shipping aggregator for most of an afternoon, and saw the GST portal crawl on filing day. Three vendor outages, peak monsoon, 2,200 orders a day. They lost zero orders. Not because their vendors were reliable — they were not — but because the order pipeline was built to assume vendors fail. This is how the idempotent retries, the COD-fallback queue, and the GSTR-3B extension flow held the line. ## The Answer in 60 Words Three patterns saved the week. Every external call carries an idempotency key, so a retry after a timeout never double-charges or double-ships. When the payment gateway died, new orders dropped into a COD-fallback queue instead of erroring at checkout. And when the GST portal choked on filing day, a documented extension-request flow gave the finance team cover. Nothing was lost. ## Why This Matters Now Monsoon is India's stress test for ops. Power dips, flaky connectivity, and vendors under load all spike between June and September. And the failure modes are real: the GST portal itself buckled on filing day in January 2025, forcing [CBIC to extend GSTR-1 and GSTR-3B deadlines after technical glitches](https://www.businesstoday.in/personal-finance/tax/story/gst-return-filing-cbic-extends-gst-return-deadlines-for-gstr-1-gstr-3b-over-gst-portal-glitches-check-details-460387-2025-01-10). CBIC again extended the September 2025 GSTR-3B due date to 25 October via [Notification 17/2025 dated 18.10.2025](https://www.taxscan.in/top-stories/cbic-extends-due-date-for-filing-gstr-3b-returns-for-september-2025-know-the-new-deadline-1435308). If your ops pipeline assumes every vendor is always up, August will find the assumption and break it. ## The Client (Specific Details) - Sector: D2C skincare and personal care (own brand, sells on website + marketplaces) - Location: Mumbai, Maharashtra (warehouse in Bhiwandi) - Size: ~70 staff, ₹40 cr annual GMV, 2,200 orders/day, spiking to 5,000 on sale days - Stack on day 0: Shopify storefront, a Node.js middleware layer for order routing, Razorpay for payments, a shipping aggregator for couriers, Tally for accounting - The trigger: A 2024 monsoon outage where a 22-minute gateway blip created 140 double-charged customers — because a retry fired a second charge. The refunds and the support load were the wake-up call. - The mandate: Make the pipeline survive any single vendor going down for an hour without a customer noticing. ## The Architecture (How We Made It Outage-Proof) The middleware is Node.js with PostgreSQL and a Redis-backed queue, deployed on AWS. The design rule was simple: no external call happens inside the request that the customer is waiting on. Checkout writes the order and returns; everything else — charge capture, courier booking, invoice push to Tally — runs as retryable background jobs. We use the same retry-and-recover discipline in our Razorpay failed-payment recovery workflow, which claws back roughly 12% of dropped payments. ## What You'll Need (Prerequisites) Before you make a pipeline outage-proof, you need the right foundations. We audited each of these against the client's stack in the first week. ## Why Monsoon Breaks Pipelines Specifically Monsoon failures are not random — they cluster, and they cluster at the worst time. Three things stack up. Power dips and brownouts in industrial areas like Bhiwandi knock out warehouse connectivity mid-dispatch. Vendor infrastructure runs hotter because everyone's volume spikes during festive-adjacent monsoon sales. And the GST filing calendar lands monthly returns squarely in the June-to-September window, when the portal is already under load. So a brand that sails through eleven months can still have one August week where the gateway, the courier API, and the tax portal all wobble inside seven days. That is exactly what happened here. The pipeline didn't need to handle a once-a-year disaster; it needed to handle three ordinary vendor wobbles landing on top of each other. ## Idempotent Retries (The Pattern That Prevents Double-Charges) An idempotency key is a unique token you send with a request so the vendor recognises a retry and returns the original result instead of doing the work twice. Razorpay, Stripe, and most serious payment APIs honour one. The trap is generating it wrong. | Approach | What happens on retry | Verdict | |---|---|---| | No key | Second charge fires. Customer double-charged. | The 2024 incident. Never again. | | Random key per attempt | Each retry looks new — still double-charges | Worse than nothing; feels safe, isn't | | Timestamp-based key | Two fast retries can collide or diverge | Fragile under concurrency | | Deterministic key (order ID + action) | Vendor returns the original result; no duplicate | What we ship | The key for capturing payment on order SK-48213 is always the same string — something like capture:SK-48213 — no matter how many times the job retries. Razorpay sees the repeat and returns the first charge's result. One order, one charge, however flaky the network. There's a subtlety with multi-step actions. A single order might capture a payment, then book a courier, then push an invoice — three separate vendor calls. Each needs its own key scoped to the action, not one shared key for the whole order. So we use capture:SK-48213, ship:SK-48213, and invoice:SK-48213 as three distinct keys. A retry on the shipping step must not collide with the payment step. Get this wrong and a courier-booking retry can silently re-trigger a charge — the exact failure we were trying to kill. We also cap the retry curve. Each job retries with exponential backoff — 2s, 8s, 32s, two minutes, ten minutes — and after six attempts it lands in a dead-letter queue that pages a human. An uncapped retry loop is its own outage: it hammers a recovering vendor and stops it from recovering. The dead-letter queue ran exactly four jobs across the whole August week, all resolved by hand within the hour. ## The COD-Fallback Queue (Capturing the Sale When Payments Die) When Razorpay went down for 40 minutes mid-week, the old behaviour would have been an error at checkout and 40 minutes of abandoned carts. Instead, a health-check on the gateway flipped checkout into fallback mode: Cash on Delivery offered up front, order written and queued, prepaid retry scheduled. During the 40-minute outage, 312 orders went through COD fallback. About 41% of those later converted to prepaid via the WhatsApp payment link. The rest shipped as COD. Zero were lost. ## The GSTR-3B Extension Flow (Surviving Filing Day) The third outage was the GST portal itself, slow on filing day during a monsoon-heavy week. This is not a code problem — it is a process one, and most teams have no runbook for it. We wrote one. The finance team didn't panic because the snapshot existed two days early and the runbook told them exactly what to do. They filed comfortably under the extended window. The lesson: treat the GST portal like any other vendor that can go down, and have a fallback. ## Where the Orders Could Have Leaked (Visualised) When the shipping aggregator went down for an afternoon, 890 orders couldn't get a courier assigned in real time. They sat in the queue. When the aggregator recovered, the backlog cleared in 18 minutes with no duplicate bookings — because every courier-booking call carried its idempotency key. Customers saw a slightly later "shipped" SMS. Nothing more. The critical design choice here is that "courier not yet assigned" is a normal, valid order state — not an error. The customer's order confirmation never depends on the courier API being up. They get "order confirmed" immediately; the "shipped" notification follows whenever the aggregator is healthy. Decoupling the customer's experience from the vendor's uptime is the whole game. ## Common Mistakes (When Not to Over-Engineer This) - Random idempotency keys. Symptom: still double-charging despite "having idempotency." A key that changes per attempt does nothing. It must be deterministic and stored. - No health-check on the gateway. Symptom: customers hit errors for the full outage. Without a probe, you can't fail over to COD automatically. - Filing GST on the deadline day. Symptom: portal slow, finance panicking. Snapshot 48 hours early; the deadline-day portal is the most likely to choke. - Infinite retries. Symptom: a poisoned job hammers a vendor forever. Cap retries, then route to a dead-letter queue a human reviews. When should you not build all this? If you do 30 orders a day, a queue and a COD-fallback flow are overkill — handle the rare outage manually. This pattern earns its cost at a few hundred orders a day and up, where an hour of downtime is real money and manual recovery doesn't scale. We covered the GST half of this stack in our Shopify-to-Tally GST auto-sync guide and the filing-day specifics in our GST 2.0 day-one bug report. ## Real Outcome Three vendor outages in one monsoon week, 2,200 orders a day, and the brand lost nothing. The COD-fallback queue captured 312 orders during the payment outage; the durable queue cleared 890 stuck shipments in 18 minutes; the GST return filed on time under the extended deadline. Support tickets for the week were flat versus a normal week — customers never knew. We saw the same resilience-pays-off pattern building Radiant Finance's multi-portal pipeline, where decoupled stages kept 50,000+ leads flowing even when one portal was under maintenance. This is the kind of work our development team and automation team do for Indian D2C and retail brands. Softechinfra was founded by Vivek Singh, who has shipped this same retry-and-recover discipline across CRM, edtech, and ops projects for eight years. ## Frequently Asked Questions ### What is an idempotency key and why does it matter for payments? An idempotency key is a unique token sent with an API request so the vendor recognises a retry and returns the original result instead of repeating the action. For payments it prevents double-charges when a timeout triggers a retry. The key must be deterministic — derived from the order ID — and stored, so a crashed worker reconstructs the same key. ### How does a COD-fallback queue work during a payment outage? A health-check probes the gateway every 30 seconds. After three failures, checkout flips to offer Cash on Delivery up front and parks the order. The sale is captured instead of bounced. When the gateway recovers, a prepaid payment link is sent over WhatsApp — in our case 41% of fallback COD orders converted back to prepaid. ### What should I do when the GST portal is down on filing day? Snapshot your GSTR-3B numbers from your books 48 hours early, screenshot the portal error with a timestamp, log a grievance reference, and file under the extended deadline. CBIC has repeatedly extended GSTR-3B due dates during portal glitches, including January 2025 and the September 2025 return extended to 25 October. ### How many orders can a queue-based pipeline handle during an outage? As many as your queue can hold — that's the point. When our client's shipping aggregator went down for an afternoon, 890 orders queued and cleared in 18 minutes after recovery, with no duplicate bookings. The customer-facing impact was a slightly later shipping notification, nothing more. ### Is this kind of resilience worth it for a small store? Not at 30 orders a day — handle the rare outage by hand. The pattern earns its build cost at a few hundred orders a day and up, where an hour of downtime is real lost revenue and manual recovery doesn't scale. Below that, a simple queue and COD option is plenty. ### Does Razorpay support idempotency keys? Yes. Razorpay and most serious payment APIs honour an idempotency key so a repeated request returns the original result. The work on your side is generating the key deterministically and persisting it, so retries after a crash or network blip reproduce the exact same key.

Written by Vivek, our co-founder, with the engineering team that built the pipeline.