Payment Architecture: Why Subscriptions Fail in the Middle, Not at Signup
Idempotency, partial failure, and where the source of truth lives between your database and your payment provider.
Payments are the one place in your product where a bug converts directly into lost money or an angry customer. And yet they are usually treated as an ordinary API call, which is where the trouble starts.
The network fails between request and response
The dangerous case is not a failed payment, it is a successful one you never heard about: you sent the request, the card was charged, and the connection dropped before the response arrived. If the user retries, they are charged twice. The fix is an idempotency key — an identifier you generate and send with the request, so the provider recognises the repeat and returns the original result instead of executing again.
The provider is the source of truth, not your database
- Never grant access based on a frontend response — it can be tampered with, and it may never arrive.
- Grant access when a signed confirmation arrives from the provider, because that is the only trustworthy event.
- Store the provider's transaction identifier on your row; it is what you return to in every dispute.
- Reconcile your state against theirs on a schedule — drift happens, and finding it late costs more.
Partial failure is the normal case
The charge succeeds but the row write fails. Or both succeed and the confirmation email does not send. Do not write these as one sequence assumed to complete. Make each step safely repeatable, persist state after each one, and separate what must happen immediately from what can happen a second later.
A customer who paid and got no access will contact you within minutes. A customer who got access without paying never will.
A subscription is not a recurring charge
A subscription is a state machine: active, past due, cancelled but with access until period end, expired. Most bugs come from collapsing that into one boolean. Define the states and the transitions explicitly, and decide what happens during a dunning period — cutting access on the first failure loses customers to expired cards, not to intent.
Renewal failure is predictable
- Notify days before renewal, since most failures are expired cards that could have been updated in advance.
- Retry with increasing gaps across days, not minutes.
- Keep access through an announced grace period rather than cutting on the first decline.
- Make updating a card a direct link in the email, not a journey through settings.
Getting this right takes weeks and never stops needing maintenance. The clearest practical documentation of idempotency keys is Stripe's page on idempotent requests, and their webhooks guide explains why every handler has to be safely repeatable — both written as specifications rather than marketing.
If you are building a subscription product, this is the area we most often see rebuilt after the first loss. Review your design with us.
FAQ
Where does the idempotency key come from?
Generate it in the client when checkout begins and keep it across retries; regenerating on every click defeats the purpose.
What if the same webhook arrives twice?
Expect it — it is guaranteed to happen. Store the event id and ignore what you have processed; every webhook handler must be safely repeatable.
Should I store card details?
No. Use a provider token; storing cards puts you under compliance requirements you do not want.