Payment System
Idempotent money movement with a double-entry ledger, provider integration, and reconciliation.
The scenario
Money movement at Stripe scale — where idempotency-key IS the architecture
Same startup, same engineer #4. Eleventh Monday.
Your CTO drops by with a slightly different tone this time. "Users want to pay us. We need a payment system. Credit cards, wallets, subscriptions, refunds. Later — payouts to sellers (marketplace flow). Ship an MVP in 12 weeks. And it MUST NOT lose money. Not a single cent, ever. Reconciliation to the penny."
She pauses. "For context — Stripe processed $1.4 trillion in payment volume in 2024, ~1.3% of global GDP. Their ledger logs 5 billion money-movement events per day. And they INVENTED the Idempotency-Key HTTP header pattern that's now an IETF draft standard. If we're going to compete, we need to grasp exactly what makes payments engineering different from every system we've built."
Here's the paradigm shift:
URL Shortener, Slack, Instagram, Twitter, WhatsApp, Netflix, YouTube, Dropbox, Uber — all had one shared property: occasional data loss was acceptable as long as it didn't happen often. A dropped tweet, a delayed message, a missed video segment — annoying but survivable. Availability + consistency + latency were the trade-off dimensions.
Payment systems add durability + auditability as ABSOLUTE constraints. You cannot lose a cent. You cannot double-charge. You cannot let a race condition create phantom transactions. You cannot let the system be "eventually consistent" about balances — regulators require exact reconciliation daily. It's not a normal distributed-systems problem. It's a distributed accounting problem with 100+ years of legacy accounting principles imposed on top.
Stripe's answer: (1) Idempotency-Key HTTP header on every write (Stripe Idempotent Requests — invented by Stripe, now IETF draft). (2) Double-entry ledger as the source of truth (Stripe Engineering: Building a modern financial data pipeline). (3) Saga pattern for cross-service money movement because 2PC is too slow. (4) Nightly reconciliation against every payment provider's transaction report — because if you don't catch discrepancies within 24h, they compound.
The real 2024 numbers
- $1.4T annual payment volume — ~1.3% of global GDP (Stripe 2024 update)
- 500M+ API requests/day — ~10-20% are actual financial transactions
- ~5B money-movement events/day in the ledger — every event immutable + audited
- BFCM 2024 peak: 137K transactions/minute = ~2,283 payment TPS (Stripe BFCM 2024)
- BFCM 2024 API peak: 27,395 requests/sec
- API uptime: >99.999% during peak events
- ~100+ countries with regional infrastructure
Compare to card networks:
- Visa: 10,242 TPS average, >65,000 TPS peak (Visa network stats 2025)
- Mastercard: 6,972 TPS average, ~30,000 TPS peak
- Global credit card transactions: ~889B/year = 28,152 TPS globally averaged
Interview soundbite: "Stripe at $1.4T volume runs Idempotency-Key on every write, a double-entry sharded ledger, Saga for cross-service transactions, and nightly reconciliation against every provider's transaction report. Their peak is 2,283 payment TPS — impressive but 30x below Visa's 65K peak, because Stripe isn't a card network, it's the API layer on top of card networks."
The whole journey at a glance
Every 10× in transaction volume forces different consistency + reconciliation strategies:
text═══════════ PAYMENT SYSTEM ARCHITECTURE ACROSS 4 SCALES ═══════════ L4 (10K txns/day) L5 (10M txns/day) L6 (100M txns/day) L7 (1.4T volume Stripe) Stripe wrapper Own ledger + Stripe Sharded ledger + Saga Multi-region + KYC/AML 12 weeks · $500/mo 6 months · $50K/mo 18 months · $5M/mo ongoing · $500M/yr ┌────────┐ ┌────────┐ ┌── Global CDN ──────┐ ┌── Own edge + WAF ───┐ │ Web │ │ Web │ │ API gateway │ │ Cloudflare + │ │Mobile │ │Mobile │ │ DDoS protection │ │ AWS Shield + WAF │ │Server │ │Server │ └─┬──┬──┬──┬─────────┘ └──┬──┬──┬──┬──────────┘ └───┬────┘ └───┬────┘ │ │ │ │ │ │ │ │ │ │ ┌─▼──▼──▼──▼─────┐ │ │ │ │ ┌──▼───┐ ┌──▼──┐ │ Regional ALB │ ┌───▼──▼──▼──▼──────────┐ │ ALB │ │ ALB │ │ + API gateway │ │ Global API Gateway │ │ │ │ │ │ + Idempotency │ │ + Idempotency-Key │ └──┬───┘ └──┬──┘ │ dedup │ │ dedup at edge │ │ │ └───┬──────┬─────┘ │ + Sorbet Ruby types │ ┌──▼──────┐ ┌───▼─────────┐ ┌───▼──────▼───┐ │ + rate limiters │ │ Django │ │ Java Spring │ │ Payment svc │ └──┬──┬──┬──┬───────────┘ │ + MySQL │ │ + MySQL │ │ Fraud svc │ │ │ │ │ │ + Stripe│ │ + Redis │ │ Ledger svc │ ┌──▼──▼──▼──▼──────────┐ │ API │ │ + Kafka │ │ Payout svc │ │ 100+ microservices │ └──┬──────┘ └───┬─────────┘ └───┬───────────┘ │ Payment / Fraud / │ │ ┌───▼──┐ ┌───▼──────────┐ │ Ledger / Reconcile / │ │ │Redis │ │Sharded Ledger│ │ Payouts / KYC / Tax │ │ │idem │ │(double-entry)│ │ + Radar ML fraud │ │ │dedup │ │+ Saga coord │ └──┬──┬──┬──┬──────────┘ │ └───┬──┘ └───┬──────────┘ │ │ │ │ │ │ │ ┌──▼──▼──▼──▼─────────┐ ┌──▼───┐ ┌───▼─┐ ┌───▼──────────┐ │Sharded Ledger │ │Postgr│ │Postgr│ │Sharded MySQL │ │(regional + async │ │(payme│ │+ shar│ │+ Saga svc │ │ global reconcile) │ │+ user│ │d by │ │+ Provider │ │+ Saga per txn │ │+ Stri│ │user │ │ abstraction │ │+ 5B events/day │ │peLink│ │_id │ │(Stripe+Adyen)│ │+ Immutable audit │ │Multi-│ │+ RF=2│ │+ Chargeback │ │ log per event │ │AZ) │ │ │ │ reconcile │ └──┬───────────────────┘ └──────┘ └──────┘ └──────────────┘ │ ┌───▼──────────────────┐ │Provider abstraction │ │Stripe/Adyen/Braintree│ │+ failover ordering │ │+ SCA/3DS EU/UK │ │+ ISO 8583 cards │ │+ ISO 20022 wire │ └───────────────────────┘ ↑ ↑ ↑ ↑ Boring Own ledger enters Saga pattern for cross- Multi-region ledger. Stripe wrapper. Redis dedup on service txns. Provider KYC/AML. Regulatory MySQL for meta. idempotency-key. failover. Chargeback compliance per country. Sharded MySQL. reconcile. $1.4T volume Stripe. Chapter 5 Chapters 6+6.5 Chapter 7+7.5 Chapter 8 walks walks through own walks through Saga + walks through Stripe's through ledger + shard by provider failover + actual 2024 stack: L4 MVP user_id + Redis chargeback flow + 5B events/day, BFCM idempotency dedup reconcile jobs 2,283 TPS, Radar ML Key insight: Payment systems are NOT distributed-systems problems with money. They are DISTRIBUTED ACCOUNTING problems, with 100 years of accounting invariants imposed on top. Idempotency-Key + double-entry + Saga + nightly reconciliation are the 4 primitives that make this work at scale. Stripe invented the Idempotency-Key pattern; it's now an IETF draft standard.
The same 4 tiers as clean architecture diagrams
L4 · 10K txns/day · Stripe wrapper + MySQL · $500/mo · 12 weeks:
flowchart TD
W([Web / Mobile / Server]) -->|HTTPS + Idempotency-Key| LB[ALB<br/>$25/mo]
LB --> API[Django API<br/>c5.large<br/>~1K txn/day]
API --> MY[(MySQL Multi-AZ<br/>payments + users + subscriptions)]
API -->|Stripe.charges.create<br/>+ idempotency_key| ST[Stripe API<br/>PaymentIntents]
ST --> API
classDef n fill:#dbeafe,stroke:#2563eb,color:#1e3a8a
class LB,API,MY,ST nL5 · 10M txns/day · Own ledger + Sharded MySQL · $50K/mo · 6 months:
flowchart TD
W([Global merchants]) -->|Idempotency-Key required| LB[Regional ALB]
LB --> API[Java Spring services]
API --> RD[(Redis idempotency-key dedup<br/>TTL 24h)]
API --> LDG[(Sharded MySQL<br/>ledger by user_id<br/>double-entry)]
API --> KF[Kafka payment events<br/>audit trail + downstream]
API -->|failover order| PROV[Provider abstraction<br/>Stripe -> Adyen -> Braintree]
classDef n fill:#dbeafe,stroke:#2563eb,color:#1e3a8a
classDef m fill:#fef3c7,stroke:#d97706,color:#78350f
class LB,API n
class RD,LDG,KF,PROV mL6 · 100M txns/day · Sharded Ledger + Saga · $5M/mo · 18 months:
flowchart TD
W([Global merchants]) -->|Idempotency-Key + SCA/3DS| CDN[CDN + WAF + DDoS]
CDN --> API[100+ microservices<br/>Payment / Fraud / Ledger / Payout]
API --> RD[(Redis idempotency-key dedup)]
API --> SAGA[Saga coordinator<br/>cross-service transactions<br/>compensating actions]
SAGA --> LDG[(Sharded Ledger MySQL<br/>double-entry<br/>immutable audit log)]
API --> FRAUD[Fraud service<br/>sync rules + async ML]
API --> RECON[Nightly reconciliation<br/>vs provider reports]
RECON --> LDG
API --> PROV[Provider abstraction<br/>Stripe + Adyen failover<br/>SCA/3DS enforcement]
classDef n fill:#dbeafe,stroke:#2563eb,color:#1e3a8a
classDef m fill:#fef3c7,stroke:#d97706,color:#78350f
classDef s fill:#dcfce7,stroke:#16a34a,color:#14532d
class CDN,API n
class RD,LDG,PROV m
class SAGA,FRAUD,RECON sL7 · Stripe-scale · $1.4T volume · Multi-region ledger + KYC/AML · $500M/yr:
flowchart TD
W([Global merchants + consumers]) -->|Idempotency-Key IETF draft| CDN[Cloudflare + AWS Shield<br/>+ WAF + rate limiters]
CDN --> API[Global API Gateway<br/>Sorbet Ruby type-checked<br/>Envoy + gRPC]
API --> IDEM[(Redis idempotency-key dedup<br/>+ per-shard MySQL UNIQUE)]
API --> LDG[(Sharded Regional Ledger<br/>double-entry per region<br/>5B events/day<br/>immutable audit)]
API --> SAGA[Saga per transaction<br/>compensating actions<br/>cross-region money movement]
API --> RADAR[Radar ML fraud<br/>real-time sync rules<br/>+ async deep ML]
API --> KYC[KYC/AML pipeline<br/>Jumio/Alloy identity<br/>OFAC/PEP screening<br/>SAR reporting]
API --> RECON[Global reconciliation<br/>nightly + intra-day<br/>vs Visa/MC/ACH/SEPA/etc]
API --> PAYOUT[Payouts service<br/>Stripe Connect pattern<br/>merchant marketplace]
classDef n fill:#dbeafe,stroke:#2563eb,color:#1e3a8a
classDef m fill:#fef3c7,stroke:#d97706,color:#78350f
classDef s fill:#dcfce7,stroke:#16a34a,color:#14532d
class CDN,API n
class IDEM,LDG,PAYOUT m
class SAGA,RADAR,KYC,RECON sWhy every 10× breaks the architecture
- Idempotency changes. L4 = trust Stripe's idempotency-key. L5+ = build your own dedup layer because you now have multiple downstream providers + retries at every layer. Reference: Stripe Idempotent Requests docs — Stripe's pattern, now IETF draft.
- Ledger topology changes. L4 = MySQL single primary. L5 = sharded by user_id (fits in one region). L6+ = sharded regional ledger with Saga for cross-shard/cross-region money movement because 2PC is too slow at high throughput. Reference: Chris Richardson's Saga pattern.
- Reconciliation changes. L4 = manual weekly. L5 = automated daily vs one provider. L6+ = intra-day + nightly + cross-provider reconciliation because your 100M txns/day would create 8 hours of accumulated error weekly. Reference: Stripe Engineering: Modern financial data pipeline.
The 3 senior insights before we start Chapter 1
- Idempotency-Key is Stripe's most influential invention. Every serious payment API (Square, Adyen, PayPal) copied it. It's now IETF draft-idempotency-key. Say "we use idempotency" is red-flag vague; say "Idempotency-Key HTTP header with 24h Redis dedup + per-shard MySQL UNIQUE constraint (Stripe pattern)" and you signal senior credibility.
- Double-entry ledger is not optional — it's the accounting foundation for the last 500 years. Not "just log payments in a database". Every money movement is 2 rows: debit source + credit destination. Sum of debits ALWAYS = sum of credits. This mathematical invariant catches bugs that would otherwise corrupt balances silently. Reference: Wikipedia — Double-entry bookkeeping.
- Saga > 2PC at payment scale. 2PC blocks all participants until commit; at 100M+ txns/day with cross-service moves, you can't afford the coordination overhead. Saga uses compensating actions (refund vs release vs void) to undo failed steps. Reference: Chris Richardson's Saga pattern. Naming compensating-action semantics (refund vs release vs void) signals senior awareness.
Chapter map for the journey ahead
- Chapter 1 — Requirements (payment, subscription, refund, dispute, payout)
- Chapter 2 — Capacity estimation ($1.4T volume, 2,283 payment TPS peak, 5B events/day)
- Chapter 3 — API design (create-charge, subscription, refund, payout — with Idempotency-Key)
- Chapter 4 — Data model (users, subscriptions, ledger with double-entry, disputes)
- Chapter 4.5 — Double-entry ledger: debit/credit invariants + immutable audit
- Chapter 5 — L4 MVP: Stripe wrapper + MySQL. Works up to 10K txns/day
- Chapter 6 — L5: Own ledger + sharded MySQL + Redis idempotency-key dedup
- Chapter 6.5 — Idempotency-Key deep-dive: Stripe's invention + IETF draft
- Chapter 7 — L6: Saga for cross-service transactions + provider failover + Radar-style fraud
- Chapter 7.5 — Reconciliation architecture: nightly vs intra-day vs cross-provider
- Chapter 8 — L7: Stripe-scale + multi-region + KYC/AML + regulatory compliance
- Chapter 9 — Failure modes: double-charge, phantom transaction, race on chargeback, ledger drift
- Chapter 10 — Trade-off matrix (Stripe vs Adyen vs Braintree vs Square)
- Chapter 11 — Interview masterclass: 45-min mock, questions to ask
- Chapter 12 — Defense: the 20 hardest interview questions on payments
Ready? Chapter 1 next: what did the CTO actually ask for?
Payment systems at Stripe scale ($1.4T volume, 2,283 payment TPS peak BFCM 2024) are distributed accounting problems, not distributed-systems problems with money. Idempotency-Key HTTP header (Stripe-invented, now IETF draft) + double-entry ledger + Saga for cross-service transactions + nightly reconciliation vs provider reports are the 4 primitives that make this work. Naming Idempotency-Key + Saga + compensating actions (refund vs release vs void) is table stakes; naming Radar + Sorbet type-checker signals senior awareness.
- Why is Idempotency-Key Stripe's most influential invention (now IETF draft)?
- Why is double-entry ledger not optional for payment systems?
- Why does Saga beat 2PC at payment scale?
- What's the difference between Stripe (2,283 payment TPS) and Visa (65K peak TPS)?
- How does nightly reconciliation vs provider reports catch bugs before they compound?
Every concept below has its own interactive, animated page in the Learning Tracks section. Read them any time you want to go deeper than the mentor prose above — they're the reusable foundation this chapter is built on.
The algorithm behind sharded-ledger placement at L5+ so account_id maps consistently to a shard across scale-outs.
Every ledger event is published to Kafka for downstream reconciliation + fraud ML + audit. Immutable append-only stream matches the ledger's immutability invariant.
Chapter 1 next: what did the CTO actually ask for? Payment, subscription, refund, dispute, payout — each has functional and non-functional requirements. Get these wrong and you'll design the wrong system for the whole 12 chapters.
Components used in this design
Study each component's deep dive to understand it in isolation.
Load Balancer
Distributes incoming traffic across a pool of servers for scale and fault tolerance.
SQL Database
A row-oriented, ACID-compliant relational database — the default for transactional workloads.
Redis
An in-memory key-value store used for caching, pub/sub, rate limiting, distributed locks, and simple queues.
Kafka
A distributed, partitioned, replicated commit log for event streaming, high-throughput ingest, and decoupled services.
Message Queue
Async task queue that decouples producers from consumers and smooths bursts. Each message is processed exactly once (per-message ACK model).
API Gateway
Single entry point that handles auth, rate limiting, routing, and protocol translation for downstream services.
Rate Limiter
Enforces per-caller (per-user, per-IP, per-tenant) request budgets to protect downstream systems from abuse and overload.
Distributed Lock (ZooKeeper, etcd, Redis Redlock)
Cross-node mutual exclusion — 'only one node can do this at a time' when you can't rely on a single-node lock.
Workflow Engine (Temporal, Cadence, AWS Step Functions)
Orchestrates long-running, stateful multi-step processes — the right tool when a business flow involves many services and can span days.
Distributed Database (DynamoDB, Cassandra, Spanner)
Horizontally-scaled database with automatic partitioning and replication — the answer when one node can't hold the data or the traffic.
Patterns applied in this design
Study each pattern's deep dive for the recurring solution logic.
Circuit breaker
A downstream service is unhealthy. Your service keeps calling it, waiting the full timeout on every call, exhausting your thread pool, and cascading the failure back to your callers.
Saga (Long-running distributed transaction)
You have a business transaction that spans multiple services or databases (place order → charge payment → reserve inventory → send confirmation). Two-phase commit is too slow, too coupling, and often not available across service boundaries.
Event sourcing
Traditional state-based storage loses history: you know the current balance, but not why it's that value. Auditing, replay, and temporal queries are hard.
CQRS (Command Query Responsibility Segregation)
Your write model and read model have very different needs. Optimizing for one hurts the other.
Bulkhead
One misbehaving caller or workload consumes all your resources (threads, connections, memory) and starves everyone else.
Rate limiting
One bad actor can consume all your capacity. And even good actors need bounds so you can capacity-plan.
Explore next — related systems
If you enjoyed this problem, these share similar patterns or challenges.