Architecture Lab
Resilient payment processing
Resilience patterns for payment-like workloads: retries, timeouts, poison messages and operator visibility.
Resilience is not “add retries everywhere”. It is designing for partial failure without corrupting money movement or customer trust.
Key concerns include timeout budgets, idempotent retry, poison-message handling, and enough observability for humans to intervene safely.
Sanitised system shape
Ingress Request
│
▼
Timeout Budget
│
▼
Idempotent Handler
│
┌─────────┴─────────┐
▼ ▼
Success Retryable
│ │
▼ ▼
Durable Result Backoff / Queue
│
▼
Poison / DLQ
│
▼
Operator Tools
Decision
Make retries idempotent and bounded, with explicit poison handling and operator visibility, rather than unbounded automatic retry.
Trade-off
Safer failure handling adds complexity in code and operations, but prevents silent corruption and cascading overload.
Failure mode
Duplicate side effects, retry amplification, stuck poison messages, and incidents where operators cannot see or intervene.
Why I chose this
In payment-like systems, “eventually processed twice” can be worse than “temporarily delayed”. Resilience must protect correctness.