Part 2: resilience patterns

Part 1 of this demo shipped with a known limitation, flagged repeatedly in its own write-up: nothing ever checked whether stock existed before an order was accepted. The async order_created event decremented stock after the fact, with a floor clamp (MAX(stock - qty, 0)) standing in for a real check. Part 2 fixes that properly, and picks up three more patterns along the way: Saga / Compensating Transaction, Circuit Breaker, and Retry.

What changed

A genuinely new integration: orders-service now calls inventory-service synchronously, before accepting an order at all.

flowchart LR
    client["Browser (UI)"]
    gw["API Gateway (nginx)"]
    orders["orders-service (FastAPI)"]
    inventory["inventory-service (FastAPI)"]
    broker[("Redis - order_created topic")]
    ordersDb[("orders.db (SQLite)")]
    inventoryDb[("inventory.db (SQLite)")]
    jaeger[("Jaeger - traces")]

    client --> gw
    gw -->|"/orders, /inventory"| orders
    gw -->|"/inventory"| inventory
    gw -->|"/jaeger/"| jaeger
    orders -->|reads/writes| ordersDb
    inventory -->|reads/writes| inventoryDb
    orders -->|"1: reserve/release (sync, retry + circuit breaker)"| inventory
    orders -->|"2: publish order_created + trace context (fulfilment log)"| broker
    broker -->|subscribe| inventory
    orders -.->|OTLP spans| jaeger
    inventory -.->|OTLP spans| jaeger

    classDef container fill:#1168bd,stroke:#0b4884,color:#fff
    class gw,orders,inventory,broker,ordersDb,inventoryDb,jaeger container

Two separate integrations now, deliberately: the synchronous call decides whether stock exists at all (strongly consistent), while order_created over Redis is now just a fulfilment/audit record (eventually consistent - stock was already settled by the time it’s processed). Conflating the two - letting the async event be the only check on stock - was Part 1’s actual problem.

Saga / Compensating Transaction

When two services each own their own database, there’s no distributed transaction spanning both - a multi-step operation that touches both can succeed partway and leave them inconsistent.

flowchart LR
    caller["orders-service (Container)"]
    resourceOwner["inventory-service (Container)"]
    store[("stock (Container)")]

    caller -->|"1: reserve"| resourceOwner
    resourceOwner -->|atomically decrements| store
    caller -->|"2: policy check"| caller
    caller -->|"3a: passed - create order"| caller
    caller -->|"3b: failed - release"| resourceOwner
    resourceOwner -->|undoes the reservation| store

    classDef container fill:#1168bd,stroke:#0b4884,color:#fff
    class caller,resourceOwner,store container

Placing an order now reserves stock first (POST /inventory/reserve), an atomic SQL UPDATE ... WHERE stock >= ? that can’t be raced by two concurrent orders both believing they have enough. A deliberately arbitrary policy check then runs - any order over 20 units is rejected, standing in for a downstream fraud/credit/compliance check that only makes sense once stock is already set aside. If it rejects, orders-service calls POST /inventory/release to undo the reservation rather than leaving stock short for an order that never goes through.

The 20-unit limit is labelled honestly as a demo hook, not a real policy - it exists purely to give anyone trying the demo a repeatable way to trigger the compensating path, by ordering a large quantity that’s still within stock.

Circuit Breaker and Retry

Composed together, not independently: Retry absorbs a brief transient blip (up to 3 attempts, linear backoff); Circuit Breaker wraps the whole retry sequence as one logical attempt, and after 3 consecutive failures opens and fails every subsequent call immediately - no network attempt at all - for a 10-second cooldown.

flowchart LR
    caller["orders-service (Container)"]
    breaker{"Circuit breaker: closed / open / half-open"}
    dependency["inventory-service (Container)"]

    caller -->|call| breaker
    breaker -->|closed: attempt, up to 3x with backoff| dependency
    breaker -->|open: fail immediately, no network call| caller
    dependency -->|3 consecutive failed attempts| breaker

    classDef container fill:#1168bd,stroke:#0b4884,color:#fff
    class caller,dependency container

Both are hand-rolled rather than pulled from a library - the point of a pattern demo is to show how the pattern works, not hide it behind a decorator. The whole mechanism is about 60 lines.

Measured directly, not estimated: stopping inventory-service and placing orders, the first three attempts each took roughly 8 seconds (a full retry sequence against a dead connection) before returning 503. The fourth request, once the circuit had opened, returned 503 in 69 milliseconds. That gap is the entire case for a circuit breaker in one number: once a dependency is known to be down, failing fast beats paying the full retry cost on every request.

See it for yourself

Every rejected or failed order on the live demo still returns a trace, with the retries, the circuit state, and (for a policy rejection) the compensating release call all visible in it - exactly the same way a successful order’s trace is shown.

Order 30 gadgets (over the limit, within stock) - reserves, then releases. Inventory stays unchanged.
Order 999 of anything - plain insufficient stock. No reservation ever happens, so nothing to release.

The circuit breaker needs a terminal, not just the browser:

docker compose stop inventory-service
# place a couple of orders - watch the first few take ~8s, then fail in ms
docker compose start inventory-service
# wait ~10s for the circuit's cooldown; orders succeed normally again

Catalogued

Both new patterns are documented in Architecture Governance Platform alongside the ones from Part 1 - see the pattern catalogue for the public copy.

On Azure instead of Docker Compose

Part 2 on Azure covers the same reservation call, and two ways to get Circuit Breaker + Retry on Azure Container Apps - porting this code as-is, or Dapr’s declarative resiliency policies instead.

Terms used on this page

See the glossary for plain-English definitions of Saga, Circuit Breaker, Retry, and the rest.

What’s next

Part 3 adds CQRS and Cache-Aside - a genuine read model built from the fulfilment event this page introduced, and a cache in front of GET /inventory.