PBO Tech

← Microservices Pattern Demo · builds on How to run event-driven microservices on Azure

Part 2 on Azure: resilience patterns

This picks up where the main Azure guide left off - its own “Optional next steps” named exactly this: “a reservation saga, replacing the stock clamp.” Assumes that guide’s Steps 1-4 are already done: a Container Apps environment, orders-service and inventory-service both running as internal Container Apps on it.

What’s new

A direct, synchronous call from orders-service to inventory-service - something the original guide’s architecture never needed, since the only coupling was the Service Bus topic. On Container Apps this needs nothing extra to provision: two apps on the same environment already resolve each other by name over HTTP.

flowchart LR
    orders["orders-service (Container App)"]
    inventory["inventory-service (Container App)"]
    pgInventory[("PostgreSQL: inventory")]

    orders -->|"1: reserve/release (sync)"| inventory
    inventory -->|atomically decrements/restores| pgInventory

    classDef container fill:#1168bd,stroke:#0b4884,color:#fff
    classDef managed fill:#999999,stroke:#6b6b6b,color:#fff
    class orders,inventory container
    class pgInventory managed

inventory-service gets two new endpoints, same shape as the Compose version, atomic against PostgreSQL instead of SQLite:

# inventory-service: reserve and release, atomic against concurrent requests
from fastapi import HTTPException

@app.post("/inventory/reserve")
def reserve(req: ReservationRequest):
    with db() as conn, conn.transaction():
        cur = conn.execute(
            "UPDATE stock SET qty = qty - %s WHERE item = %s AND qty >= %s",
            (req.qty, req.item, req.qty))
        if cur.rowcount == 1:
            return {"reserved": True}
    raise HTTPException(status_code=409, detail=f"insufficient stock for {req.item}")

@app.post("/inventory/release")
def release(req: ReservationRequest):
    with db() as conn, conn.transaction():
        conn.execute("UPDATE stock SET qty = qty + %s WHERE item = %s", (req.qty, req.item))
    return {"released": True}

orders-service calls reserve before writing an order, and release if a downstream check rejects it afterwards - the same Saga shape as the Compose version, portable unchanged. The httpx instrumentation from the original demo still works here too: configure_azure_monitor() (Step 3d of the main guide) auto-instruments outbound HTTP calls, so this reservation call shows up in Application Insights’ trace the same way Service Bus calls already do.

Circuit Breaker and Retry: two ways to get there

Option A - port the code as-is. The demo’s own hand-rolled resilience.py (a small CircuitBreaker class, a retry() function) is plain Python with no Redis- or Compose-specific dependency - it runs inside orders-service exactly as written, calling inventory-service’s new URL instead of http://inventory-service:8000 becoming whatever Container Apps resolves it to internally. Nothing to provision; fastest path if you want to compare directly against the Compose version’s code.

Option B - Dapr’s resiliency policies, configuration instead of code. Azure Container Apps has built-in Dapr support (--enable-dapr on az containerapp create/update), and Dapr ships retry and circuit-breaker policies as a declarative spec rather than application code - genuinely the more “Azure-native” way to get this, if you’d rather tune resilience as config than ship a new container image to change a retry count.

# resiliency.yaml - a Dapr Resiliency spec targeting inventory-service
apiVersion: dapr.io/v1alpha1
kind: Resiliency
metadata:
  name: inventory-resiliency
spec:
  policies:
    retries:
      inventoryRetry:
        policy: constant
        duration: 300ms
        maxRetries: 3
    circuitBreakers:
      inventoryBreaker:
        maxRequests: 1
        interval: 10s
        timeout: 10s
        trip: consecutiveFailures >= 3
  targets:
    apps:
      inventory-service:
        retry: inventoryRetry
        circuitBreaker: inventoryBreaker

With Dapr enabled on both apps (--dapr-app-id orders-service --dapr-app-port 8000, same for inventory), orders-service would call through its sidecar’s service-invocation API (http://localhost:3500/v1.0/invoke/inventory-service/method/inventory/reserve) instead of calling inventory-service’s URL directly, and the sidecar applies this policy transparently - resilience.py wouldn’t be needed at all. Applying a custom resiliency spec like this one to a Container Apps environment is managed through Bicep/ARM or the portal in most current setups rather than a single stable CLI flag - check az containerapp env -h for what your CLI version supports before assuming a specific command here, the same hedge the main guide makes about Postgres flags drifting across versions.

Either option gets you the same two behaviours this demo’s own README verifies directly: a brief blip absorbed invisibly by retry, and a sustained outage failing fast once the breaker opens rather than paying the full retry cost on every request.

Verify

Same two checks as the Compose version, run against the deployed Container Apps:

1. Compensation. Order more than the demo’s policy limit (20), within stock - reserves, then releases:

curl -sS -X POST $GW/orders -H 'Content-Type: application/json' -d '{"item":"gadget","qty":30}'
# -> 422, "reservation released"; GET $GW/inventory shows gadget unchanged

2. Circuit breaker. Deactivate inventory-service’s current revision - the same technique the main guide’s own Step 5 uses to simulate an outage - place a couple of orders, then bring it back:

REV=$(az containerapp revision list -g $RG -n inventory-service --query "[0].name" -o tsv)
az containerapp revision deactivate -g $RG -n inventory-service --revision $REV

curl -sS -X POST $GW/orders -H 'Content-Type: application/json' -d '{"item":"widget","qty":1}'
# -> 503, after several retried attempts
curl -sS -X POST $GW/orders -H 'Content-Type: application/json' -d '{"item":"widget","qty":1}'
curl -sS -X POST $GW/orders -H 'Content-Type: application/json' -d '{"item":"widget","qty":1}'
curl -sS -X POST $GW/orders -H 'Content-Type: application/json' -d '{"item":"widget","qty":1}'
# -> circuit open, this one returns 503 fast rather than retrying

az containerapp revision activate -g $RG -n inventory-service --revision $REV
# after the cooldown, orders succeed normally again

Part 3 on Azure adds a CQRS read model (no new resource needed) and Cache-Aside (which does - there’s no Redis already running in this variant to reuse).

Unfamiliar with a term on this page? See the glossary for plain-English definitions, including the Azure-specific ones.