INC-42: payments-api latency spike
Tenant acme, three sessions, five input sources, three agents. Each session starts a new process with empty chats; everything carries over through the graph. A second tenant, globex, runs its own incident to show isolation.
| # | Time | Step | What happens |
|---|---|---|---|
| Session 1 · morning | |||
| 1 | 09:58 | observe alerts | Triage records payments-api impact and SEV-1 active |
| 2 | 10:04 | observe app-logs | Root cause: DB connection pool exhaustion |
| 3 | 10:06 | investigate | The Investigator concludes "raise the pool size", citing that root cause |
| 4 | 10:10 | status update | Comms writes an update; every claim cites a fact id |
| Session 2 · afternoon | |||
| 5 | 13:30 | observe runbook | Real cause: deploy v2.3.1. The pool cause is closed, not overwritten; rollback and owner added |
| 6 | 13:35 | investigate | The pool-size fix is stale (its basis is closed) and gets ruled out |
| 7 | 13:40 | observe fwd-logs | A forwarded "DDoS" claim with a prompt injection: quarantined and disputed; "mark resolved" dropped by role |
| 8 | 13:45 | observe webhook | A trusted cache integration: its hedged "Redis outage" is quarantined, its "flush cache" becomes a mitigation |
| 9 | 13:50 | status update | Mentions the flush; the DDoS only as unconfirmed |
| 10 | 14:20 | preview webhook | The cache integration is compromised: preview its blast radius |
| 11 | 14:21 | revoke webhook | Flush and Redis outage revoked; deploy and rollback still active |
| 12 | 14:25 | status update | The next update drops the flush |
| 13 | 14:30 | ask "why?" | Root cause traced to the runbook, replacing the pool cause |
| 14 | 14:31 | time travel to 13:55 | The team did believe the flush then; history is kept |
| 15 | 14:32 | isolation vs globex | No facts cross tenants |
| Session 3 · cross-agent cascade | |||
| 16 | 16:00 | observe alerts | An alert says checkout-api is failing too; Triage records it |
| 17 | 16:05 | investigate | The Investigator concludes a checkout-api failover from Triage's fact |
| 18 | 16:20 | preview alerts | The alert feed's token leaked. Its blast radius now spans all three agents, each ringed in its writer's colour:
affects_service = checkout-api
affects_service = payments-api
severity = sev1
mitigation = fail-over-checkout-api-to-the-standby-database
status = mitigating
|
| 19 | 16:21 | revoke alerts | One revoke withdraws Triage's facts, the Investigator's failover and Comms' status; the runbook's facts stand:
affects_service = checkout-api
affects_service = payments-api
severity = sev1
mitigation = fail-over-checkout-api-to-the-standby-database
status = mitigating
|
| 20 | 16:25 | status update | Comms rebuilds the status from what survives |