What this tests

timeoutsmonitoring

Point your client at this specimen and find out — before prod does — whether it has a timeout, and whether the timeout has a timeout.

Anatomy

Kinesis lost front-end capacity in us-east-1 in November 2020, and since half of AWS drinks from Kinesis — CloudWatch, Cognito, autoscaling, the very dashboards you'd use to notice — the blast radius included services whose only crime was standing too close at the bar. Correlated failure: the thing you don't think you depend on, failing because of the thing you didn't know you depended on. The signature is the nasty variant: not a clean outage but a malingering one. Elevated latency AND random 500s AND intermittent success, all at once, for hours. Every request a lottery ticket; the jackpot is your data, eventually, maybe, subject to availability. This is the failure mode that makes dashboards glow green while users queue for the fire exits. If your alerting watches error rate OR latency but never both, this specimen is about to embarrass it in front of management.

Source: AWS post-event summary, Kinesis/us-east-1 · 2020-11-25

Replay it

curl -i https://faultydata.com/r/FD-2026-003

// response appears here

Signature

{Status:0 Headers:map[] Body: Corrupt: Delay:<nil> Flaky:<nil> Sequence:<nil> Drip:<nil> StallAfter:0 Hang:false Reset:false Raw: Phases:[{Weight:60 Signature:{Status:200 Headers:map[] Body: Corrupt: Delay:0x69e720a14e50 Flaky:<nil> Sequence:<nil> Drip:<nil> StallAfter:0 Hang:false Reset:false Raw: Phases:[]}} {Weight:25 Signature:{Status:500 Headers:map[] Body:{"message":"Internal server error"} Corrupt: Delay:0x69e720a14e80 Flaky:<nil> Sequence:<nil> Drip:<nil> StallAfter:0 Hang:false Reset:false Raw: Phases:[]}} {Weight:15 Signature:{Status:200 Headers:map[] Body: Corrupt: Delay:<nil> Flaky:<nil> Sequence:<nil> Drip:<nil> StallAfter:0 Hang:false Reset:false Raw: Phases:[]}}]}

Keep digging

FD-2026-022 — AWS console/API errors (us-east-1, Dec 2021) FD-2026-090 — Backbone congestion black hole (Cloudflare, Jul 2020) FD-2026-043 — Datacenter literally on fire (OVH, Mar 2021) FD-2026-092 — Nationwide core-network kill (Rogers, Jul 2022) FD-2026-002 — CDN edge degradation (Fastly, June 2021) FD-2026-021 — S3 us-east-1 subsystem removal (AWS, Feb 2017)