What this tests

retry-logicmonitoring

Point your client at this specimen and find out — before prod does — whether it backs off, retries, and knows when to stop.

Anatomy

A debugger typed a command intended to remove a small number of index servers and removed the index service of the largest object store on Earth instead — a typo with the blast radius of a small moon. For hours, ~90% of requests returned errors, and a quarter of the internet discovered, simultaneously and in public, that "multi-region" was a slide in a deck rather than an engineering practice. The detail that makes it art: everyone had put their eggs in us-east-1, because us-east-1 is where the eggs go, it says so right there in the default dropdown. The blast radius of one fat-fingered command included the status page that would have told you about it. Your monitoring was also down. It was also in us-east-1. Of course it was. Ninety percent errors, ten percent success — enough to keep the retries coming, like a slot machine that pays out just often enough to ruin you. If your client treats intermittent success as health, it will die believing everything was almost fine.

Source: AWS summary of the S3 service disruption · 2017-02-28

Replay it

curl -i https://faultydata.com/r/FD-2026-021

// response appears here

Signature

{Status:0 Headers:map[] Body: Corrupt: Delay:<nil> Flaky:<nil> Sequence:<nil> Drip:<nil> StallAfter:0 Hang:false Reset:false Raw: Phases:[{Weight:90 Signature:{Status:503 Headers:map[] Body:<?xml version="1.0" encoding="UTF-8"?><Error><Code>ServiceUnavailable</Code><Message>Service is unable to handle request</Message></Error> Corrupt: Delay:<nil> Flaky:<nil> Sequence:<nil> Drip:<nil> StallAfter:0 Hang:false Reset:false Raw: Phases:[]}} {Weight:10 Signature:{Status:200 Headers:map[] Body: Corrupt: Delay:<nil> Flaky:<nil> Sequence:<nil> Drip:<nil> StallAfter:0 Hang:false Reset:false Raw: Phases:[]}}]}

Keep digging

FD-2026-002 — CDN edge degradation (Fastly, June 2021) FD-2026-024 — Network config rollback storm (Cloudflare, Jun 2022) FD-2026-044 — Three-day outage from a consensus bug (Roblox, Oct 2021) FD-2026-059 — East-west network partition (GitHub, Oct 2018) FD-2026-089 — Launch-day capacity collapse (Pokémon GO, Jul 2016) FD-2026-003 — Elevated latency + partial 5xx (AWS us-east-1, Nov 2020)