the customer who paged twice in the same hour

The page came in at 2:14 AM, then again at 2:51 AM, same customer, same vague description: ‘API is down.’ It wasn’t. Our 429s were up, sharply, but everything else was nominal. After twenty minutes of staring at dashboards that disagreed with the customer’s reality, we pulled their request logs and found the shape of the problem: a retry loop with no jitter, no backoff cap, and a budget that refilled faster than it drained. They were turning our rate limiter into a kick drum.

The interesting part wasn’t the fix. The interesting part was that we’d shipped a docs change two weeks earlier covering exactly this failure mode, and it hadn’t surfaced. Buried in a ‘best practices’ section nobody reads at 2 AM. Docs that describe the happy path are reference material. Docs that describe how things break are the ones engineers actually internalize. I’ve been rewriting our integration guide ever since, leading with the failure modes. The happy path can wait.