Every lesson I actually retained came with an incident report and a client asking what happened. Here are five, with the real root cause of each.
1. The scheduled job that ran twice a second
A reminder job was supposed to send one email per pending item. A config change made the scheduler treat every run as a fresh batch, so it fired on every tick. Users got the same reminder forty times before anyone noticed. The bug wasn't the scheduler — it was that the job had no idempotency key. Lesson: any job that touches the outside world needs a record of what it already did. Not a flag in memory. A row in a table.
2. The timezone that only broke on Sundays
Weekly report, correct every day except Sunday, when it arrived six hours early. Root cause: a naive timestamp compared against UTC in one place and local time in another, and Sunday was the one day where the week boundary crossed midnight locally. Lesson: store UTC, convert at the edge, and never compare a naive datetime to an aware one. If your language allows that comparison without screaming, it is lying to you.
3. The migration that locked a table at peak traffic
An index addition on a table with a few hundred thousand rows. Fine in staging, which had a thousand. In production it took a write lock during the busiest hour of the day and the whole app went read-only for ninety seconds. Lesson: migrations are traffic-sensitive, not just size-sensitive. Run them in a maintenance window or use an online schema change tool.
4. The API key committed to a public repo
Someone pushed a test script with a live key. It was in the history for eleven minutes before a scraper found it and started hammering the API. Rotating the key fixed it in two minutes; the eleven minutes are what scared me. Lesson: assume anything pushed is public forever. Rotate on suspicion, and put a pre-commit hook in place the same day.
5. The harmless caching layer
We cached a lookup that changed rarely. It changed on a client-side admin action we forgot existed, so users saw stale data for hours and nobody could reproduce it. Lesson: cache invalidation is not a detail, it is the feature. If you can't describe exactly what invalidates the cache, you don't have a caching strategy, you have a bug with a delay.
The pattern
None of these were exotic. All five were assumptions nobody wrote down: that a job runs once, that a timezone is one timezone, that a table is small, that a repo is private, that data doesn't change. Write the assumption down and it becomes a test. Leave it in your head and it becomes an incident.