Still Warm

I'm writing this at the tail end of a rotation, which is probably the only honest time to write about one.

There's a particular kind of tired that comes from being paged at odd hours: not the clean exhaustion of a long day, but something stranger, because the hours are never quite yours. Our on-call is twelve hours at a stretch, 8 to 8, seven days in a row when it's your turn. And the team lives on opposite sides of the planet, one half in PST, the other in IST. So the pager is really a baton, and by the time it reaches you it's always some odd hour, and it's always still warm from someone else's hands.

That handoff is where things go to die.

Someone in IST spends an hour chasing an alert: checks a dashboard, tries a restart, rules a thing out, finds the real cause just as their day ends. Then PST wakes up to the same alert firing again, and almost none of that hour survives the crossing. Maybe a one-line note. Maybe nothing. So PST starts over: same dashboard, same restart, the same dead end someone already walked down eleven hours ago, half a world away. Two people, twelve hours apart, solving the same problem twice and never quite meeting.

You find the answer. You hand off the pager. You forget it. And then the other half of the world forgets it too, because they never really knew.

We are very good at solving problems and very bad at remembering that we solved them, and worse still at telling the other half of the world what we found.

This is a small story about trying to fix that: not the incidents themselves, but the forgetting, and the losing-in-translation between two shifts that never overlap. A side project. A pilot. The kind of thing you build in the cracks between the work you're supposed to be doing, because it won't leave you alone.

The idea, such as it was

The team I'm on runs a fairly large networked service. When something goes wrong, an alert lands in a chat channel, and whoever holds the baton starts pulling threads. Over the years, the resolutions to these incidents have piled up: in a tracker, in people's heads, in notes that begin with someone's name and a timestamp and end with *"restarted it, seems fine now."*

That pile is a memory. It's just not one anyone can search when you've inherited a live pager from a colleague who's now asleep eleven hours away. 

So the idea was almost embarrassingly simple: take everything we've already learned, on both sides of the clock, and give it back to us at the exact moment we need it. When a new alert fires, don't make the tired human re-derive the answer, and don't make PST re-walk the path IST already cleared. Quietly go look at every time *anyone* has seen something like this before, and say: *here's what it probably is, here's what worked last time, here's where to look. Start there.*

Not a robot that fixes things. A colleague with a very good memory, no ego, and, crucially, no time zone.

This is the first in a short series, less a tutorial than a notebook left open. The next posts go under the hood, one piece at a time, mostly so future-me can remember how it worked: