Still Warm

I'm writing this at the tail end of a rotation, which is probably the only honest time to write about one.

There's a particular kind of tired that comes from being paged at odd hours: not the clean exhaustion of a long day, but something stranger, because the hours are never quite yours. Our on-call is twelve hours at a stretch, 8 to 8, seven days in a row when it's your turn. And the team lives on opposite sides of the planet, one half in PST, the other in IST. So the pager is really a baton, and by the time it reaches you it's always some odd hour, and it's always still warm from someone else's hands.

That handoff is where things go to die.

Someone in IST spends an hour chasing an alert: checks a dashboard, tries a restart, rules a thing out, finds the real cause just as their day ends. Then PST wakes up to the same alert firing again, and almost none of that hour survives the crossing. Maybe a one-line note. Maybe nothing. So PST starts over: same dashboard, same restart, the same dead end someone already walked down eleven hours ago, half a world away. Two people, twelve hours apart, solving the same problem twice and never quite meeting.

You find the answer. You hand off the pager. You forget it. And then the other half of the world forgets it too, because they never really knew.

We are very good at solving problems and very bad at remembering that we solved them, and worse still at telling the other half of the world what we found.

This is a small story about trying to fix that: not the incidents themselves, but the forgetting, and the losing-in-translation between two shifts that never overlap. A side project. A pilot. The kind of thing you build in the cracks between the work you're supposed to be doing, because it won't leave you alone.

The idea, such as it was

The team I'm on runs a fairly large networked service. When something goes wrong, an alert lands in a chat channel, and whoever holds the baton starts pulling threads. Over the years, the resolutions to these incidents have piled up: in a tracker, in people's heads, in notes that begin with someone's name and a timestamp and end with "restarted it, seems fine now."

That pile is a memory. It's just not one anyone can search when you've inherited a live pager from a colleague who's now asleep eleven hours away.

So the idea was almost embarrassingly simple: take everything we've already learned, on both sides of the clock, and give it back to us at the exact moment we need it. When a new alert fires, don't make the tired human re-derive the answer, and don't make PST re-walk the path IST already cleared. Quietly go look at every time anyone has seen something like this before, and say: here's what it probably is, here's what worked last time, here's where to look. Start there.

Not a robot that fixes things. A colleague with a very good memory, no ego, and, crucially, no time zone.

Teaching it to forget the right things

The first surprise was that the hardest part had nothing to do with AI.

Our alerts are noisy. The same underlying problem shows up as a hundred near-identical messages that differ only in the parts that don't matter: a pod's random suffix, a timestamp, which availability zone it happened in, and a number that ticked up by one. If you treat each of those as a unique event, you drown. If you can strip away the noise and see that they're all the same shape, a hundred alerts collapse into one thing worth remembering.

So before any of the clever stuff, I wrote something deliberately dumb: a function that takes an alert title and scrubs out everything volatile (the hashes, the timestamps, the cluster names) and leaves behind a canonical signature. No model. No tokens. Just patterns and a bit of stubbornness.

On real data, roughly eighteen hundred alerts collapsed into a couple hundred signatures. Ninety percent of the noise, gone, for free. I've come to think this is a small law of the universe: before you reach for intelligence, spend a while on subtraction.

Giving it a Memory

Once the noise was gone, the incidents that were left had something to say. The trick was letting a new alert find the old ones even when the words don't match: "pods are crashing" and "deployment not ready" can be the same story told by two different people.

The way you do this now is with embeddings: you turn each incident's text into a long list of numbers (a point in a space with more dimensions than anyone can picture) arranged so that things which mean similar things land near each other. Store those points in a database built for the purpose. Ask it "what's near this new alert?" and it hands you the neighbours.

I won't pretend I find this unmagical. You take a sentence about a broken DNS pod and it becomes fifteen hundred numbers, and somehow the geometry of those numbers knows that this incident is a cousin of one from four months ago. I know how it works. It still feels like something I shouldn't be allowed to do on a laptop.

Have we seen this before?

With a memory in place, the actual question became answerable. A new alert comes in; the system pulls the closest past incidents, but then does something a human does instinctively and a naive search doesn't: it narrows. Same kind of service? Same flavour of environment? Then those matter more. It ranks the fixes people actually tried, favouring the ones that worked, and quietly noting the ones that didn't: "last time, restarting didn't help; clearing the stale config did."

And when it has nothing close, it says so, and widens its net rather than inventing confidence it hasn't earned. That part mattered to me more than I expected. A tool that admits "I'm not sure, but here are the nearest things I've got" is one you can trust when you've just inherited someone else's week. A tool that always sounds certain is one you learn to ignore.

Closing the loop

The last piece was making it show up where the pain already is. The alerts land in a chat channel, so that's where the answer should appear, threaded under the alert, a few seconds later, like a colleague leaning over. No new dashboard to check. No new habit to build. The best tools are the ones you don't have to remember to use.

What it got wrong (and what that taught me)

It was not clean. It's never clean.

A key I'd been careless with got revoked mid-test and everything went dark until I understood why. A model I'd assumed I could use turned out to be off-limits, and the error message lied to me about the reason. The very first real incident it looked at, it correctly labelled a false alarm, which was right, and also useless, and made me realise the machine will faithfully remember our noise too if we let it.

Every one of those was a small lesson wearing a frustrating disguise. Rotate your secrets. Trust the data, not your assumptions about the data. And a memory is only as good as what you choose to put in it; remembering everything is just a different way of forgetting.

On memory, mostly

I set out to build a tool and ended up thinking a lot about memory: how much of what we call expertise is really just retained experience, and how little of it survives the handoff from one shift to the next. We treat forgetting as neutral, the absence of a thing. It isn't. It has a cost, and we pay it one lost hour at a time, twice, on opposite sides of the world.

I don't know yet if this pilot becomes something the team leans on, or a clever thing that quietly rusts in a repo. That's genuinely not up to me, and I've made a sort of peace with that. I built it because the forgetting bothered me, and building the thing was the only way to stop it bothering me.

Which, if I'm honest, is most of why I make anything at all.

This is the first in a short series, less a tutorial than a notebook left open. The next posts go under the hood, one piece at a time, mostly so future-me can remember how it worked:

The boring magic: turning noisy alerts into signatures
Giving it a memory: embeddings and a vector database
"Have we seen this before?": retrieval and ranking
Closing the loop: reading the room, answering back