Giving it a memory

Oncall-agent, part 3. Part 2 was about teaching the thing what to ignore. This one is about the opposite: teaching it to notice that two problems nobody worded the same way are, underneath, the same problem. This is the part where I start to feel like I'm getting away with something.

In the very first post, I let one sentence slip that I've been chewing on ever since. I said that you take a sentence about a broken DNS pod, it becomes fifteen hundred numbers, and the geometry of those numbers somehow knows this incident is a cousin of one from four months ago; and that even though I know how it works, it still feels like something I shouldn't be allowed to do on a laptop. 

I want to pull on that thread, because it's the whole trick of this post, and because the feeling turns out to be pointing at something real.

The problem signatures couldn't solve

Part 2 had gotten me further than I expected, for free. Eighteen hundred noisy alerts had collapsed into a couple hundred signatures, each one a canonical fingerprint of a kind of problem. When a new alert reduced to a signature I'd already seen — exact match, done, no intelligence required. I was pleased with myself for about a week. 

But exact match is brittle in exactly the way humans aren't. Two incidents can be the same story and never share a signature:

• pods are crashing on startup, CrashLoopBackOff
• deployment never becomes ready, readiness probe failing

You read those and your gut says same neighbourhood. A string comparison says nothing in common — not one word lines up in a way a signature would catch. Signatures dedupe the identical. They're helpless against the merely similar. And "similar" is where most of the value of a memory lives, because the tired human at 3am didn't hit the exact same wall as last time. They hit a wall that rhymes. 

So I needed a way to ask a question no regex can answer: what do we have on record that means something like this?

Turning meaning into geometry

The modern answer is embeddings, and it really is a little uncanny. 

I hand a piece of text to an embedding model and it hands back a long list of numbers — a vector. The model I settled on is text-embedding-3-small (served through the same in-org gateway as everything else), and the list comes back 1536 numbers long. That vector is a point. Not a metaphorical point — an actual coordinate, in a space with 1536 axes instead of the 3 our eyes are built for. 

The magic — the part I keep circling — is what the arrangement encodes. The model was trained so that texts which mean similar things land near each other in that space, and texts that mean different things land far apart. "Pods are crashing" and "deployment never becomes ready" drift close together. "Certificate expired" ends up somewhere else entirely. Nobody wrote a rule for that. Nobody enumerated the synonyms. The closeness falls out of the numbers. 

So "have we seen something like this?" stops being a language question and becomes a geometry question: embed the new alert, then go find the nearest points. Distance in that 1536-dimensional space is a stand-in for similarity of meaning. That's the whole idea. It's absurd that it works, and it works. 

I can't picture 1536 axes — I doubt anyone can — so I flattened the intuition down to a plane I could draw, and scribbled it in the margin of my notes. Three incidents, embedded, laid out by meaning:

meaning-space (flattened from 1536 dims to 2 so it fits on a page)

                      · "certificate expired"

                       (far away — different problem)

        · "pods are crashing on startup"

         ╲

          ╲  small distance = similar meaning

           ╲

        · "deployment never becomes ready"

   new alert ──embed──▶ ● ──"nearest points?"──▶ the two on the left

Nothing in those two phrases lines up as text — but as points, they're neighbours, and the cert incident isn't. The model put them there; I just measure the gap. 

This is the sentence from part 1, unfolded: I understand each step, and the sum of the steps still feels like sleight of hand. I can hold the whole mechanism in my head — tokens in, floats out, cosine distance between floats — and the result still seems to know things the mechanism has no business knowing. I've stopped treating that feeling as ignorance. It's the correct response to compressing "what this sentence is about" into 1536 numbers and watching the compression survive.

What I actually embed (and what I don't)

Here's the detail I nearly got wrong, and would have, if the shape of the data hadn't stopped me: I don't embed the raw alert. 

My first instinct was to just throw the whole alert at the model and let it sort things out. But the raw alert is mostly noise, the very noise part 1 spent an afternoon stripping out. Embed all of it, and you've asked the model to find meaning in cluster names and pod hashes; it'll dutifully try, and the neighbourhoods come out muddier for it. The vector should carry signal, not ceremony. It took me a couple of muddy runs to actually believe that. 

So the text that gets embedded is a small, deliberate distillation of each incident; the human-meaningful part, and only that:

title

root cause

error signature

the on-call note ("restarting didn't help; clearing the stale config did")

Four lines, glued together, blanks skipped. That's what becomes the point. Everything else about the incident, i.e., which environment, which flavour of deployment, which components, which fixes were tried and whether they worked, is not embedded. It rides along as structured payload attached to the point:

{

  "signature":       "<env> ipsec pod not ready alert - <env>: ...",

  "env":             "...",

  "flavor":          "...",

  "component_tags":  ["ipsec", "networking"],

  "root_cause":      "...",

  "fixes_attempted": [{"action": "restart", "outcome": "no-op"},

                      {"action": "clear stale config", "outcome": "resolved"}]

}

This split — vector for meaning, payload for facts — turned out to be one of the more important calls I made, though I didn't know it at the time; the next post (retrieval and ranking) is really just a post about exploiting it. Search by meaning to find the candidates, then filter and rank by facts: same flavour of environment? favour it. this fix didn't work last time? remember that. The embedding gets you into the right neighbourhood; the payload is what lets you be a good neighbour once you're there. But that's getting ahead of myself.

The database that thinks in distance

A regular database is built to answer "give me the row where id = X." Useless here — I don't have an id, I have a location, and I want whatever's close to it. That's a different shape of question, and it wants a different kind of store: a vector database. I used Qdrant, which is open source and runs happily as a single container on a laptop, which is exactly the altitude this pilot lives at. 

Mechanically it's plainer than the word "vector database" makes it sound, which was a relief when I opened the docs braced for the worst. I created a collection and told it two things: how long the vectors are (1536, to match the model) and how to measure distance (I went with cosine, which cares about a vector's direction rather than its length — the right choice, it turns out, for "do these mean the same thing"). Then, for each incident, one point goes in: an id, its vector, and its payload.

create collection  (size: 1536, distance: Cosine)

for each incident:

    text  = distill(incident)          # the four signal lines

    vec   = embed(text)                # 1536 floats

    point = { id, vector: vec, payload: facts(incident) }

    upsert(point)

Later, when a new alert comes in, I embed it the same way and put a question to Qdrant that no SQL database could answer: "here's a point — hand me the ten nearest, with their payloads." It does, ranked by distance, in milliseconds. That query is the beating heart of the next post.

The unglamorous parts that actually mattered

Three things I'd tell my past self, none of them about embeddings: 

Make re-running free. The ingest job runs on a schedule, over an overlapping backlog, so the same incident gets processed again and again. If each run created a new point, the memory would bloat with duplicates — the part-1 problem sneaking back in through the front door. The fix is boring and total: derive each point's id deterministically from the incident id (a hash), so re-writing the same incident overwrites the same point. Run it once or a hundred times; the collection lands in the identical state. Idempotence isn't a nice-to-have here, it's what keeps the memory honest. 

Keep the network at arm's length. Embedding means a call to the gateway, which means a key, latency, and a thing that can be down. I put the embedder behind a one-method interface — give me texts, I'll give you vectors — so every piece that depends on it can be tested against a fake that returns canned vectors. No network, no tokens, no flakiness in the test suite. The clever, expensive, unreliable part is sealed behind a seam the rest of the code doesn't have to know about. (This sounds like a throwaway engineering note. It's the reason the project didn't stall.) 

The dimension is a contract. The collection is created with a fixed vector size. Swap the embedding model for one with a different dimension and every old point is suddenly incompatible — the geometry doesn't line up. So the model choice isn't a knob you casually turn; changing it means re-embedding everything. Worth knowing before, not after.

What it taught me

Part 2's lesson was subtraction — decide what to ignore. This part's lesson is its twin: once you've thrown away the noise, the thing left over has a shape, and embeddings are how you let a machine feel that shape without ever teaching it the words. 

There's something almost tender about it, if I'm allowed to be soft for a sentence. We spend the noise-stripping being ruthless, deleting everything specific and human — the cluster someone was staring at, the exact minute it broke. And then we hand what's left to a model whose entire job is to notice that this stripped-down husk of a problem resembles another one, months old, that some other tired person on the other side of the planet already solved. The forgetting I started this whole series worried about — embeddings are, quietly, a machine for its opposite. Not memory as a filing cabinet. Memory as recognition. 

Which is the part I don't think I should be allowed to do on a laptop. And yet: docker run, and there it is. 

Next time: the query itself. We've got a memory that answers "what's near this?" — but nearness alone is naive. A human doesn't just find similar incidents; they narrow: same kind of service, same flavour of environment, and they trust a fix that worked over one that didn't. That's retrieval and ranking, and it's where the payload finally earns its keep.