The boring magic

On-call agent, part 2. The first one was the story; this one goes under the hood, starting with the least glamorous piece, which is also the one that mattered most.

Here's a thing nobody tells you about building something with "AI" in it: the part that does the most work often has no AI in it at all. 

I spent a while, before any model, before any database, on a function so dumb I hesitated to commit it. It takes an alert and throws most of it away. And it turned out to be the load-bearing wall of the whole project.

The problem: a hundred alerts, one problem

Our alerts are machine-generated, which means they are precise, and precision is the enemy here. The same underlying failure shows up worded almost identically across every cluster it happens in, differing only in the parts that don't matter. Here's one real alert as it fires in three different environments:

• ddi-prd-nva1 ipsec Pod not ready alert - ddi-prd-nva1: ipsec Pods are not ready for more than allowed transient duration
• ddi-prd-syd1 ipsec Pod not ready alert - ddi-prd-syd1: ipsec Pods are not ready for more than allowed transient duration
• ddi-prd-fra1 ipsec Pod not ready alert - ddi-prd-fra1: ipsec Pods are not ready for more than allowed transient duration

 You and I read those and see one thing: IPSec pods aren't coming up. A computer, told to be helpful, sees three unrelated events and cheerfully files three tickets. Feed that to a "memory," and your memory fills with near-duplicates until it's as useless as the spreadsheet it replaced. So before I could teach the thing to remember, I had to teach it what to ignore.

The idea: keep the shape, drop the noise

The trick is a signature: strip out everything volatile and keep what's left. The cluster name, the pod hash, the timestamp, the availability zone, the number that ticked up by one, all of it is noise from the perspective of "what kind of problem is this?" Take it out, replace it with a placeholder, and the three lines above collapse into one:

• <env> ipsec pod not ready alert - <env>: ipsec pods are not ready for more than allowed transient duration

That string is the key. Every alert that reduces to it is, as far as the system is concerned, the same story.

How it works (it really is this plain)

It's an ordered list of find-and-replace rules. Order matters, most specific first, so a timestamp gets caught before a bare number does:

• lowercase, trim 
• env / cluster names    -> <env> 
• full timestamps          -> <ts> 
• bare dates                 -> <date> 
• UUIDs                       -> <uuid> 
• replicaset + pod hash -> -<pod> 
• long workload IDs      -> <id> 
• availability zone         -> -<az> 
• IP addresses             -> <ip> 
• leftover numbers       -> <n> 
• collapse whitespace

No model. No API call. No tokens. Just regular expressions and a stubborn refusal to treat two identical problems as different. A few real ones, straight out of the tool (only the volatile bits change; the shape survives):

Raw alert title: [FIRING:1] eu-com-1 - K8s Deployment under HR atlas-teleport-app in namespace debug-teleport Down

Signature: [firing:1] <env> - k8s deployment under hr atlas-teleport-app in namespace debug-teleport down

Raw alert title: us-com-1 Config Generator dependent service(s) failure alert (failed vs total requests) - us-com-1 : Requests to Config Generator has failed

Signature: <env> config generator dependent service(s) failure alert (failed vs total requests) - <env> : requests to config generator has failed

The payoff

On the real backlog, roughly eighteen hundred alerts collapsed to about two hundred signatures. Close to ninety percent of the volume was duplication, gone, for the price of some regex and an afternoon. Every later step, the embeddings, the vector search, the ranking, now runs over two hundred distinct things instead of eighteen hundred noisy ones. Cheaper, faster, and the "memory" actually reads like memory instead of a stuck printer.

The part where I was too clever

Regex does exactly what you tell it, which is precisely the problem. Two moments I remember:

One of my rules split a field on a slash to pull out a component, and happily produced a phantom component called `A` from titles containing `N/A`. Meaningless, but now the system "knew" about a service that didn't exist.

And a matcher meant to recognise one product line kept swallowing a different one whose name was a prefix of it, quietly mislabelling a whole category of alerts. Both traced back to the same blind spot: the rules are only as good as the messiness you actually looked at, and I hadn't looked hard enough. I fixed them by reading more real titles, not by writing cleverer patterns.

What it taught me

Subtraction is a feature. The cheapest, most durable, most testable part of this whole system is the one with no intelligence in it, and it's the part I'd port first if I rebuilt everything else from scratch. Before you reach for a model, spend a while deciding what the machine is allowed to ignore. Most of the value is there, waiting, for free.

Next time: what happens when the noise is gone, and you want a new alert to find its old cousins, even when nobody worded them the same way. That's where the embeddings come in, and where I start feeling like I'm getting away with something.