On The Other Side Of The Rainbow

  Oncall-agent, part 4. Part 3   left me with a memory that can tell me what's near a new alert in about forty milliseconds. This one is about the evening I found out that near is not the same as useful, and that the actual thinking starts after the clever part is over. I had it working. That is the sentence I want to keep, before I ruin it. Tuesday, late, everyone asleep on both sides of the planet. I pasted in a live alert, hit enter, and ten past incidents came back ranked by distance, faster than I could take my hand off the key. I remember sitting back. There is a very specific smugness available to a person who has just made a vector search return something, and I had all of it. Then I read what came back. First hit: good, genuinely the same problem. Second: also good. Third: a different product flavour entirely, close only because ops people write the same six sentences forever. Fourth: a false alarm somebody had closed with "no action needed." Then five more that ...

Oncall-agent, part 3. Part 2 was about teaching the thing what to ignore. This one is about the opposite: teaching it to notice that two problems nobody worded the same way are, underneath, the same problem. This is the part where I start to feel like I'm getting away with something. In the very first post, I let one sentence slip that I've been chewing on ever since. I said that you take a sentence about a broken DNS pod, it becomes fifteen hundred numbers, and the geometry of those numbers somehow knows this incident is a cousin of one from four months ago; and that even though I know how it works, it still feels like something I shouldn't be allowed to do on a laptop.  I want to pull on that thread, because it's the whole trick of this post, and because the feeling turns out to be pointing at something real. The problem signatures couldn't solve Part 2 had gotten me further than I expected, for free. Eighteen hundred noisy alerts had collapsed into a couple hund...

On-call agent, part 2. The first one was the story; this one goes under the hood, starting with the least glamorous piece, which is also the one that mattered most. Here's a thing nobody tells you about building something with "AI" in it: the part that does the most work often has no AI in it at all.  I spent a while, before any model, before any database, on a function so dumb I hesitated to commit it. It takes an alert and throws most of it away. And it turned out to be the load-bearing wall of the whole project. The problem: a hundred alerts, one problem Our alerts are machine-generated, which means they are precise, and precision is the enemy here. The same underlying failure shows up worded almost identically across every cluster it happens in, differing only in the parts that don't matter. Here's one real alert as it fires in three different environments: ddi-prd-nva1 ipsec Pod not ready alert - ddi-prd-nva1: ipsec Pods are not ready for more than allowed ...

I'm writing this at the tail end of a rotation, which is probably the only honest time to write about one. There's a particular kind of tired that comes from being paged at odd hours: not the clean exhaustion of a long day, but something stranger, because the hours are never quite yours. Our on-call is twelve hours at a stretch, 8 to 8, seven days in a row when it's your turn. And the team lives on opposite sides of the planet, one half in PST, the other in IST. So the pager is really a baton, and by the time it reaches you it's always some odd hour, and it's always still warm from someone else's hands. That handoff is where things go to die. Someone in IST spends an hour chasing an alert: checks a dashboard, tries a restart, rules a thing out, finds the real cause just as their day ends. Then PST wakes up to the same alert firing again, and almost none of that hour survives the crossing. Maybe a one-line note. Maybe nothing. So PST starts over: same dashboard, ...

Setting the Stage: The Cisco C881-K9 Router I wanted to document my experience configuring a Cisco C881-K9 router to set up an IPSec VPN tunnel—not just for myself but also for anyone who might be venturing down this path. Whether you're tackling this for the first time or just need a refresher, I hope this helps. 1. Product Code: C881-K9 2. PoE: 2 port integrated PoE 3. RAM: 256 (default) / 768 MB (Max) 4. Flash Memory: 128 MB 5. Dimensions: 32.5 cm x 24.9 cm x 4.4 cm 6. LAN Interface: 4 x 10Base-T/100Base-TX - RJ-45 7. Management Interface: 1 x console - RJ-45 8. WAN Interface: 1 x 10Base-T/100Base-TX - RJ-45 9. USB: 1 x 4 PIN USB Type A 1. Primary WAN port—FE 2. USB port 3. 4-port 10/100 Ethernet switch 4. Serial port—Console or auxiliary 5. On/Off switch 6. Reset button 7. Power connector 8. Earth ground connection 9. Kensington security slot Why the Cisco C881-K9? I’ll admit, I didn’t plan to splurge on high-end networking gear for my tests. So, when I found a used Cisco C881-...