On The Other Side Of The Rainbow

Oncall-agent, part 3. Part 2 was about teaching the thing what to ignore. This one is about the opposite: teaching it to notice that two problems nobody worded the same way are, underneath, the same problem. This is the part where I start to feel like I'm getting away with something. In the very first post, I let one sentence slip that I've been chewing on ever since. I said that you take a sentence about a broken DNS pod, it becomes fifteen hundred numbers, and the geometry of those numbers somehow knows this incident is a cousin of one from four months ago; and that even though I know how it works, it still feels like something I shouldn't be allowed to do on a laptop.  I want to pull on that thread, because it's the whole trick of this post, and because the feeling turns out to be pointing at something real. The problem signatures couldn't solve Part 2 had gotten me further than I expected, for free. Eighteen hundred noisy alerts had collapsed into a couple hund...

On-call agent, part 2. The first one was the story; this one goes under the hood, starting with the least glamorous piece, which is also the one that mattered most. Here's a thing nobody tells you about building something with "AI" in it: the part that does the most work often has no AI in it at all.  I spent a while, before any model, before any database, on a function so dumb I hesitated to commit it. It takes an alert and throws most of it away. And it turned out to be the load-bearing wall of the whole project. The problem: a hundred alerts, one problem Our alerts are machine-generated, which means they are precise, and precision is the enemy here. The same underlying failure shows up worded almost identically across every cluster it happens in, differing only in the parts that don't matter. Here's one real alert as it fires in three different environments: ddi-prd-nva1 ipsec Pod not ready alert - ddi-prd-nva1: ipsec Pods are not ready for more than allowed ...

I'm writing this at the tail end of a rotation, which is probably the only honest time to write about one. There's a particular kind of tired that comes from being paged at odd hours: not the clean exhaustion of a long day, but something stranger, because the hours are never quite yours. Our on-call is twelve hours at a stretch, 8 to 8, seven days in a row when it's your turn. And the team lives on opposite sides of the planet, one half in PST, the other in IST. So the pager is really a baton, and by the time it reaches you it's always some odd hour, and it's always still warm from someone else's hands. That handoff is where things go to die. Someone in IST spends an hour chasing an alert: checks a dashboard, tries a restart, rules a thing out, finds the real cause just as their day ends. Then PST wakes up to the same alert firing again, and almost none of that hour survives the crossing. Maybe a one-line note. Maybe nothing. So PST starts over: same dashboard, ...

Setting the Stage: The Cisco C881-K9 Router I wanted to document my experience configuring a Cisco C881-K9 router to set up an IPSec VPN tunnel—not just for myself but also for anyone who might be venturing down this path. Whether you're tackling this for the first time or just need a refresher, I hope this helps. 1. Product Code: C881-K9 2. PoE: 2 port integrated PoE 3. RAM: 256 (default) / 768 MB (Max) 4. Flash Memory: 128 MB 5. Dimensions: 32.5 cm x 24.9 cm x 4.4 cm 6. LAN Interface: 4 x 10Base-T/100Base-TX - RJ-45 7. Management Interface: 1 x console - RJ-45 8. WAN Interface: 1 x 10Base-T/100Base-TX - RJ-45 9. USB: 1 x 4 PIN USB Type A 1. Primary WAN port—FE 2. USB port 3. 4-port 10/100 Ethernet switch 4. Serial port—Console or auxiliary 5. On/Off switch 6. Reset button 7. Power connector 8. Earth ground connection 9. Kensington security slot Why the Cisco C881-K9? I’ll admit, I didn’t plan to splurge on high-end networking gear for my tests. So, when I found a used Cisco C881-...

A few weeks back, while working on Twinstrata project (as a shadow resource), I had some spare time and I wondered if I could provide an SMS notification (in addition to email notification) whenever Nagios detected that the services on the production cluster are down. Hence, I started searching different communication libraries and found Gammu, an open source mobile phone communication library. Below are the steps I took to install Gammu on Linux (CentOS 7) and generate SMS notifications to the development team through it.  First, I had to find a spare mobile phone that could be used for this purpose and a SIM card. I had one Samsung GT-S3653W mobile phone which I wasn't using anymore ever since I bought the smartphone. One of the reasons why I chose Gammu instead of any other library (e.g Gnokki) was their database that indicated that GT-S3653 ( http://wammu.eu/phones/samsung/5152/ ) was compatible with this library. For installation, I downloaded Gammu (version 1.34.0) t...