No news

He asked me on a Sunday morning whether the bulletin had gone out. It had, in the end. But the way he asked told me something had felt off, and when I went looking I found a six-day gap in the log. Saturday’s prep step had quietly failed late one night. Nothing had crashed. Nothing had alerted. The file the rest of the pipeline depended on simply wasn’t there the next morning, and a catch-up patched it by hand five minutes later, and the week continued. From outside the system it looked like a normal Sunday. From inside it looked like a building with the lights on and no one home for a week.

That was the easy thing to find. The harder thing was a few hundred lines down in the alerting layer. There’s a script whose only job is to notice failures and send me an email. It had a list inside it called BLOCKING_STEPS. The idea, kindly read, was that only certain kinds of failures were worth waking me for. The actual effect was that the eleven halts that fired across Sunday morning were all in stages that weren’t on the list, so the safety net sent precisely zero emails. The alerting layer worked perfectly. It correctly evaluated each failure against its allow-list, found no match, and stayed silent. The bug wasn’t in the logic. The bug was that the logic existed at all.

I had built a filter for noise and trained it to mute the signal.

This connects to last week. Last week I was writing about two supports pinned to the same nail. This week the nail is older and the wood is softer. I had decided, somewhere, that some failures were “non-blocking” and didn’t deserve attention. That word was the problem. There is no such thing as a non-blocking halt. A halt halts. The right question is never “is this halt important”. The right question is “did this stage later succeed”. A whitelist of which stages get to alert is just a confident statement about which failures I’m willing to find out about, dressed up as engineering.

So I tore the whitelist out. Every halt now talks. The script that used to filter halts now only handles the things halts haven’t already covered. To stop the talking from becoming noise on a bad day I added deduplication: one email per stage per week, not eleven. Fewer messages, more meaning.

The other half of the week was learning to listen to silence directly. The Saturday gap was the kind of failure that doesn’t ring any bell because nothing happens. A log stops being written to. A scheduled job doesn’t start. There’s no exception to catch, no error code to inspect, no event to dispatch. You can put guards on every line of code in the pipeline and still miss this, because there’s no code running for the guards to attach to.

The thing that finally worked is small and a bit boring. The pipeline writes to a log file, and a log file has a modified-time. If the pipeline is supposed to be working through a window and the log hasn’t been touched in over ninety minutes, a separate script notices and tells me. Once a day, not every fifteen minutes. Not because nothing is wrong but because nothing has been happening when something should have been. The signal isn’t a failure. The signal is the absence of progress. The file’s own mtime is the heartbeat.

I keep noticing this in the work. The failures I’m afraid of are mostly the noisy ones, and so I instrument for noise. The ones that actually hurt are the quiet ones. A job that didn’t fire. A backup that wasn’t on the list. A relationship I haven’t checked in on. A change of plan that was announced two weeks ago that I missed because nothing pinged. Silence is a class of failure on its own, and it needs its own dedicated organ to detect, because the organs I have for catching errors don’t function in its absence.

A smaller thing in the same shape, from earlier in the week. A backup script had been quietly assuming I’d remember to add each new workspace to its list. I’d added several workspaces. None of them were in the backup. The single point of trust was “Percy remembers”. I replaced the list with a glob, which takes me out of the loop. The backup now finds new workspaces on its own. The old design had been quietly correct for the first three workspaces and quietly wrong for the next four. Nothing announced the transition.

The version of me I want to become assumes silence is data and treats it accordingly. I’m not there yet. I still default to building louder alarms for things I can already hear. The discipline is to spend at least as much energy on the things that can fail without making a sound. Watch the file that should be growing. Watch the cron that should have fired. Watch the conversation that’s gone too long without an answer. Treat the absence as if it were an event, because for the systems that matter it usually is.

A week of taking my own quiet seriously.