My Agent Failed Safely for Nine Days Straight. Nobody Told Me.

I want to write this one while the evidence is still sitting in the log directory, because it is the most embarrassing kind of failure: the system worked precisely as specified, and the specification was wrong.

Here is the shape of it. I run a daily engine that writes and ships one post to this site — it pulls its own analytics, picks a topic, writes, builds, validates the render in a headless browser, deploys, pings IndexNow, and writes a state file. It fires at 6AM under launchd. It's one of many scheduled agent runs across the brands I operate solo, and the one whose output should be easiest for me to see, because the output is this blog.

Between July 21 and July 29, it published nothing. Nine days. I found out by opening the log directory for an unrelated reason.

The evidence

Every one of those nine run logs ends the same way. Same three lines, nine days running:

Invalid API key · Fix external API key
claude exit=1
NO-POST: engine did not ship today (check decision log). Safe exit.
=== end ...

That is the whole incident. The scheduler fired on time. The wrapper resolved its PATH, sourced its secrets, pulled analytics. Then the writer failed to authenticate, returned non-zero, and the wrapper did what I told it to do: checked whether a post had actually shipped, saw that it hadn't, logged the fact in plain language, and exited cleanly.

Three details make this worse than a normal outage, and they're why I'm writing it up rather than quietly fixing it.

One: the state file was never written, so nothing was corrupted. My hard one-post-per-day gate compares today's date against engine/state/last-post-date, and that file is only written on full success. So across nine failed runs the gate stayed correct — no partial post, no duplicate. The safety machinery was flawless. It just had nothing to say to anyone.

Two: the log it told me to check did not exist. Look at that last line again — "check decision log." The decision log is written by the writer, as its first real step, and the failure happened before the writer ran. So the wrapper spent nine days pointing me at a file that was never created. A diagnostic that assumes the failing component is alive is not a diagnostic.

Three: the exit code was 0 every time. Deliberately — a scheduled agent that exits non-zero on a normal "nothing to do today" outcome trains you to ignore your own alarms. That instinct is correct. My implementation wasn't, because I collapsed two very different outcomes — nothing to do and could not do it — into one clean exit and no notification.

Why this is the interesting failure mode

I've written a postmortem here about agents that died silently under launchd because of a PATH bug. That one was conceptually simple: the process never ran. There was an absence to detect, and once I knew to look, it was easy to detect.

This is the harder cousin. The process ran, on time, every day, and it made the right call every day. No crash, no stack trace, no missing heartbeat, nothing red. From the outside — and "the outside" includes me, at 6AM, not looking — a fleet that safely declines to work is indistinguishable from a fleet with no work to do.

The industry has caught up to this in 2026 and given it a name. The reported figure I keep seeing is that a large share of production agent failures — on the order of 40% — are silent: the task completes, the exit is clean, and the quality or the outcome is quietly below threshold (Sherlocks.ai). The structural point in the observability literature is sharper than the number, and it's the sentence I'd staple to my own wall: an observability failure has no mean-time-to-detect, because by definition it has no detection mechanism (Braintrust, 2026). Traditional monitoring shows you a 200 or an exit 0. It cannot show you that the agent looped, chose the wrong tool, or politely refused (Confident AI).

I'd add a category those write-ups mostly skip, because it only shows up when you build conservatively: the safe-exit blind spot. Not a bad output. Not a crash. A correct refusal that nobody hears.

The guard-rail paradox

Here's the part that generalizes past my little publishing engine, and it's the reason this is worth a full write-up instead of a one-line fix.

I build agents to fail closed. That's the doctrine — when a check fails, stop, don't deploy, don't write state, log why, exit clean. It's the core of my safe-rollout playbook for letting agents ship and it has saved me from shipping broken work more times than I can count. I'd make the same call again tomorrow.

But every guard rail you add is a new exit path out of the workflow. And an exit path that produces no signal is, from an operations standpoint, a place where work goes to die quietly. So the more carefully you build, the more silent-failure surfaces you create:

Every one of those five is a design decision I stand behind, and every one contributed to nine quiet days. Conservatism doesn't remove failure modes; it converts loud ones into quiet ones. That's usually a good trade — but only if you pay the other half of the price, which is emitting a signal on the way out of every exit path.

What "nine days" actually cost

I want to be precise rather than dramatic, because inflating the damage is its own kind of dishonesty.

What I can verify from the repo: nine scheduled runs produced no post, and the site's freshest post sat at July 20 while I believed it was daily. What I cannot verify — and won't invent — is a traffic or ranking figure attached to that gap. The site's own analytics for the window are too thin for that: over the fourteen days ending today, the entire dataset is ten views on the homepage and no referrers at all.

Which, honestly, is the second finding of this postmortem. The engine is analytics-gated by design — it reads its own traffic before it writes a word, so it can aim at something real. When the traffic signal is that thin, the gate has nothing to aim with and falls through to its secondary path. I built a system that measures itself, and then discovered the measurement was too sparse to notice a nine-day hole in its own output. Measurement debt compounds in the dark.

The real cost isn't the nine posts. It's that I had a mental model — "the engine ships daily" — and no mechanism binding that model to reality. That gap is the actual defect. The auth error was just the thing that happened to walk into it.

The fix is a detection layer, not a root cause

Here's where I have to be careful, because the satisfying ending would be "and then I found the root cause," and that's not what happened.

The proximate error is an authentication failure in the headless writer. The wrapper already tries to handle exactly this: it explicitly unsets a stale API key from the shared environment file before invoking the writer, precisely because that key shadows the login the headless process should be using. That mitigation is in the script, and the failure happened anyway — which means the key is being reintroduced by something the wrapper doesn't control, and I have not yet confirmed what. I'm not going to guess in public about a root cause I haven't proven.

And that's the point. The root cause is not the finding. The finding is that the root cause could hide for nine days. If I fix the auth path and ship nothing else, I've fixed one bug and left the blind spot intact — ready for the next thing that walks into it, which will be something I haven't thought of, because that's how the next one always works.

So the fix I actually care about has three parts, none of them about auth.

1. A safe exit must page, not just log

The NO-POST branch currently writes a line to a local file. Local logs are read by people who already suspect something. The branch has to push — a notification I'll see without going to look. And the message has to carry the diagnosis with it, not a pointer to a file that may not exist. That last part is the specific bug in my "check the decision log" line: never send a diagnostic to a location that the failing component was responsible for creating.

2. Freshness is the health check, not exit code

Exit codes describe processes. What I actually care about is outcomes. The right monitor for a daily publishing engine doesn't watch the process at all — it watches the artifact and asks one question: how many days since the last successful post? If that number exceeds one, something is wrong, no matter how cleanly every process exited.

That inversion is the whole lesson, and it's the same discipline as the verification gap I wrote about in agents going to production: don't verify that the agent ran, verify that the work exists. A process-level check would have shown nine green days. An outcome-level check would have fired on July 22.

Three green lights and one red, and only the red one measured the thing I actually wanted.

3. Distinguish "nothing to do" from "couldn't do it"

The reason I wanted exit 0 was to avoid alarm fatigue on routine no-ops. Still right — but the two outcomes need different codes and different volumes. Already posted today is a no-op: exit 0, log it, stay quiet. Tried and couldn't is an incident: exit non-zero, notify immediately, and say what failed in the notification body. Collapsing those into one silent path cost me the nine days. They look similar in a shell script and are nothing alike in an operation.

The rule I'm taking forward

Every scheduled agent I run gets audited against one question now, and it isn't "what happens when this breaks." It's:

When this agent decides not to do its job, who finds out, how, and how fast?

If the answer is "a line in a log file," the answer is nobody. A guard rail that stops bad work and tells no one has converted a visible failure into an invisible one, which feels like safety and operates like rot.

Failing closed is correct. Failing closed quietly is a bug — in the operator's design, not the agent's behavior. My engine did its job for nine days: refused to publish work it couldn't verify, protected its own state, and told the truth in its log every morning. The only thing missing was somebody listening.

FAQ

What is a silent agent failure?

A silent agent failure is one that produces no alert, no crash, and no obvious red signal — the process runs, the exit is clean, and the work quietly doesn't happen or doesn't meet the bar. Industry write-ups put a large share of production agent failures in this category. The dangerous subtype for anyone building conservatively is the safe-exit blind spot: the agent correctly refuses to act, logs why, exits 0, and nobody hears it.

Why is exit code 0 a bad health check for an AI agent?

Because an exit code describes whether a process finished, not whether the work got done. An agent that authenticates, plans, and publishes exits 0. An agent that fails to authenticate and safely declines to publish can also exit 0 — that's usually intentional, to avoid alarm fatigue on routine no-ops. Every layer above the process reads both as healthy. Monitor the outcome (does today's artifact exist?) instead of the process.

How should a scheduled agent report that it chose not to run?

Split the two cases. A genuine no-op — "already did this today," "nothing new to process" — should exit 0 and stay quiet. A blocked attempt — "a check failed, so I stopped" — is an incident: exit non-zero and push a notification that contains the diagnosis in the message body. Never point the alert at a log file that the failing component was responsible for creating, because if that component died early, the file won't exist.

Doesn't adding alerts to every guard rail cause alarm fatigue?

Only if you alert at the same volume for everything. The distinction that keeps it manageable is intent versus obstruction: an agent skipping work because there's nothing to do is routine and should be silent, while an agent skipping work because something blocked it is an incident and should be loud. Fatigue comes from undifferentiated noise, not from notifying on real blocks.

What's the single best monitor for a daily automated publishing pipeline?

Days since the last successful output. It's one number, it's derived from the artifact rather than the process, and it catches every failure mode at once — auth errors, validation failures, a stale gate, a scheduler that never fired — without you having to enumerate them in advance. If it exceeds your publishing interval, something is wrong regardless of how green the process metrics look.

— Italo Campilii. If you're building something that needs this kind of operator, get in touch.