Every Way My AI Agent Fleet Has Failed (A Running Log)

I write a postmortem every time something in my agent fleet breaks in a way worth a permanent fix. Thirteen of them exist now, scattered across a year of running infrastructure for five brands solo. Scattered is the problem — a reader (or an AI engine trying to answer "what actually goes wrong with agent fleets") has to find each one separately. This post is the fix: a taxonomy of the failure classes, and one paragraph per incident with what broke and what changed afterward. Every claim below traces to a full postmortem already published on this site — nothing here is new data, only the index.

The taxonomy#

Five failure classes cover all thirteen incidents:

1. Silent exit — a scheduled job stops running, or refuses safely, and nothing tells you. The system's health signal (exit code, log presence) says fine while the actual output stopped.
2. Drift — two parts of a system that used to agree quietly stop agreeing: a status vocabulary, a search-intent classification, a shared script that got copied instead of imported. Nobody changed anything on purpose; it just diverged.
3. Provenance gap — an agent states something as verified fact — a supplier, a product spec, a photo — that was never actually checked against a source. The failure isn't the hallucination itself; it's that nothing caught it before it shipped.
4. Measurement blindness — the tracking layer itself is broken, so every downstream decision is being made on a zero that looks like real data instead of like an instrument failure.
5. Code and context sprawl — the fleet keeps writing new one-off scripts or letting a memory file grow instead of maintaining shared, bounded infrastructure, so a fix in one place never reaches the other copies.

Silent exit#

My Agents Died Silently for Days: A launchd PATH Postmortem — A launchd job fired every night, wrote its log header, and died at line 75 for days, because launchd runs jobs with a minimal environment (no shell profile, no PATH entries a terminal would have) and the script called a binary from ~/.local/bin that wasn't on it. The fix: set EnvironmentVariables explicitly in every plist, and never trust that "it works in the terminal" means it will work under launchd.

My Agent Failed Safely for Nine Days Straight. Nobody Told Me. — A daily publishing agent hit an auth error, correctly refused to publish, logged one honest line, and exited 0 — which reads as healthy to every layer above it. Nine days of not-shipping produced zero alerts. The fix: a safe exit is still an incident, and every guard rail needs its refusal wired to page someone, not just log quietly.

One Command Audited 27 Scheduled Agents. Seven Had Failed, and One Never Ran at All. — A two-minute launchctl sweep across 640 loaded jobs found 27 belonging to this fleet, 7 with a non-zero last exit code, and one blocked by macOS file permissions that had been firing on schedule and executing nothing. The fix: a recurring exit-code audit, not a one-time check, because jobs that pass today can silently start failing tomorrow.

Drift#

My Pipeline Has 1,503 Rows. The Watcher Reading It Sees 76. (full postmortem) — A career-pipeline writer stopped using the status value applied and switched to submitted on a specific date, and a downstream watcher agent kept filtering for the old value — 681 rows written under the new term against 16 under the old, with the status column quietly accumulating 46 distinct values total. The fix: status vocabularies need a single enforced enum, not an implicit contract between two agents that never talk to each other.

My AI Agent Fleet Post Is Ranking for Truck Fleets. A Term-Collision Postmortem. — "AI agent fleet" (software) and "AI for fleet management" (vehicles) are different industries sharing a word, and Google was scoring the same page for both — average position 12–32 for the software-sense queries, 33–86 for the vehicle-sense ones. The fix: sharpen the content toward the sense actually being served rather than chasing the higher-volume sense the fleet knows nothing about.

My Agent Wrote 149 Throwaway Scripts in 14 Days. The Bug Fix in Pass #102 Never Reached Pass #145. (full postmortem) — 164 Python files in one agent's working directory, 149 of them run exactly once, 93% of the code written for a single execution, nine groups byte-identical, one helper function redefined 101 separate times — despite a shared library already existing on disk. The fix: a hard rule that new scripts import from the shared library or get deleted, checked at review time, not left to agent discretion.

Provenance gap#

The Agent Invented a Supplier. Provenance Rules Saved the Business. — Two incidents in one postmortem: an invented supplier that nearly reached a compliance submission, and AI-generated "product photos" with garbled brand labels. Both were caught by one thing — a human viewing the artifact before it shipped — which became three written provenance rules rather than a promise to be more careful next time.

The Listing Said 32oz. The Bottle Said 1 Liter. A Product-Truth Postmortem. — A batch of agent-generated Amazon listings shipped with the wrong product size in every title and described aluminum sprayers as plastic, because content was generated before the product facts were verified. The fix: a doctrine that no asset gets generated until a verified PRODUCT-TRUTH file exists, and every generated asset is checked against it before it ships.

The Deliverable Looked Fine in the Terminal. The Customer Saw Something Else. — Agents verify what they can measure — the code ran, the file exists, the page returns 200 — while customers judge what they can see. The fleet failed that second test three documented times: CSS-built book covers that collapsed into empty wireframes at real browser size, blog images depicting the wrong place entirely, and ad assets shipped without anyone actually looking at them. The fix: a customer-eye QA step is mandatory before anything ships, separate from the technical checks agents already run.

Measurement blindness#

I Audited Six Brands and Search Console Was Broken on Five. Measurement Debt Is Real. — Google Search Console was broken or absent on five of six brand properties — one 403, one tracker on the wrong domain variant, three never wired at all — while agents kept publishing regardless. One brand had 144 of 145 tracked keywords returning zero data because its programmatic pages weren't even indexed. The cause was verification drift across accounts; the fix was a recurring cross-brand tracking audit instead of a one-time setup.

I Gated My Content Engine on Analytics. The Analytics Has Been Blind for 42 Days. — A publishing rule required a traffic pull before any agent could write, but fourteen consecutive daily pulls had returned exactly one path (the homepage) and never a single blog URL, against 33 published posts, because the client-side RUM beacon itself was broken. A gate is only as honest as the instrument feeding it, and this one had been reporting a false "all clear" for six weeks before anyone read the whole log series instead of just the day's file.

My Personal Site Was Blocking Every AI Crawler. I Found Out During My Own Audit. — campilii.com, a site whose entire job is being cited when an AI answers "who is Italo Campilii," was serving a Cloudflare-managed robots.txt disallowing ClaudeBot, GPTBot, Google-Extended, CCBot, Applebot-Extended, and Bytespider, plus a Content-Signal: ai-train=no line — found during a routine SEO audit, not by design. The fix lived under AI Crawl Control → Overview, not the Security → Bots card where it would normally be checked.

Code and context sprawl#

My Agent's Memory File Is 486 KB. Every Hourly Run Is Told to Read All of It. (full postmortem) — A BLOCKERS.md file that every hourly agent run is instructed to read in full and append to, with nothing ever removing from it, had grown to 486,652 bytes across 5,158 lines — an estimated 120,000 tokens, roughly 60% of a 200k context window consumed before the agent reads anything else. The fix: append-only memory needs a pruning or archival pass, not just a growing instruction to read more.

What the taxonomy says, together#

No incident above was caught by an agent being smarter or a human being more careful. Every durable fix was one of three things: a written rule an agent must follow before it acts (provenance gap fixes), a gate that blocks output until a condition is machine-checked (silent exit and measurement blindness fixes), or a scheduled audit that re-verifies something instead of assuming a one-time setup holds (drift and sprawl fixes). If there's one lesson that generalizes across all thirteen: agents will faithfully keep doing whatever they were told to do, including doing it wrong, until something outside the agent itself checks the work. See what an AI systems builder actually does and AI systems builder vs. AI operator vs. prompt engineer for where that verification layer fits in the job.

FAQ#

What is the most common failure mode in AI agent fleets?#

Across these thirteen incidents, silent exit and measurement blindness are the most common and the most dangerous, because both produce a false "all clear" signal — a healthy exit code, a gate that looks satisfied — while the actual work has stopped or the instrument reading the work has broken. Drift and provenance gaps are usually caught faster because something downstream eventually looks wrong; silent failures can run for days or weeks unnoticed.

How do you catch a scheduled agent that failed silently?#

A recurring exit-code audit, not a one-time check. The launchctl exit-code audit postmortem found 7 of 27 scheduled jobs with a non-zero last exit and one job that had been firing on schedule while executing nothing, all invisible until someone ran the audit deliberately. Jobs that pass today can start failing tomorrow with no notification unless something is checking on a schedule of its own.

Why does a safe, intentional exit still count as an incident?#

Because "safe" describes the code's behavior, not the business outcome. An agent that correctly refuses to publish on an auth error and exits 0 is behaving exactly as designed — and the underlying problem, whatever caused the refusal, is still unresolved and still unseen. The nine-days-no-alert postmortem is the clearest example: nine days of not-shipping, zero pages, because every layer above the agent read exit 0 as healthy.

What fixes actually stopped these failures from recurring?#

Written rules an agent must satisfy before acting, gates that block output until a condition is machine-verified, and scheduled audits that re-check assumptions instead of trusting a one-time setup. None of the thirteen postmortems above resolved with "be more careful" — every durable fix moved the check out of human vigilance and into something the system itself enforces.

— Italo Campilii. Every incident above is drawn from full postmortems already published on this site, linked in the text.