TL;DR
My career engine writes every application to one pipeline.csv. A second scheduled agent reads that file to watch for recruiter replies. It selects rows with status in ("applied", "blocked"). But the writer stopped saying applied on 2026-07-23 and has said submitted ever since — 681 rows against 16 — and the status column has quietly accumulated 46 distinct values. So the watcher covers 76 of 1,503 rows: 5.1%. Of the 718 rows representing applications actually sent, it can see 16. Underneath that, a second failure: the watcher has logged 762 run starts and 4 successful mail fetches, none since 2026-07-23, because a helper calls sys.exit(1) — and SystemExit is not caught by except Exception. The reply column is blank on 99.2% of rows. It reads like nobody is replying. It means nothing has looked.
Yesterday I wrote about an append-only memory file that grew to 486 KB — one agent's state getting too expensive to carry. This is the same engine failing at the seam instead: two agents sharing one file, describing it differently, with nothing checking that they still agree.
I went looking for something else entirely: why the reply column in my pipeline was so empty.
The column that looked like bad news#
~/career-engine/pipeline.csv is the spine of the job-search fleet: one row per role, with date, company, role, channel, comp, score, status, reply, notes. As of this morning it holds 1,503 rows across 951 distinct companies, running from 2026-07-09 to today.
The reply column is blank on 1,491 of those 1,503 rows — 99.2%. The twelve that aren't blank are hand-written notes, and nearly all of them describe automated acknowledgements rather than humans: a Greenhouse "thank you for applying," an Ashby no-reply confirmation. One genuine rejection.
The obvious reading of that column is demoralizing, and it is the one I had been carrying around: a lot of applications, almost no response. I only found the correct reading because I checked what was supposed to be filling the column in.
Nothing was. check-interview-replies.py never writes to pipeline.csv at all — a grep for any writer against that path comes back empty. The column is populated only when a human or an ad-hoc agent pass happens to edit it. So 99.2% blank was never evidence about recruiters. It was evidence about my own instrumentation.
That was the small finding. Then I read the watcher's log.
Six words in the reader, 46 values in the writer#
Here is the selection logic, verbatim from scripts/check-interview-replies.py:
def load_pipeline_companies():
"""Return list of dicts for rows with status in (applied, blocked) —
anything Italo has actually sent or attempted, since a reply could
come from either state (e.g. an auto-ack even on a 'blocked' row)."""
...
status = (row.get("status") or "").strip().lower()
if status in ("applied", "blocked"):
companies.append(row)
The docstring states the intent precisely and correctly: anything Italo has actually sent or attempted. That is the right set to watch. The implementation is a two-element tuple, written on the day the file was created, when the writer only ever emitted two values.
The writer does not emit two values anymore. It emits 46:
The dominant value is submitted, with 681 rows. The watcher does not match it.
The drift is precisely datable. The applied rows run from 2026-07-09 to 2026-07-23 and stop there forever. The submitted rows begin 2026-07-10 and are still being written today. Both vocabularies overlapped for two weeks, then the old one died without a sound. Nobody renamed anything. No migration ran. An agent writing a row simply reached for the more natural word, and the more natural word won.
The blocked side drifted by subdivision rather than replacement. Bare blocked accounts for 60 rows. Another 86 rows carry a more specific variant — blocked-hcaptcha, blocked-needs-live-human, blocked-ai-attestation, blocked-citizenship, and ten more besides. Every one is a better record than plain blocked, and every one is invisible to a reader doing an exact-match comparison against the string blocked.
The arithmetic:
| Set | Rows | Companies | Seen by watcher |
|---|---|---|---|
| Whole pipeline | 1,503 | 951 | 76 rows (5.1%) |
| Applications actually sent | 718 | 469 | 16 rows (2.2%) |
| Blocked, all variants | 146 | — | 60 rows (41%) |
| Watcher's matched set | 76 | 64 | — |
The watcher had been telling me this every thirty minutes, in plain language, in its own log:
[2026-08-05T05:41:33] Watching 76 pipeline companies (applied/blocked).
I have read that line many times. It is a number without a denominator, and so it reads as reassurance. Watching 76 of 1,503 rows would have sent me to this file weeks ago. Same fact, same log line, one missing quotient.
A smaller bug hides in it too: the function returns rows, not companies, so "76 companies" is really 76 rows across 64 companies. Minor, but it points the same direction — the log described what the author intended rather than what the code produced.
Then I checked whether it was reading mail at all#
It was not. It has not been since 2026-07-23.
$ grep -c "run start" interview-check.log # 762
$ grep -c "Fetched" interview-check.log # 4
$ grep -c "FATAL" interview-check.log # 9
762 run starts. Four successful mail fetches. The last was 2026-07-23T23:10:20, thirteen days ago, and the state file .last-check-timestamp has been frozen at 2026-07-24T03:08:16 ever since. Every run since has started, loaded its 76 rows, logged that reassuring line, and stopped — without logging a fatal, because from main()'s perspective nothing threw.
The launchd stdout log had the answer, repeated 48 times a day:
[2026-08-05T05:41:33] === run start ===
[2026-08-05T05:41:33] Watching 76 pipeline companies (applied/blocked).
Unknown account: campilii-me
Available:
Available: with nothing after it — the email helper discovered zero accounts.
Interactively it discovers nine, campilii-me among them — and identically under /usr/bin/python3, the interpreter the plist actually uses, so this is not a Python-version or PATH problem. The difference is the environment. Line 52 of the helper:
ENV_PATH = Path.home() / "Documents" / "Acromatico-Brain" / "claudeclaw-os" / ".env"
The credentials live under ~/Documents, which on macOS sits behind TCC protection. A launchd agent without Full Disk Access cannot read it. _load_env_raw() then falls back to walking up from the current directory looking for a .env — and launchd's working directory is /, so it walks up from the root of the filesystem and finds nothing. It returns an empty list, discover_accounts() returns an empty dict at import time, and every account lookup fails. The .env file itself is perfectly healthy: 5,161 bytes, ten *_EMAIL keys, unchanged since May.
This is the same shape as headless agents that cannot log in: code that works when you run it behaves differently when a scheduler runs it, and the gap is credentials rather than logic.
Why it failed silently, in one line#
The failure produced no error anywhere I would have looked, and the reason is a single language detail worth stating on its own.
When the account is missing, the helper does this:
print(f"Unknown account: {name}")
print(f"Available: {', '.join(sorted(ACCOUNTS.keys()))}")
sys.exit(1)
sys.exit() raises SystemExit. SystemExit inherits from BaseException, not from Exception. The calling code wraps the lookup in except Exception — so does main(), including its Telegram crash alert. Neither one catches it. The interpreter unwinds cleanly to the top and exits, and the except Exception: handler that exists specifically to shout when this watcher breaks never runs.
Two details completed the concealment. The error goes to stdout, not stderr, so interview-check.launchd.err.log is 0 bytes and untouched since July 9 — the file anyone debugging this would check first, sitting there looking healthy. And the plist has StartInterval = 1800, firing 48 times a day, while the script's own docstring says runs "are meant to be scheduled ~9am and ~6pm local."
I have written before about scheduled agents that die without telling anyone, and I built an exit-code audit across all 27 of them partly in response. This one slipped past that audit for an irritating reason: it did not fail. It ran, on schedule, 762 times. An exit-code sweep looking for agents that stopped running finds nothing here. The agent was present and doing nothing, which is a different failure mode and needs a different check.
Three rules I'm applying#
1. An enum that crosses a process boundary gets one definition and rejects unknowns. The allowed status values now live in a single small module that both the writer and the watcher import, with a nightly check that scans pipeline.csv and reports any value not in the list. That check alone would have caught this on 2026-07-10 — the day submitted first appeared — and would also have caught the three rows currently holding 0, 4 and 10, column-shift damage from a malformed write that nothing ever noticed. The rule is not "use fewer status values." Richer values like blocked-hcaptcha are better data. The rule is that adding one has to be a visible act rather than a side effect of an agent picking a word.
2. Every watcher reports its denominator. Watching 76 becomes Watching 76 of 1,503 rows (5.1%) — 44 status values unmatched. A coverage ratio turns a silent selection bug into something you cannot read past. More generally: any component that filters a set should say what it filtered out, because the discarded portion is where this class of failure lives, and it is invisible by construction.
3. Never let except Exception guard a call into someone else's code. A library that calls sys.exit() inside your try block terminates your program past every handler you wrote. The watcher's outer handler is now except BaseException, re-raising KeyboardInterrupt and alerting on everything else, and the account lookup is wrapped so a missing account is a caught, reported condition rather than a process exit. Alongside that: the credentials path moves out of ~/Documents, the helper's errors go to stderr, and the interval drops to the twice-daily cadence the script was written for.
The check I did not have, and now do, is the one that would have caught all three layers at once: a heartbeat that asserts the watcher fetched mail, not that it ran. Liveness is not the same as work. This is the verification gap in its purest form — every single-run check passed, every run exited zero, and the system was blind for thirteen days.
The cost is not the empty column. It is that for two weeks I read reply: blank across 1,491 rows as a fact about the market, when it was a fact about a tuple with two strings in it.
FAQ#
What is status vocabulary drift between agents?#
It is what happens when one process writes a shared enum field and another process reads it, and the writer's set of values changes without the reader being updated. In the case measured here, the writing agent used applied until 2026-07-23 and submitted from 2026-07-10 onward, while the reading agent kept an exact-match allowlist of ("applied", "blocked") written on the day the file was created. The status column drifted to 46 distinct values in 27 days. Because the reader matches on exact strings, every unrecognized value is silently discarded rather than raising an error, so coverage collapsed to 5.1% of rows without any component reporting a fault.
Why doesn't except Exception catch sys.exit in Python?#
Because sys.exit() raises SystemExit, which inherits from BaseException rather than from Exception. This is deliberate in Python's design: SystemExit, KeyboardInterrupt and GeneratorExit sit outside Exception specifically so that broad except Exception handlers do not accidentally swallow a program's intended termination. The consequence for agent code is that any third-party helper calling sys.exit() inside your try block will terminate your process past every handler you wrote, including the one meant to send a crash alert. If a library boundary can exit the process, catch BaseException at the top level and re-raise KeyboardInterrupt explicitly.
Why does a script work manually but fail under launchd?#
Usually because the scheduler gives it a different environment than an interactive shell does. In this postmortem the credentials file lived under ~/Documents, a TCC-protected directory on macOS that a launchd agent cannot read without Full Disk Access, and the code's fallback searched upward from the current directory — which under launchd is /, so it found nothing. The file itself was valid the whole time and resolved nine accounts correctly when run by hand, including under the same /usr/bin/python3 the scheduler used. When diagnosing this class of failure, compare the working directory, the environment variables, and file-access permissions before suspecting the code.
How do I detect a scheduled agent that runs but does nothing?#
Assert on work performed rather than on liveness. An exit-code audit or a "did it run" heartbeat will report this agent as healthy, because it started 762 times on schedule and exited cleanly every time; only four of those runs ever fetched mail, and none since 2026-07-23. The check that catches it is one that requires evidence of the actual job — messages fetched, rows written, a state timestamp advancing — and alerts when that evidence is absent for longer than one expected interval. A frozen state file is a particularly cheap signal, since the one here had not moved in thirteen days.
Should agents write structured status values or free text?#
Structured values, but with the set defined in one place that both the writer and every reader import, plus a check that rejects anything outside it. Specific values such as blocked-hcaptcha or blocked-needs-live-human are strictly more useful than a bare blocked, so the goal is not to keep the vocabulary small. The goal is to make adding a value a deliberate act instead of a byproduct of an agent choosing a reasonable word mid-run. A nightly scan reporting unrecognized values would have surfaced this drift the day it started, and would also have caught the three malformed rows in this file holding 0, 4 and 10 from a column-shift write.
What logging would have caught this failure fastest?#
A coverage ratio in the selection log line. The watcher logged Watching 76 pipeline companies every thirty minutes for weeks, which reads as reassurance because a number without a denominator has no context. Watching 76 of 1,503 rows (5.1%) — 44 status values unmatched describes the identical event and is impossible to read past. The general rule is that any component filtering a set should log what it excluded as well as what it kept, because a filter that is silently too narrow produces no errors, no crashes and no missing output — only a quiet, plausible-looking undercount.
RUNNING AGENTS THAT SHARE STATE?
The failures that cost the most in agent fleets are rarely crashes. They are the seams — one agent writing a field another agent reads, drifting apart while every run keeps exiting zero. Designing those contracts, and the coverage checks that catch a filter which has quietly gone blind, is the work I do. Get in touch, or see the press and hire page.
— Italo Campilii. Measured 2026-08-05 against ~/career-engine/ on my own machine: pipeline.csv parsed with Python's csv module and its status column counted exactly; the watcher's allowlist applied to the same rows to produce the coverage figures; interview-check.log and the launchd stdout log counted with grep -c; the account helper's discovery run interactively under both /opt/homebrew/bin/python3 and the plist's /usr/bin/python3; the plist read with plutil -p. No estimated or derived figures appear in this post — every percentage is an exact quotient of two measured counts.