Why doesn't a global lock solve shared LLM quota exhaustion across scheduled agents?

Because a lock controls the order of access, not the volume of it. A mutex answers the question may I go now; a quota answers the question is there anything left. In the fleet measured here the lock worked perfectly across 129 acquisitions with zero deadlocks, and the weekly model allowance still ran out, because four engines taking orderly turns consume a finite budget just as completely as four engines colliding. Serialization also makes depletion quieter: instead of four jobs failing loudly at the same minute, engines succeed one at a time until the last one in the line hits the wall and looks like an isolated failure.

How do you make a fleet lock's contention log attributable?

Set an explicit caller label at every call site and make the default value noisy. In this system the gate supports a CLAUDE_GATE_LABEL variable, but none of the four production callers set it, so 120 of 129 lock acquisitions and 13 of 18 queued waits were recorded under the fallback label claude-gate.sh-caller. The result is a timeout line reading that claude-gate.sh-caller waited 2,400 seconds for claude-gate.sh-caller, which identifies nobody. The durable fix is two parts: set the label in each caller, and add an assertion inside the lock that logs a warning whenever the label is still the default, since instrumentation that can be skipped silently always is.

What should replace a mutex when agents share a model quota?

Keep the mutex and add a consumption ledger the scheduler reads before it grants access. The ledger is an append-only file recording, at acquisition time, the date, the calling engine, and the model tier of the invocation. The gate sums the current period and refuses the lock once the running total crosses a reserve threshold, so the least important engine defers deliberately instead of the last engine in the queue discovering the wall. This matters most on plan-based access where no endpoint reports remaining allowance, which forces the consumption estimate to be maintained locally.

Why do agent fleets pile up at a quota reset boundary?

Because deferred work accumulates and then arrives together. In this fleet, three days of deferred jobs reached for the allowance within minutes of its restoration, producing nine queued waits in a four-minute span and both of the only two lock timeouts in the entire log. The waits at that boundary ranged from 180 to 2,370 seconds, against 8 to 30 seconds during normal operation. The fix is a randomized per-engine delay applied to any post-outage or post-reset restart, which costs nothing and removes the herd.

How should wait budgets and priority be configured across multiple agent engines?

In one central table that the lock reads, not in environment variables at the call sites. This fleet's log shows three different budgets in use: 2,400 seconds on fifteen waits, 300 seconds on two, and 5,400 seconds on one. Only two are accounted for, the 300 being a deliberate short budget on cheap preflight probes and the 2,400 being the wrapper default inherited by omission. The 5,400-second wait belongs to no caller in the known inventory, which means the inventory is incomplete and the effective priority ordering at a contention boundary was never decided by anyone. Declaring the ordering in one reviewable place turns priority from an emergent property into a policy.

A Lock Is Not a Budget: My Agent Fleet Queued 8,283 Seconds for One Model Quota, Then Ran Out Anyway

TL;DR

Four of my scheduled engines share one locally-authenticated model session. I built a global mutex so they could never call it at the same time, and by its own terms the mutex works: 129 acquisitions, 18 queued waits, 2 stale locks correctly reclaimed, zero deadlocks. It did not save the week. This site missed three consecutive publish days — two to a weekly quota that was already spent by the time the job woke up, one to a lock timeout. The lesson is structural: a mutex prevents simultaneous consumption, not cumulative consumption. Serializing four engines through one quota does not make them fit inside it; it just makes them take turns running it down. The second failure is worse than the first. 120 of the 129 acquisitions (93%) are logged under the default label claude-gate.sh-caller, because not one of the four production callers sets CLAUDE_GATE_LABEL. So the log faithfully records 8,283 seconds of queue time and cannot tell me which brand caused any of it.

On August 9 and 10, my Build Log engine woke at 8:10 AM, pulled its analytics, invoked the model, and got back one line: You've hit your weekly limit · resets Aug 11 at 10pm. On August 11 at 10:07 PM — seven minutes after the reset, with every other engine on the machine reaching for the same freshly restored quota — it waited forty minutes in a queue and exited 75, deferring.

Three days, no post. Every component behaved exactly as designed.

The thing I built, and why it was right#

Every content engine here shells out to the same locally-authenticated CLI, which draws on one session quota. When four launchd jobs fire at 6:00 AM simultaneously, they don't get a quarter each. They collide, and the day's output is lost across all four.

So I wrote a global fleet lock. It is mkdir-based rather than flock-based, because macOS ships no flock(1) binary while mkdir is POSIX and atomic on every filesystem here — the kernel guarantees exactly one caller wins the create. The lock is a directory holding a meta file with the holder's PID, that process's start time, a label, and the acquisition epoch. It breaks and reclaims a lock when the holder's PID is gone, when the PID has been reused by a different process (start time mismatch), or when the hold exceeds 45 minutes.

I am not walking any of that back. Here is what the log says it actually did across the measured window:

Event	Count	Note
ACQUIRE	129	serialization held every time
WAIT (queued behind a holder)	18	correct behavior, not an error
ACQUIRE after waiting	15	8,283 s total queued, longest 2,370 s
TIMEOUT (exit 75, defer)	2	bounded wait, no infinite block
BREAK STALE LOCK	2	both holder-PID-gone, correctly reclaimed
Deadlocks	0	—

That is a healthy mutex. It is also completely beside the point, which took me three missed days to see.

The actual defect: a mutex is not a rate limiter#

A lock answers one question — may I go now? A quota answers a different one — is there anything left? I had built a careful, well-tested answer to the first question and no answer at all to the second.

Serialized access to a finite budget consumes that budget just as fast as parallel access does. It consumes it more politely, one engine at a time, in an orderly line. The line does not create capacity. By Sunday morning the week's allowance was gone, and the queue discipline that prevented four engines from colliding on Sunday was irrelevant, because there was nothing left for any of them to collide over.

Worse, the lock actively hid the depletion. A collision is loud: four jobs fail at once, at the same minute, with the same error, and you go looking. Serialized depletion is quiet. Engines succeed one after another, each one a little closer to the wall, and the first job to hit it looks like an isolated failure of that job rather than the last event in a chain. The Build Log is scheduled at 8:10 AM. It is near the back of the line. Its position was not incidental — it was structurally last, every single day, and that is a scheduling decision I never consciously made.

Then, at the reset boundary, the failure inverted. When the weekly quota came back at 10 PM on the 11th, the deferred jobs from three days did not come back politely. The log shows nine WAIT events between 22:06 and 22:10 that evening — a thundering herd against a lock designed for a trickle. Both TIMEOUTs in the entire log are from that night.

The observability defect underneath it#

Here is the line the log wrote when my Build Log engine gave up on the night of the 11th:

2026-08-11 22:47:28 [claude-gate.sh-caller] pid=85985 TIMEOUT after 2400s
  waiting for pid=85922 label=claude-gate.sh-caller; exiting 75

Read the two labels. claude-gate.sh-caller waited 40 minutes for claude-gate.sh-caller, and then deferred.

claude-gate.sh-caller is the default value of CLAUDE_GATE_LABEL — the fallback the wrapper uses when the caller doesn't identify itself. I built the label field on purpose, specifically so a queued engine could report who was ahead of it. Then I wrote four production callers and set it in exactly zero of them:

Caller	Sets `CLAUDE_GATE_LABEL`?	Gated calls	Wait budget
`bin/atlas-daily-blog-cron.sh`	no	3	300 s preflight, 2400 s generation
`bin/oceanfl-daily-blog-cron.sh`	no	2	300 s preflight, 2400 s generation
`campilii-site/engine/daily-buildlog.sh`	no	1	2400 s (default)
`mentorme-authority-engine/run.sh`	no	1	2400 s (default)

120 of 129 acquisitions (93%) and 13 of 18 waits are anonymous. The only labeled entries in the whole log — LEDGER, BUILDLOG, SEO-ACRO, TRAVELDRD, VERIFY-ATLAS — come from the test harness I ran on August 6 to prove the lock worked. I labeled the test. I did not label production. That is such a common shape of mistake that it deserves its own name: the instrumentation exists, was validated once, and was never wired into the thing it was built for.

The right-hand column has its own problem. Three different budgets appear in the log — 2,400 s on fifteen waits, 300 s on two, and 5,400 s on one — and I can account for only two of them. The 300-second budget is a deliberate choice on the cheap preflight probes in two engines, which is correct: a preflight should give up quickly. The 2,400 is the wrapper default that everything else inherits by omission. The 5,400-second wait belongs to no caller in the table above. Something on this machine invokes the gate with a 90-minute budget and is not in my inventory of four scripts — which means my inventory is not the fleet, it is the part of the fleet I remembered to look for. Anonymous labels are exactly why I cannot simply read the log and find out which one it is.

The general defect is that contention priority is emergent rather than declared. Which engine yields to which, at a boundary where they all want the same quota, is currently decided by environment variables scattered across call sites and written months apart — not by any statement of what matters most.

This is the same failure class as an analytics gate that has been blind for 42 days and a watcher that reads 76 rows out of 1,503: the check runs, exits zero, and reports on a slice of reality nobody has re-verified since the day it was built.

How this shows up in setups other than mine#

The specific numbers here are mine, but the shape generalizes badly across situations, and the right fix depends on which one you are in.

If you are one person with a personal-plan subscription and cron jobs — my situation — your binding constraint is a weekly allowance you cannot inspect programmatically. There is no endpoint that tells you how much is left. This is the hardest version, because you must estimate consumption locally: count invocations, record their approximate cost yourself, and stop before the wall rather than discovering it. The lock is necessary but nowhere near sufficient.

If you are a small team on an API key with per-minute rate limits, your constraint is a rate, not a pool, and it usually refills. Here a lock plus retry-with-backoff genuinely does solve most of it — the queue drains faster than the budget depletes. Do not import my conclusion wholesale; check whether your limit is a bucket that refills or an allowance that resets.

If you are early in a build, with two or three scheduled jobs, you will not see this at all for weeks. That is the trap. Contention is invisible below a threshold and then arrives all at once, usually the week you add the fourth engine. The cheap insurance is to add the label and the ledger now, when they cost ten minutes, not after a three-day outage.

If your failure mode is a herd rather than a drip — everything scheduled at the top of the hour, or everything retrying after an outage — the lock is your least important problem. Stagger the schedule and jitter the retries first. My four engines fire within a 130-minute window and still produced a nine-job pileup at one reset boundary, because deferred work does not stay deferred; it accumulates and then arrives together.

If you are running agents that call other agents, check reentrancy before anything else. My wrapper exports a held-flag that children inherit, so a nested invocation runs straight through instead of blocking on its own ancestor for the full 40-minute budget. Without that guard, one engine shelling out to another is a guaranteed self-deadlock, and it will look exactly like quota exhaustion in your logs.

What I am changing#

Four things, in order of how much they would have saved this week.

1. A consumption ledger, not just a lock. One append-only file, written at acquisition: date, label, engine, and the invocation's model tier. The gate reads it before granting the lock and refuses when the running weekly total crosses a reserve threshold. Refusing at 85% is not lost work — it is work that gets deferred deliberately, with the reserve pointed at whatever matters most.

2. Set the label. Everywhere. Four one-line changes: CLAUDE_GATE_LABEL=BUILDLOG and its siblings at each call site. Then add an assertion to the gate itself — if the label is the default value, log a warning. Instrumentation that can be silently skipped will be silently skipped, so make skipping it noisy.

3. Priority as policy, in one file. The wait budgets move out of the call sites and into a single table the gate reads: which engine yields to which, expressed once, reviewable in one place. Revenue-adjacent engines get the long budget; this Build Log gets the short one and defers to them. Today that ordering is not stated anywhere, so I cannot tell you whether it currently holds — and the unaccounted 5,400-second waiter is the proof that I cannot. The same reasoning applies one level down, to which model tier each engine is allowed to spend on: tier and priority are the same decision viewed from two sides.

4. Jitter the reset boundary. Nine jobs reaching for a quota within four minutes of its restoration is a herd I created. A randomized delay of a few minutes per engine at any post-outage restart costs nothing and removes the pileup that produced both of the TIMEOUTs in this log.

There is a fifth item that is really a note to myself. The log file is 994,409 bytes across 11,599 lines, of which 406 are actual events — 3.5%. The rest is the full text of the prompts, echoed into every acquire and release line. An audit log that is 96.5% payload is one I will not read casually, which is a large part of why this ran for days before I looked. Log the identity and the outcome; put the payload somewhere else.

The rule#

If your agents share a finite resource, the scheduler needs to know the size of the resource, not just the order of the line.

Every fleet operator eventually builds the mutex. It is the obvious fix, it is satisfying to build, and it is provably correct against the failure that prompted it. Then it quietly converts a loud simultaneous failure into a silent cumulative one, and the day the budget runs out you go looking for a bug in the lock — where there isn't one.

The audit is two commands. Grep your lock log for how many entries carry a default or missing label; that number is the fraction of your contention you cannot attribute. Then ask what your scheduler knows about how much of the shared resource is left. If the answer is nothing, you have an ordering mechanism where you need an accounting one, and the difference will show up as scheduled agents that failed without telling you.

RUNNING SCHEDULED AGENTS THAT SHARE ONE ACCOUNT?

The failures that actually cost output are almost never the model. They are the shared resources nobody is accounting for — quota, locks, rate limits, the one API key four jobs are quietly fighting over. Designing that layer so a one-person operation can run a real fleet is the work I do. Get in touch, or see the press and hire page.

— Italo Campilii. Measured 2026-08-12 on my own machine. Event counts, labels, wait durations, timeouts and stale-lock breaks are exact counts parsed from ~/.local/share/claude-gate.log (11,599 lines, 994,409 bytes; 406 lines matching the event-line pattern). The 8,283-second total and 2,370-second maximum are the sum and max of the 15 ACQUIRE after waiting Ns entries. Caller inventory and the CLAUDE_GATE_LABEL finding come from grepping the four production caller scripts; wait budgets are read from their invocation lines and from the gate's defaults. The three missed publish days are read from engine/log/run-2026-08-09.log through run-2026-08-11.log and engine/state/last-post-date. The chart plots the 15 measured wait values in log order. Nothing here is estimated.