Kill Criteria: Every Trading System I Built Failed Its Tests. That's the System Working.

There's a genre of post I never see: the one where the builder lists everything that didn't work, with the numbers. So here's mine.

Over several months I built, backtested, and walk-forward tested a stack of trading systems. Not casually — with scripts, holdout data, logged trades, and a written gate every system had to pass before a single real dollar touched it. The result: every system that reached its verdict failed and was killed. The ones still running are paper-only, sitting behind the same gate with scheduled verdict dates.

If you're evaluating whether to trust someone with your production systems, this postmortem tells you more than any success story I could write. Anyone can show you the demo that worked. The question is what happens when the evidence says no.

What kill criteria are, in principle

A kill criterion is a pass/fail threshold you write down before you see results, attached to a decision you've pre-committed to: if the system doesn't clear this bar on data it has never seen, it dies — no renegotiation, no "but it almost passed," no extending the test until it looks better.

Every system I tested faced the same gate:

1. Holdout profit factor ≥ 1.15. Not on the training data the strategy was tuned on — on a held-out period it never saw. A system that only wins on the data it was fit to has learned the past, not the market.
2. At least 100 trades. Eight winning trades is an anecdote. A hundred is the start of a sample.
3. Holdout performance not materially worse than training. A big gap between in-sample and out-of-sample results is the signature of curve-fitting, even when both numbers look positive.

Three lines. The entire value is that they were written before the tests ran. Kill criteria written after you've seen the results aren't criteria — they're rationalizations with a threshold attached.

What the tests actually showed

Here is the ledger, straight from my own test logs. No system on this table ever traded real capital.

Notice the shapes of failure, because they generalize far beyond trading:

• Curve-fitting — brilliant on the data it was tuned on, losing on data it never saw. The gold EA and the forex robot both died here.
• Sample-size theater — the BTC crossover "worked" on 8 trades. Its real holdout profit factor was zero.
• Vanity metrics — 73–77% win rates while losing money overall. High win rate with profit factor under 1 just means many small wins funding fewer large losses. I now treat any impressive-sounding rate metric, in any system, as a prompt to ask what the denominator is hiding.
• The bug that flatters you — the stat-arb model's spectacular numbers came from a code artifact. The tell: it produced the same spectacular numbers on random noise. If your test can't fail on noise, it isn't a test.
• Structural absence — some edges simply don't exist once you account for fees, spreads, and faster competitors. No amount of tuning summons them.

Why shipping nothing was the win

The honest accounting: I spent real engineering time and shipped zero trading systems to real capital. By demo-culture standards, that's a total loss.

By operator standards, it's the whole point. The counterfactual isn't "shipped a winner" — the tests demonstrate no winner existed in that pile. The counterfactual is deploying real money into systems that provably lose out of sample, then discovering it live, with a bankroll instead of a log file.

The kill criteria converted what would have been an expensive live lesson into a cheap documented one. Every tombstone in that table is a loss that didn't happen.

And the tests surfaced something more valuable than any single verdict — a pattern: the entire retail trading content economy sells the search for a system, because the search is what's profitable to the sellers. Courses, EAs, signal channels, YouTube strategies — the product is hope, renewed monthly. Eight-plus systems chased, all failed the same gate. The search is the trap. Only a pre-committed gate lets you see that, because without one, every near-miss reads as "almost there" and the search continues forever.

Two candidates are still alive as I write this — a prediction-market strategy with 257 settled paper trades, and a macro copy-trade being shadow-tracked. Both are paper-only, both sit behind the same 100-trade, PF ≥ 1.15 gate, and both have scheduled verdict checkpoints on the calendar. Surviving so far earns them continued testing. It does not earn them capital.

The same discipline runs my agent fleets

Here's why this postmortem belongs on a blog about AI agent operations: trading was where I stress-tested the discipline, but the discipline is the product. Agent systems fail in exactly the same shapes trading systems do — they demo brilliantly and degrade quietly in production. I've written about that failure mode as the verification gap; kill criteria are the other half of the answer.

Concretely, the same rules transfer one-to-one:

Pre-committed gates instead of vibes. Before an agent pipeline touches production, it has a written pass condition — outputs verified against source data, structure checks passing, a defined sample of runs reviewed — the equivalent of holdout PF ≥ 1.15. "The output looks good" is the agent-fleet version of a backtest on training data.

Verification on data the system hasn't seen. A walk-forward test asks: does this hold up out of sample? An agent verification gate asks the same question every run: does the output survive contact with reality — real inventory numbers, real page renders, real deliverability — not just a plausible-looking draft? What that gate consists of in practice is a core part of what an AI systems builder actually does.

Scheduled checkpoints, on the calendar. The surviving trading candidates have named verdict dates and scripts that render the verdict. My agent fleets get the same treatment: review dates where a pipeline must justify its continued existence with evidence, not tenure.

Retiring what underperforms — and writing it down. The maker bot was retired the day the adverse-selection evidence was in, and the retirement is documented so the idea can't sneak back in six months later wearing a new name. Agent pipelines that underperform get the same tombstone. A fleet where nothing ever gets killed isn't a healthy fleet; it's an unaudited one.

Sample size before conclusions. Eight trades proved nothing about the BTC strategy; eight good agent runs prove nothing about a pipeline. Gates specify minimum run counts before a verdict, in both domains.

The transferable skill isn't trading. It's the willingness to define, in advance, what evidence would make you stop — and then actually stopping.

What this means if you're hiring

When you evaluate someone to build or run autonomous systems for you, the portfolio of wins tells you what worked. The postmortems tell you how they'll behave when things don't — which, in production, is most of the interesting time.

An operator who documents negative results against pre-committed criteria will tell you when your pipeline is underperforming, kill the feature that isn't earning its keep, and hold your systems to gates instead of demos. An operator who only has wins to show you has either never tested anything hard, or has tested things and isn't telling you. Neither is who you want holding the keys. If you're running that evaluation right now, I wrote a screening guide: how to hire an AI systems builder.

FAQ

What are kill criteria in evidence-based systems?

Kill criteria are pass/fail thresholds defined before a system is tested, attached to a pre-committed decision: if the system fails the threshold on data it hasn't seen, it is shut down — no renegotiation. Mine were holdout profit factor ≥ 1.15, a minimum of 100 trades, and holdout performance not materially worse than training. The pre-commitment is the entire value; criteria written after seeing results are rationalizations.

Why did every trading system fail the same gate?

Because the gate tested for the things retail strategies are optimized to hide: curve-fitting (great in training, losing out of sample), tiny sample sizes, and vanity win rates masking sub-1 profit factors. Systems tuned to look good on past data reliably fall apart on held-out data — which is exactly what walk-forward testing exists to expose.

Isn't shipping nothing just failure?

No — the tests showed no deployable edge existed in the pile, so the alternative to shipping nothing was deploying capital into systems that provably lose out of sample. The kill criteria converted expensive live losses into cheap documented ones. In evidence-based systems, a correct "no" is a successful outcome.

How do kill criteria apply to AI agent systems?

Identically. Agent pipelines get written pass conditions before production, verification gates that check outputs against reality every run, minimum sample sizes before any verdict, scheduled checkpoint dates where each pipeline must justify itself with evidence, and documented retirement when one underperforms. Agent systems fail in the same shapes trading systems do — impressive in the demo, degrading quietly in production — and the same discipline catches both.

Why publish your own negative results?

Because they're the strongest trust signal an operator can offer. Anyone can curate wins. Documented negative results against pre-committed criteria show how someone behaves when the evidence says no — and that behavior is what you're actually hiring when you hand over production systems.

— Italo Campilii. If you're building something that needs this kind of operator, get in touch.