My Personal Site Was Blocking Every AI Crawler. I Found Out During My Own Audit.

There's a specific kind of embarrassing that only happens to people who do this for a living. I build agent fleets and GEO pipelines for five brands. I write about getting cited by AI engines. My personal site exists for exactly one reason: so that when a hiring manager asks ChatGPT or Claude "who is Italo Campilii," the answer comes from my own words instead of a scraped fragment. That only works if AI crawlers can actually read the site.

And for its entire life until July 14, campilii.com was telling every one of those AI crawlers to go away.

Not through anything I wrote. Through a Cloudflare default I never knowingly enabled — a managed robots.txt injected at the edge, invisible in my repo, doing the precise opposite of the site's mission. This is the postmortem: how I found it, why it hides where you won't look for it, the 60-second fix, and how to verify you're actually clean.

How the block was found

It surfaced during a full SEO audit of the domain — the same kind of pass I run on client properties, finally pointed at my own. The audit line item that flagged red was blunt: Cloudflare's "AI bot block" managed rules were live. The served robots.txt contained a # BEGIN Cloudflare Managed content block with a Content-Signal: ai-train=no line and explicit Disallow rules for the crawlers that matter most to a personal-brand site.

Here's what was being turned away, and what each crawler actually feeds:

For a news site guarding paid content, that list might be a deliberate choice. For a site whose whole point is being cited by AI engines, it's self-sabotage — every door I was trying to walk through, bolted from the inside.

The sharper sting: I had already fixed this exact class of problem once. On 2026-06-30, during a tri-brand SEO review, I found acromatico.com — an AI-visibility studio, a business that literally sells "get recommended by AI" — returning HTTP 403 to GPTBot, PerplexityBot, and ClaudeBot while Googlebot and browsers got 200. That one was Cloudflare's ai_bots_protection bot-management setting, flipped to "block". Fixed via the API, re-verified all three bots at 200. Two weeks later, a different Cloudflare mechanism was doing the same damage on a different zone. Same failure class, new disguise.

Where the toggle actually lives (not where you'd look)

This is the part worth bookmarking, because the Cloudflare dashboard actively misdirects you here.

When you hear "Cloudflare is blocking AI bots," the instinct is to go to Security → Bots and look at the "Block AI bots" card. I did. That card's three AI policies — Search, Agent, Training — were already set to Allow. By every visible indicator on the screen you'd naturally check, the zone was open to AI crawlers.

It wasn't. The block was coming from somewhere else entirely: the Managed robots.txt toggle, which lives under AI Crawl Control → Overview in the zone dashboard. When that toggle is on, Cloudflare injects its managed Disallow block into your robots.txt at the edge — layered on top of whatever file your origin serves. Your repo's robots.txt can be a perfectly clean all-crawlers-allowed file, your deploys can be green, and visitors to /robots.txt still receive the managed block appended to it.

That's what makes this failure mode nasty:

• It's invisible in your source. Nothing in the repo shows it. git log is innocent. The file you wrote is not the file being served.
• The obvious dashboard card says you're fine. Security → Bots showed Allow across the board while the edge served Disallow lines.
• It fails silently. No error, no warning, no dropped traffic alarm. AI crawlers just quietly read the Disallow and leave, and you never rank in the one channel you built the site for.

The 60-second fix

Once you know where to look, the fix is almost anticlimactic:

1. Cloudflare dashboard → select the zone (campilii.com).
2. AI Crawl Control → Overview.
3. Find Managed robots.txt. Turn it off.

That's it. The edge-injected # BEGIN Cloudflare Managed content block — the ai-train=no signal and every managed Disallow line — disappeared from the served file within seconds of flipping the toggle. No deploy, no purge, no waiting on propagation.

Verify it — never trust the toggle

I've written before about the verification gap — the distance between "the system says it's done" and "I watched it be true." The same discipline applies to infrastructure toggles. A dashboard telling you a setting is off is a claim, not evidence. The evidence is what the edge actually serves:

curl -s https://campilii.com/robots.txt

Post-fix result, verified 2026-07-14: HTTP 200, zero Disallow lines — only the repo's own all-crawlers-allow file plus the Sitemap line. That curl output is the artifact I trust, not the toggle state. Run it against your own domain right now; grep for Disallow and for Cloudflare Managed. If either shows up and you didn't put it there, you've found your problem.

curl -s https://yourdomain.com/robots.txt | grep -iE "disallow|cloudflare managed"

Empty output is what a clean bill of health looks like.

The micro-lesson: defaults can cancel strategy

Here's the transferable part, compressed:

A security default you didn't choose can silently cancel a growth strategy you spent months on. Cloudflare's AI-blocking features are reasonable products for sites that want them. But they've shipped through multiple mechanisms — bot-management settings, managed rules, managed robots.txt — enabled at different times through different dashboard surfaces, and the surface you'd naturally check doesn't tell you about the others. Twice in two weeks I found AI-blocking active on a property whose explicit purpose was AI visibility, via two different mechanisms, on two different accounts.

So the operating rule I've adopted, and the one I'd hand any operator running multiple properties:

1. Audit the served robots.txt on every property you own. Not the file in the repo — the file at the URL. They can differ, and the edge wins.
2. Check all three Cloudflare surfaces: Security → Bots (bot management / ai_bots_protection), AI Crawl Control → Overview (Managed robots.txt), and any managed WAF rules on the zone.
3. Re-check after Cloudflare ships new AI features. Both of my incidents trace to features layered onto zones over time. New feature launches are when defaults shift.
4. Verify with curl, from outside, every time. The toggle is a claim. The response body is the fact.

This kind of check belongs on a schedule, not a memory — the same way I treat every recurring guarantee in my fleet (if it isn't scheduled and verified, it doesn't exist). A robots.txt curl in a daily health script costs nothing and would have caught this on day one instead of during a manual audit.

FAQ

Why was Cloudflare blocking AI crawlers on my site if I never enabled it?

Cloudflare has shipped several AI-crawler controls — bot-management settings and a Managed robots.txt feature under AI Crawl Control — and they can be active on a zone without you having deliberately configured them. The served robots.txt is the ground truth: fetch it with curl and look for a # BEGIN Cloudflare Managed content block or Disallow lines you didn't write.

Where is the setting that injects Disallow lines into robots.txt?

In the Cloudflare dashboard: select the zone, then go to AI Crawl Control → Overview, and turn off Managed robots.txt. It is not the "Block AI bots" card under Security → Bots — on my zone those policies were already set to Allow while the managed robots.txt block was still being served.

How do I verify AI crawlers can actually reach my site?

Run curl -s https://yourdomain.com/robots.txt and confirm there are no Disallow lines targeting ClaudeBot, GPTBot, Google-Extended, CCBot, Applebot-Extended, or Bytespider, and no ai-train=no content signal. For the 403-style block (a separate mechanism), request a page with an AI crawler user-agent and confirm you get HTTP 200, not 403.

Does blocking AI crawlers hurt regular Google rankings?

Blocking Google-Extended doesn't affect classic Google Search crawling (that's Googlebot), but it cuts you out of Gemini's use of your content. The larger cost is generative-engine visibility: if ClaudeBot, GPTBot, and CCBot can't read you, AI assistants answer questions about you and your topics from other people's pages.