Three Bots. One Decision.

Sponsored by

Are all bots actually bad?

That question came up in our Bots and Marketing roundtable two weeks ago.

We spent time digging into something Cloudflare just formalized in their settings: a three-way distinction between bot types, and a recommended policy for each.

It's cleaner than anything I've seen on this topic before, and it resolves much of the confusion around "should I be blocking bots?"

Here's the core framework, straight from Cloudflare's own interface:

Search bots - bots that scan your site so it shows up in search results.

Recommended setting: Allow.

Agent bots - bots that pull information from your site to answer user questions in AI tools.

Recommended setting: Allow.

Training crawlers - bots that scrape content to train AI models. They ignore your robots.txt, consume bandwidth, and give you nothing back.

Recommended Cloudflare setting: Block on pages with ads.

The reasoning behind blocking Training crawlers is a nuanced discussion.

Open access to your site creates real resource costs. Bandwidth, hosting, resources consumed for very little value delivered.

Not only that, but you need to consider whether you want large language AI models training on your language.

Especially open-source models or models from other countries, not just the household names like ChatGPT and Claude.

The only reliable way to stop it is at the DNS level. With this setting enabled, Cloudflare blocks them before they ever reach your site.

This gives you more direct protection than a robots.txt file, which crawlers have to choose to respect. So, this decision isn’t just about your SEO rankings anymore; there’s more to consider than ever.

One nuance Cloudflare flags: blocking Training crawlers can sometimes catch a bot a company also uses for Search.

That's why the recommended default is "Block on pages with ads" rather than a blanket block.

But honestly, if you've decided you don't want AI training on your data, blocking it outright is a reasonable call.

Hopefully seeing it presented this way can help you make that decision faster.

To Your Measured Success!

–Jeff Sauer
CEO of MeasureU

P.S. Curious how exposed your site might be? The free Bot Exposure scan at botexposure.com gives you a quick read on how vulnerable your domain might be to bot activity.

Unify Your Teams and Tech Stack With HubSpot

Connect your customer data, teams, and tools without the hassle of complex integrations or lengthy setups. One easy platform gives marketing, sales, and service teams a unified customer view and the tools to turn it into growth.

Why HubSpot and what's new
• Generate leads and automate marketing with Marketing Hub
• Build your pipeline and close more deals with Sales Hub
• Scale customer support and drive retention with Service Hub
• Keep customer data clean, connected, and actionable with one, unified platform

Join 306,000+ in over 135 countries using HubSpot to grow their businesses.

See what a more connected approach to growth can do for you and your team. Get setup quickly and start checking off your hardest tasks.

Get started free

Coming Up This Week at MeasureU

Mastermind (MeasureMastermind)

Wednesday, September 9, 12:00 PM – 1:00 PM EDT

Ready to level up your measurement game, get those burning questions answered, and connect with your amazing MeasureU family? Then you absolutely DON'T want to miss our weekly MeasureMastermind group coaching call! Let's break it down, shall we? 😉 🤔 What is it?Think of MeasureMastermind as your weekly power-up session! 🔋 It's live, interactive group coaching where our incredible MeasureU Instructors and super-smart MeasureU Mentors dive deep into YOUR questions. We kick things off by exploring "

This call is for MeasureU Pro Members

Brain Gainz: The Skill Loop

Thursday, September 10, 12:00 PM – 1:00 PM EDT

🧠 The Skill Loop: Turning AI Research Into Repeatable Measurement Skills Jordan Hill is back to build on the ideas from his MeasureSummit talk, GTM Autoresearch, with a framework designed to turn AI-assisted research into something you can use again and again in your measurement work. In this Brain Gainz, Jordan will introduce The Skill Loop and show how research, execution, feedback, and iteration can work together to help you build reusable measurement skills instead of starting from scratch

This call is for MeasureU Pro Members