One Switch Blocks AI Training. The Other Blocks Your Customers.

Sometime this winter, someone will ask whether to block the AI bots

It will sound like a small IT question. A new panel appears in Bing Webmaster Tools. A new kind of line becomes available in robots.txt, the small file on your site that tells crawlers what they’re allowed to do. Someone asks whether you want the AI companies taking your content, and the answer gets made in the same five minutes as the firewall renewal.

If your phone stopped ringing while your rankings held steady, treat that question with more respect. On October 30, Google, Microsoft, OpenAI, and Anthropic announced they’ll honor a shared robots.txt signal, and Microsoft followed the next day with controls that make the underlying split official. Blocking the AI companies is now two switches. The one most people flip by accident controls whether the machines recommend you at all.

What changed, in plain terms

For years, keeping an AI crawler off your site meant writing a rule for that crawler: Google-Extended for Google, GPTBot for OpenAI, ClaudeBot for Anthropic. One company, one rule, and if you never wrote any rules, everything was allowed.

On October 30, all four providers said they’ll honor a common signal being built in the IETF, the standards body behind the web’s plumbing. One line in robots.txt states your preference about AI training, and every participating company reads it. OpenAI says it will implement the signal in the coming months. Google is aiming at 2027.

The next day, Microsoft went further and exposed the architecture everyone is converging on. Bing’s new controls split AI Training and AI Grounding into separate settings, each with its own allow or prevent choice. A middle position, Limited Grounding, lets your content appear in Microsoft’s own answers but in no one else’s. The temporary grounding controls go live December 15, 2026. Bing starts honoring the new preferences February 1, 2027. The old directives stop working March 1, 2027.

Training is the memory. Grounding is the referral

Training is whether the models learn from your pages. It shapes future versions of these systems, months and years out.

Grounding is whether the systems pull your pages when they answer a question today. When a buyer asks ChatGPT for a supplier of what you make, that’s a grounding question. The machine searches, reads, and cites what it finds right now, which is why the same fundamentals that make you citable apply before any robots.txt line does.

Blocking training changes nothing about tonight’s answers. The model answering your buyer’s question was trained long ago, on whatever it was allowed to read then. Blocking training is a statement about the future. Blocking grounding is a statement about this quarter’s quotes.

The broad opt-out is a package deal

Now the trap. When Google’s John Mueller explained how the new signal will work, he spelled out the default: if a site uses the common token to opt out of training, Google reads that as a Google-Extended disallow, unless the site sets Google-Extended explicitly on its own. And the sites that specifically allow Google-Extended are the ones eligible to appear in Gemini’s answers, with the clicks that come with them.

Read that again as an owner. The single opt-out line, the one a busy webmaster pastes in to be done with the topic, says two things at once. Don’t train on me. And, quietly, don’t cite me either.

Nothing errors. Nothing shows up in a log. Rankings hold steady. But the next time a prospect asks Gemini for a shortlist, you’re absent by design, and the competitor who never touched the file is sitting right there.

Bing’s panel lets you block training and stay citable in Copilot’s answers, but only if someone opens the settings and sets each switch deliberately.

The first bill for opting out has already arrived

Plenty of owners will hear all this and still want out. It’s their content. But the first real measurement of what opting out costs came in late October, and it deserves a look before anyone decides.

Cyrus Shepard, a well-known search practitioner, used Google’s own opt-out controls to remove one of his sites from AI Overviews and AI Mode, then watched the numbers in Google Search Console. AI impressions went to zero immediately, as expected. Two findings weren’t. Roughly 0.9 percent of the times his site appeared in an AI answer had been turning into a click. And when the AI features went off, his regular organic clicks dropped too. His read: appearing in the AI answers carries a halo that lifts the classic results underneath.

One site, self-reported, no control group. It’s still the first measurement anyone has published, and it points one direction. Opting out cost him the AI answers and clicks in the results he thought he was protecting.

There’s also a layer you can’t control

Search Engine Journal documented in late October that ChatGPT runs a background safety check, visible in network traffic as an Aura call, that can refuse to open a site entirely when the assistant browses the live web. When that fires, it ignores robots.txt, it ignores your allow rules, it caches for about a day, and there’s no public appeal path. Established media sites have reportedly been caught by it with no explanation.

The lesson generalizes past OpenAI. Crawlable no longer means openable. Your reach into the buyer’s question now runs through settings on your side and filters on theirs. You control one of those halves, which is exactly why your half deserves better than a copy-paste line.

What we’d tell a client

The playbook is short.

• Decide the two switches separately.
• If you want to be recommended by these systems, leave grounding open everywhere. That’s the switch wired to the phone.
• If training bothers you, opt out per provider, with the specific rules that have existed for years. Skip the bundled token. It’s the one setting that takes a position on both switches while describing only one.
• Put three dates on the calendar: December 15, February 1, March 1. That’s when Bing’s controls land, take effect, and retire the old directives. Whoever manages your site should re-check the panel after each date.

When we diagnose a company’s standing with the AI systems, the first layer of the work is exactly this: whether the machines can reach the pages, open them, and read the facts. The new settings just added a row to that checklist. If you want the rest of the checklist, the thirty-minute version is on this site, and it starts with a test you can run tonight.

The test that settles it

Ask a machine that doesn’t know you. Your own ChatGPT account carries your history and will flatter you, so use a fresh session or hand the question to a friend across town: who are the top suppliers of what you make, in your region.

If your name comes up, grounding is working for you, and every conversation about blocking these systems is a conversation about closing a door buyers are walking through. If it never comes up, the settings panel isn’t your problem. The machines aren’t citing you because they don’t hold solid enough facts about you, and no switch on that panel fixes that.

The referral network moved into these machines. The new controls are being sold as control over AI, and they are. One of the two switches decides whether the machine can still refer you.