BLACK FRIDAY: 50% OFF all Pro plans Code: BLACKFRIDAY50
Pro from $10 a year or $30 lifetime, no code needed: Code: BLACKFRIDAY50
Ends Cyber Monday, 30 Nov. Lock in the price today. Code: BLACKFRIDAY50
Get 50% Off

How to Block AI Crawlers in WordPress Without Losing Search Traffic (2026 Guide)

AI bots now crawl almost every WordPress site. Here is how to block the ones that train models, keep the ones that send visitors, and check it works.
White humanoid robot standing in front of a brown wall

Open your server logs today and you will see names like GPTBot, ClaudeBot and CCBot next to Googlebot. These are AI crawlers. Some collect pages to train language models. Others fetch pages so an AI assistant can quote and link to you. If you want to block AI crawlers in WordPress, the trick is to stop the first group without losing the second, and without hurting your Google rankings.

This guide shows you how to do that in 2026 with a few lines in robots.txt, an optional Cloudflare setting, and a quick test. No coding skills needed, and it works the same on Elementor, Gutenberg and classic sites.

Why AI crawlers matter for your WordPress site

AI bots now visit small business sites as often as big news sites. They can add server load, and they reuse your content in ways you did not choose. At the same time, AI search tools are becoming a real source of visitors, so blocking everything is rarely the smart move.

The good news: the big AI companies now split their bots by job. That lets you say “no” to training and “yes” to search citations.

Step 1: Know the three types of AI bots

Before you write any rules, learn who is knocking. Here are the main user agents in 2026 and what each one does.

Bot Company Job Suggested rule
GPTBot OpenAI Collects pages for model training Block if you do not want training
OAI-SearchBot OpenAI Indexes pages for ChatGPT search answers and links Allow
ChatGPT-User OpenAI Fetches a page when a user asks ChatGPT to open it Usually allow (robots.txt may not apply)
ClaudeBot Anthropic Collects pages for model training Block if you do not want training
Claude-SearchBot Anthropic Indexes pages for search answers Allow
Claude-User Anthropic Fetches a page a user asked for Usually allow
Google-Extended Google A control token for Gemini training and grounding, not a separate crawler Block if you do not want Gemini training
CCBot Common Crawl Builds open datasets that many AI models train on Block if you do not want training
PerplexityBot Perplexity Indexes pages for answer citations Allow

So there are three groups: training bots, search bots and user-triggered fetchers. Most site owners block the first group and allow the other two.

Step 2: Edit robots.txt in WordPress

Lines of code on a dark editor screen

robots.txt is a plain text file at yourdomain.com/robots.txt that tells crawlers what they may visit. WordPress creates a virtual one for you if no real file exists. You can change it in two easy ways:

  1. With your SEO plugin. Most SEO plugins include a file editor. In Yoast SEO, go to Yoast SEO > Tools > File editor and edit or create robots.txt there.
  2. With your hosting file manager or FTP. Create a file called robots.txt in the root folder of your site (the same folder as wp-config.php) and paste your rules in.

Always keep a copy of your current robots.txt before you change it. One wrong line, such as “Disallow: /” under “User-agent: *”, can hide your whole site from Google.

Step 3: Paste a “block training, allow search” rule set

Here is a safe starting point. It blocks the main training crawlers and keeps search and AI answer bots welcome.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yourdomain.com/sitemap_index.xml

A few notes:

  • Google-Extended does not touch your rankings. Google says it is not used as a ranking signal and does not affect inclusion in Google Search. It only controls Gemini training and grounding.
  • It will not remove you from AI Overviews. AI Overviews are part of Google Search and follow Googlebot. Never block Googlebot unless you want to leave Google completely.
  • Change the sitemap line to the sitemap URL your SEO plugin shows.
  • robots.txt is a request, not a lock. Well known AI companies respect it. Unknown scrapers may not, which is where Step 4 helps.

Step 4: Add a firewall layer with Cloudflare (optional)

Rack of servers with blue lights in a data center

If your site runs through Cloudflare, you get bot controls that work even when a crawler ignores robots.txt. Cloudflare now sorts AI traffic into three groups: Search, Agent and Training, and lets you set a rule for each.

Since 15 September 2026, Cloudflare’s new defaults block AI Training and Agent crawlers on pages that show ads, while search crawlers stay allowed. According to Cloudflare’s announcement as reported by eWeek, these defaults apply to new customers, new sites, and free plan sites whose owners never changed the setting. Cloudflare also offers Pay Per Use, an early program that pays publishers when their content is used in AI answers.

What to do:

  • Log in to Cloudflare and open the AI crawler settings for each site. Check what is blocked today, because the default may already have changed for you.
  • Match the settings to your robots.txt: block Training, allow Search, and decide on Agent based on whether you want AI assistants to act on your pages.
  • If you run ads, be aware that a crawler doing both search and training can be judged by all of its behaviors. Test a few key pages after any change.

Not on Cloudflare? Many hosts and security plugins offer bot blocking by user agent. robots.txt alone is still enough for the major AI companies.

Step 5: Check that it works

ai-crawler-traffic-analytics.webp

Do not set it and forget it. Spend five minutes checking:

  1. Open yourdomain.com/robots.txt in a private browser window and confirm your new rules show. If they do not, clear your caching plugin and your CDN cache.
  2. Use the robots.txt report in Google Search Console to make sure Googlebot can still fetch your pages.
  3. Look at your server or hosting logs after a week. Training bots should get fewer page hits; search bots should keep visiting.
  4. Watch referral traffic in your analytics from chatgpt.com, perplexity.ai and similar sources. If AI referrals drop, you may have blocked a search bot by mistake.
  5. Review the list every few months. AI companies add and rename bots. For example, older Anthropic names like anthropic-ai and Claude-Web are now deprecated.

Should you block AI crawlers at all?

There is no single right answer. Think about your goals:

  • Block training bots if your content is your product: courses, recipes, research, premium guides or original photos.
  • Allow search bots if you want to be cited and linked in AI answers. For most shops, agencies and local businesses, those citations are free marketing.
  • Allow everything if reach matters more than control, for example a brand new blog that needs every mention it can get.

Most WordPress site owners land in the middle: block AI crawlers in WordPress that only train, and welcome the ones that send people back.

FAQ

Will blocking AI crawlers hurt my Google rankings? No, as long as you do not block Googlebot. Google says the Google-Extended token is not a ranking signal and does not affect inclusion in Google Search.

Can I stop my site from appearing in Google AI Overviews? Not with Google-Extended. AI Overviews are part of Google Search and follow Googlebot rules. Snippet controls like nosnippet also remove normal search snippets, so use them with care.

Do all AI bots respect robots.txt? The major AI companies say their crawlers do. User-triggered fetchers such as ChatGPT-User may not follow it, and unknown scrapers often ignore it. A firewall rule, such as Cloudflare’s AI crawler controls, adds a second layer.

Do I need a plugin to block AI crawlers in WordPress? No. A few lines in robots.txt, added through your SEO plugin’s file editor or your hosting file manager, cover the major AI bots.

Conclusion: control AI bots, keep your visitors

AI crawlers are here to stay, but you decide how they use your site. To block AI crawlers in WordPress the smart way, block the training bots in robots.txt, keep the search bots, add a Cloudflare rule if you use it, and check your logs and Search Console once in a while. Ten minutes of setup gives you control without losing traffic.

While you are tidying up your site, give your pages a design worth protecting. Bright Addons has three free plugins on WordPress.org: Bright Addons for Elementor with 56 free widgets, the Bright Addons Blocks Library with 62 free Gutenberg blocks, and Bright Addons for WooCommerce with 28 free modules. Need more? Browse the Elementor widgets and Gutenberg blocks, or see the pricing page: Pro plans start from $10 a year during our Black Friday sale, with the discount already applied and no code needed, until 30 November 2026.

Share this post

Keep reading