Open your server logs today and you will see names like GPTBot, ClaudeBot and CCBot next to Googlebot. These are AI crawlers. Some collect pages to train language models. Others fetch pages so an AI assistant can quote and link to you. If you want to block AI crawlers in WordPress, the trick is to stop the first group without losing the second, and without hurting your Google rankings.
This guide shows you how to do that in 2026 with a few lines in robots.txt, an optional Cloudflare setting, and a quick test. No coding skills needed, and it works the same on Elementor, Gutenberg and classic sites.
Why AI crawlers matter for your WordPress site
AI bots now visit small business sites as often as big news sites. They can add server load, and they reuse your content in ways you did not choose. At the same time, AI search tools are becoming a real source of visitors, so blocking everything is rarely the smart move.
The good news: the big AI companies now split their bots by job. That lets you say “no” to training and “yes” to search citations.
Step 1: Know the three types of AI bots
Before you write any rules, learn who is knocking. Here are the main user agents in 2026 and what each one does.
| Bot | Company | Job | Suggested rule |
|---|---|---|---|
| GPTBot | OpenAI | Collects pages for model training | Block if you do not want training |
| OAI-SearchBot | OpenAI | Indexes pages for ChatGPT search answers and links | Allow |
| ChatGPT-User | OpenAI | Fetches a page when a user asks ChatGPT to open it | Usually allow (robots.txt may not apply) |
| ClaudeBot | Anthropic | Collects pages for model training | Block if you do not want training |
| Claude-SearchBot | Anthropic | Indexes pages for search answers | Allow |
| Claude-User | Anthropic | Fetches a page a user asked for | Usually allow |
| Google-Extended | A control token for Gemini training and grounding, not a separate crawler | Block if you do not want Gemini training | |
| CCBot | Common Crawl | Builds open datasets that many AI models train on | Block if you do not want training |
| PerplexityBot | Perplexity | Indexes pages for answer citations | Allow |
So there are three groups: training bots, search bots and user-triggered fetchers. Most site owners block the first group and allow the other two.
Step 2: Edit robots.txt in WordPress
robots.txt is a plain text file at yourdomain.com/robots.txt that tells crawlers what they may visit. WordPress creates a virtual one for you if no real file exists. You can change it in two easy ways:
- With your SEO plugin. Most SEO plugins include a file editor. In Yoast SEO, go to Yoast SEO > Tools > File editor and edit or create robots.txt there.
- With your hosting file manager or FTP. Create a file called robots.txt in the root folder of your site (the same folder as wp-config.php) and paste your rules in.
Always keep a copy of your current robots.txt before you change it. One wrong line, such as “Disallow: /” under “User-agent: *”, can hide your whole site from Google.
Step 3: Paste a “block training, allow search” rule set
Here is a safe starting point. It blocks the main training crawlers and keeps search and AI answer bots welcome.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yourdomain.com/sitemap_index.xml
A few notes:
- Google-Extended does not touch your rankings. Google says it is not used as a ranking signal and does not affect inclusion in Google Search. It only controls Gemini training and grounding.
- It will not remove you from AI Overviews. AI Overviews are part of Google Search and follow Googlebot. Never block Googlebot unless you want to leave Google completely.
- Change the sitemap line to the sitemap URL your SEO plugin shows.
- robots.txt is a request, not a lock. Well known AI companies respect it. Unknown scrapers may not, which is where Step 4 helps.
Step 4: Add a firewall layer with Cloudflare (optional)
If your site runs through Cloudflare, you get bot controls that work even when a crawler ignores robots.txt. Cloudflare now sorts AI traffic into three groups: Search, Agent and Training, and lets you set a rule for each.
Since 15 September 2026, Cloudflare’s new defaults block AI Training and Agent crawlers on pages that show ads, while search crawlers stay allowed. According to Cloudflare’s announcement as reported by eWeek, these defaults apply to new customers, new sites, and free plan sites whose owners never changed the setting. Cloudflare also offers Pay Per Use, an early program that pays publishers when their content is used in AI answers.
What to do:
- Log in to Cloudflare and open the AI crawler settings for each site. Check what is blocked today, because the default may already have changed for you.
- Match the settings to your robots.txt: block Training, allow Search, and decide on Agent based on whether you want AI assistants to act on your pages.
- If you run ads, be aware that a crawler doing both search and training can be judged by all of its behaviors. Test a few key pages after any change.
Not on Cloudflare? Many hosts and security plugins offer bot blocking by user agent. robots.txt alone is still enough for the major AI companies.
Step 5: Check that it works
Do not set it and forget it. Spend five minutes checking:
- Open yourdomain.com/robots.txt in a private browser window and confirm your new rules show. If they do not, clear your caching plugin and your CDN cache.
- Use the robots.txt report in Google Search Console to make sure Googlebot can still fetch your pages.
- Look at your server or hosting logs after a week. Training bots should get fewer page hits; search bots should keep visiting.
- Watch referral traffic in your analytics from chatgpt.com, perplexity.ai and similar sources. If AI referrals drop, you may have blocked a search bot by mistake.
- Review the list every few months. AI companies add and rename bots. For example, older Anthropic names like anthropic-ai and Claude-Web are now deprecated.
Should you block AI crawlers at all?
There is no single right answer. Think about your goals:
- Block training bots if your content is your product: courses, recipes, research, premium guides or original photos.
- Allow search bots if you want to be cited and linked in AI answers. For most shops, agencies and local businesses, those citations are free marketing.
- Allow everything if reach matters more than control, for example a brand new blog that needs every mention it can get.
Most WordPress site owners land in the middle: block AI crawlers in WordPress that only train, and welcome the ones that send people back.
FAQ
Will blocking AI crawlers hurt my Google rankings? No, as long as you do not block Googlebot. Google says the Google-Extended token is not a ranking signal and does not affect inclusion in Google Search.
Can I stop my site from appearing in Google AI Overviews? Not with Google-Extended. AI Overviews are part of Google Search and follow Googlebot rules. Snippet controls like nosnippet also remove normal search snippets, so use them with care.
Do all AI bots respect robots.txt? The major AI companies say their crawlers do. User-triggered fetchers such as ChatGPT-User may not follow it, and unknown scrapers often ignore it. A firewall rule, such as Cloudflare’s AI crawler controls, adds a second layer.
Do I need a plugin to block AI crawlers in WordPress? No. A few lines in robots.txt, added through your SEO plugin’s file editor or your hosting file manager, cover the major AI bots.
Conclusion: control AI bots, keep your visitors
AI crawlers are here to stay, but you decide how they use your site. To block AI crawlers in WordPress the smart way, block the training bots in robots.txt, keep the search bots, add a Cloudflare rule if you use it, and check your logs and Search Console once in a while. Ten minutes of setup gives you control without losing traffic.
While you are tidying up your site, give your pages a design worth protecting. Bright Addons has three free plugins on WordPress.org: Bright Addons for Elementor with 56 free widgets, the Bright Addons Blocks Library with 62 free Gutenberg blocks, and Bright Addons for WooCommerce with 28 free modules. Need more? Browse the Elementor widgets and Gutenberg blocks, or see the pricing page: Pro plans start from $10 a year during our Black Friday sale, with the discount already applied and no code needed, until 30 November 2026.


