AI crawlers explained: How AI bots interact with your WordPress site

Websites aren’t built just to publish content, and metadata isn’t fine-tuned for fun; it’s all of these activities that work together so your pages can be discovered more easily. For years, Google Search has been the primary gateway to that visibility, thanks largely to its web crawlers.

Since the late 1990s, Googlebot and other traditional crawlers have scanned websites, fetched HTML pages, and indexed them to help people find what they’re looking for. As of January 2024, Google accounted for 63% of all U.S. web traffic, driven by the top 170 domains.

But now, according to a survey by McKinsey, half of customers now turn to AI tools like ChatGPT, Claude, Gemini, or Perplexity for instant answers, and even Google is blending AI-generated summaries into search results through features like AI Overviews.

Behind these new AI-driven experiences is a growing class of bots known as AI crawlers. If you run a WordPress site, understanding how these crawlers access and use your content is more important than ever.

What are AI crawlers?

AI crawlers are automated bots that scan publicly accessible web pages, similar to search engine crawlers, but with a different purpose. Instead of indexing pages for traditional ranking, they collect content to train large language models or supply fresh information to AI-generated responses.

Broadly, AI crawlers fall into two groups:

1. Training crawlers, such as GPTBot (OpenAI) and ClaudeBot (Anthropic), collect data to teach large language models how to answer questions more accurately.
2. Live retrieval crawlers like ChatGPT-User access websites in real time when someone asks something that requires the latest data, like checking a product description or reading documentation.

Other crawlers, PerplexityBot or AmazonBot, for example, are building their own indexes or systems to reduce their dependence on third-party sources. And while their goals differ, they all have one thing in common: they fetch and read content from websites like yours.

How AI crawlers work

When an AI crawler visits your site, it typically does the following:

• Sends a basic GET request to the page’s URL (no interaction, scrolling, or DOM events).
• Fetches only the initial HTML returned by the server. It doesn’t wait for client-side JavaScript to load or execute.
• Extracts all <a href="">, <img src="">, <script src="">, and other resource links, then adds internal (and sometimes external) URLs to its crawl queue. In many cases, it also hits broken links that return 404 errors.
• May attempt to fetch linked assets like images, CSS files, or scripts, but only as raw resources, not to render the page.
• Repeats this process recursively across discovered links to map out the site.

How AI crawlers interact with WordPress websites

WordPress is a server-rendered platform that uses PHP to generate full HTML pages before sending them to the browser. When a crawler visits a WordPress site, it usually gets everything (content, headings, metadata, navigation) it needs in the HTML response.

This server-rendered structure makes most WordPress sites naturally crawler-friendly. Whether Googlebot or an AI crawler, they can usually scan your site and easily understand your content. In fact, easily crawlable content is one of the reasons WordPress performs well in both traditional search and newer AI-driven platforms.

Should you allow AI crawlers to access your content?

AI crawlers can already read most WordPress sites by default. The real question is what you want them to access — and how you can control that visibility.

Content-driven businesses are abuzz with this conversation right now. The subject extends to blog posts, documentation, landing pages … anything written for the web, really. You’ve probably heard advice like “write for the machines” since AI platforms increasingly pull live data and, in some cases, now include links to sources. We all want to show up in LLM output, just as much as we want to show up in Google search results.

For example, in the screenshot below, we ask ChatGPT to tell us some of the latest features released by Kinsta. It searches the web, scans changelogs and linked pages, and provides a summarized answer with direct links back to the source.

It’s early, but AI crawlers already influence what people see when they ask questions online. And that reach could matter.

Guillermo Rauch, CEO of Vercel, shared in April that ChatGPT accounts for nearly 10% of new Vercel sign-ups, up from less than 1% just six months earlier. That demonstrates how quickly AI-driven referrals can evolve into a significant acquisition channel.

And AI crawlers are widespread. According to Cloudflare, AI bots accessed around 39% of the top one million websites, but only about 3% of those sites actually blocked or challenged that traffic.

So even if you haven’t made a decision yet, AI crawlers are almost certainly visiting your site already.

Should you allow or block AI crawlers?

There’s no one-size-fits-all answer. There’s no universal answer, but here’s a framework:

• Block crawlers on sensitive or low-value routes like /login, /checkout, /admin, or dashboards. These don’t help discovery and only waste bandwidth.
• Allow crawlers on “discovery content” such as blog posts, documentation, product pages, and pricing information. These pages are the ones most likely to be cited in AI responses and drive qualified traffic.
• Decide strategically for premium or gated content. If your content is your product (e.g., news, research, courses), unlimited access to AI may undercut your business.

New tools are emerging to help. Cloudflare, for example, is experimenting with a model called Pay Per Crawl, which allows site owners to charge AI companies for access. It’s still in private beta, and real-world adoption is early, but the idea has gained strong support from large publishers who want more control over how their content is used.

Others in the search and marketing community are more cautious, as default blocking could unintentionally reduce visibility in AI search results for sites that actually want the exposure. For now, it’s a promising experiment rather than a mature revenue stream.

Until these systems mature, the most practical approach is selective openness, where you keep discovery content crawlable, block sensitive areas, and revisit your rules as the ecosystem evolves.

How to control AI crawler access on WordPress

If you aren’t comfortable with AI crawlers accessing your WordPress site and scanning its content, the good news is that you can take back control.

Here are three ways to manage AI crawler access on WordPress:

1. Manually editing your robots.txt file.
2. Use a plugin to do it for you.
3. Use Cloudflare’s bot protection.

Let’s walk through all three options.

Option 1: Block AI crawlers manually with robots.txt

Your robots.txt file tells bots what parts of your site they’re allowed to crawl. Most well-known AI crawlers, like OpenAI’s GPTBot, Anthropic’s Claude-Web, and Google-Extended, respect these rules.

You can block specific bots entirely, allow them full access, or restrict access to certain sections of your site. For example, to block everything, you can add this to your robots.txt file, although this is not recommended for most sites:

User-agent: GPTBot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: Google-Extended
Disallow: /

To allow full access to OpenAI’s GPTBot:

User-agent: GPTBot
Disallow:

To block just a section of your site from OpenAI’s GPTBot. For example, your login page, where crawlers add no value:

User-agent: GPTBot
Disallow: /login/

This kind of selective blocking is key. Sensitive routes like /login, /checkout, or /admin don’t help with discoverability and should almost always be blocked. On the other hand, product pages, feature overviews, or your help center are good candidates to keep open to crawlers since they can drive citations and referrals.

You can add this robots.txt file manually by:

• Using an SEO plugin like Yoast (Tools > File editor).
• Using a file manager plugin like WP File Manager.
• Or editing your robots.txt file directly on the server via FTP.

Option 2: Use a WordPress plugin

If you’re not comfortable editing the robots.txt file directly or just want a faster, safer way to manage AI crawler access, plugins can do the job for you with a few clicks.

Raptive Ads

The Raptive Ads WordPress plugin includes built-in support for blocking AI crawlers:

• You can toggle which bots to block directly from the plugin’s settings.
• Most AI bots (like GPTBot and Claude) are blocked by default.
• Google-Extended is not blocked by default, but you can check the box if you want to opt out of Google’s AI training.

One key benefit of using this plugin is that blocking Google-Extended does not affect your Google rankings or visibility in regular search results.

Block AI Crawlers

The Block AI Crawlers plugin was built specifically to give WordPress site owners more control over how AI crawlers interact with their content. Here’s how:

• Blocks 75+ known AI bots by automatically adding the right Disallow rules to your site’s robots.txt.
• No configuration is required. Install the plugin, go to Settings > Reading, and check the box labeled Block AI Crawlers.
• Lightweight and open-source, with regular updates pulled from GitHub.
• Designed to work out of the box on most WordPress installations.

The Block AI Crawlers plugin is one of the easiest ways to keep unwanted AI bots off your site, especially if you’re not using advanced SEO plugins.

Option 3: Use Cloudflare’s one-click AI bot Blocker

If your WordPress site uses Cloudflare (and many do), you can block dozens of known and unknown AI bots with a single toggle.

In mid-2024, Cloudflare launched a dedicated AI Scrapers and Crawlers feature, available even on the free plan. This feature doesn’t just rely on robots.txt; it blocks bots at the network level, even those that lie about who they are.

You can enable it by doing the following:

1. Log in to your Cloudflare Dashboard
2. Go to Security > Settings
3. Under the Filter by section, choose Bot traffic.
4. Find Bot fight mode and toggle it on.

If you’re using a paid Cloudflare plan, you have access to Super Bot fight mode, an enhanced version of Bot fight mode with more flexibility. It builds on the same technology but lets you choose how to handle different traffic types, enabling JavaScript detections to catch headless browsers, stealthy scrapers, and other malicious traffic.

For example, instead of blocking all crawlers, you can configure the tool to block only “definitely automated traffic” and allow “verified bots” like search engine crawlers:

That’s it. Cloudflare automatically blocks requests from AI bots.

If you want a deeper look at how these tools work together, including Bot Fight Mode, Super Bot Fight Mode, and targeted challenge rules, you can read our full guide on protecting your WordPress site from unwanted bot traffic with Cloudflare.

What this shift means for your WordPress site

AI crawlers are now part of how people discover information online. The technology is new, the rules are still forming, and site owners are deciding how much of their content they want to make available.

The good news is that WordPress sites are already in a strong position. Because WordPress outputs fully rendered HTML, most AI crawlers can interpret your content clearly without special handling. The real strategic decision isn’t whether AI crawlers can access your site — it’s how much access helps your goals.

And as the mix of traffic types evolves, it’s helpful to have hosting options that make resource usage easier to understand and manage. Kinsta’s new bandwidth-based plans offer a more predictable way to account for total data transfer, regardless of the source of the requests. Combined with Cloudflare’s bot protections and your own crawler rules, you have full control over how your site is accessed.

The post AI crawlers explained: How AI bots interact with your WordPress site appeared first on Kinsta®.