Your analytics say traffic is flat. Your server says it’s busier than ever. Both can be true, because the visitor growing fastest on most websites in 2026 isn’t a person. It’s a crawler collecting pages for an AI model.
How much traffic is now AI bots?
More than most site owners assume:
- Bots made up 57.4% of HTML requests on Cloudflare’s network in the week ending 5 June 2026, according to Cloudflare Radar data cited in Kinsta’s report.
- AI bots went from about 1 in 200 web visits to 1 in 31 during 2025, according to TollBit.
- About 80% of AI crawling is for model training, not for answering a user’s question in real time.
On small business sites the gap with search engines is stark. One 2026 analysis found AI crawlers made 3.1 requests for every one from Googlebot on the median small site.
Who is knocking
More than half of HTML requests come from bots
Why bots cost more than they look
A crawler reading a blog post is usually cheap, because your host can serve that page from cache. The problem is pages that can’t be cached.
Kinsta analysed more than 10 billion requests across its WordPress hosting and found one AI crawler sent 3.75 million requests to a single shopping cart page in 24 hours. That’s roughly one request every 23 milliseconds. Across all bots, add-to-cart URLs were hit 7.67 million times in the same day.
Kinsta data, June 2026
Requests to WooCommerce add-to-cart URLs in 24 hours, by bot
Every one of those requests skips the cache, starts PHP, queries the database and creates a session. None of them buys anything. On shared hosting, that load can slow your site for real customers or push you into your plan’s resource limits.
Should you block AI crawlers?
Not all of them. There are three kinds, and they deserve different treatment:
| Type | Examples | What it does | Our suggestion |
|---|---|---|---|
| AI search | OAI-SearchBot, PerplexityBot, Claude-SearchBot | Finds pages to cite in AI answers, often with a link | Allow |
| User-triggered | ChatGPT-User, Claude-User | Fetches a page because a person asked about it | Allow |
| Training | GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent | Collects pages to train future models | Block if you don’t want to contribute |
Blocking everything can remove you from AI-powered search results without saving much. Blocking training crawlers while allowing search bots keeps you visible and cuts the heaviest crawling.
Step 1: Find out who’s crawling you
In cPanel, open Metrics → Awstats and look at the “Robots/Spiders visitors” table. It lists the bots that visited, how many hits they made, and how much bandwidth they used.
On a VPS, you can count requests by user agent straight from your access log:
awk -F'"' '{print $6}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20
Change the log path to match your server (for example /usr/local/apache/domlogs/ on cPanel servers). If GPTBot or ClaudeBot sits near the top, you’ve found your bandwidth.
Step 2: Tell polite crawlers what to skip
The major AI companies say their crawlers respect robots.txt. Add this to the file in your site’s root folder:
# Keep every bot away from store pages that can't be cached
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*add-to-cart=
Disallow: /*?s=
# Block AI training crawlers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Bytespider
Disallow: /
# Opt out of Google and Apple AI training (normal search is unaffected)
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
Adjust the store paths to match your shop. The Google-Extended line doesn’t stop Googlebot from indexing you; it only tells Google not to use your content for its AI models.
Step 3: Enforce it for crawlers that don’t listen
robots.txt is a request, not a lock. Some crawlers have been reported to ignore it. On LiteSpeed or Apache hosting, you can refuse them in your .htaccess file:
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|CCBot|Bytespider|Meta-ExternalAgent) [NC]
RewriteRule .* - [F,L]
</IfModule>
Place it above the WordPress block. Keep a backup of the file first, and test your site afterwards.
Step 4: Make the remaining crawls cheap
- Turn on full-page caching. LiteSpeed Cache serves most pages without touching PHP, so a crawler reading your blog costs almost nothing.
- Use your firewall’s bot protection. Imunify360 and services like Cloudflare can challenge unverified bots before they reach WordPress.
- Don’t block Googlebot from your whole site. If it’s hitting cart URLs, block those paths, not the bot.
What we do on LogicWeb hosting
Every web hosting and WordPress hosting plan runs on LiteSpeed with LSCache, Imunify360 and CloudLinux account isolation. Caching keeps crawler visits cheap, Imunify360 filters malicious traffic, and CloudLinux stops one busy account from slowing its neighbors.
Transfer is unmetered on every plan, so a crawler won’t land you with an overage bill. Your CPU, memory and process limits still apply, and that’s where heavy bot traffic shows up. If you see high usage you can’t explain, open a ticket and we’ll look at your logs with you.
The bottom line
AI crawlers aren’t going away, and some of them will become a real source of visitors as more people search through AI assistants. Treat them like any other traffic: let in the ones that send people back, keep everyone off the pages that cost money to serve, and check your logs once a month. Ten minutes with your robots.txt today can take a real load off your server.
FAQ
Does blocking AI crawlers hurt my SEO?
Blocking AI training crawlers like GPTBot or ClaudeBot doesn’t affect Google or Bing rankings. Blocking AI search bots like OAI-SearchBot or PerplexityBot can stop your pages from being cited in AI answers, so we suggest allowing those.
Will robots.txt stop all AI bots?
No. It works for crawlers that choose to respect it, which includes the major AI companies’ official bots. For crawlers that ignore it, use .htaccess rules, your hosting firewall or a CDN’s bot protection.
Why are bots hitting my WooCommerce cart?
Cart and add-to-cart links appear on product pages, so crawlers follow them like any other link. Those URLs can’t be cached, which makes each visit expensive. Disallow them in robots.txt and block them at the firewall for unverified bots.
How do I see bot traffic in cPanel?
Open Metrics, then Awstats, and check the “Robots/Spiders visitors” table. Raw Access logs show every request with its user agent if you need more detail.
