- AI bot traffic is no longer just a security or SEO issue. It is becoming an infrastructure problem for website owners.
- Kinsta says inefficient crawlers are hitting dynamic WordPress endpoints, creating loops and consuming real server resources.
- The bigger question is no longer whether bots are good or bad. It is which bots should be allowed, where they should go and what their access is worth.
AI crawlers are no longer just sitting quietly in the background of the web.
A new Kinsta analysis based on more than 10 billion requests across managed WordPress infrastructure suggests that bot traffic is becoming a real operational issue for site owners, publishers and agencies.
The report argues that the main problem is not simply that more bots are visiting websites. The bigger issue is how they behave.
According to Kinsta, crawlers are increasingly hitting dynamic endpoints, following endless URL parameter variations, bypassing cache and creating traffic patterns that look less like normal indexing and more like broken automation at scale.
That matters because every request has a cost. On a simple static page, traffic can often be handled cheaply through cache. On a WordPress or WooCommerce site, bot requests can trigger PHP execution, database queries, session handling, cart logic and other expensive server-side work.
At scale, that turns bot traffic from a visibility question into an infrastructure cost.
AI Bots Are Now a Measurable Traffic Layer
Kinsta’s report pulls together infrastructure data and outside research to show how quickly AI-driven bot activity has grown.
The report cites Akamai research showing a 300% surge in AI bot traffic over one year. It also points to TollBit data suggesting that by the end of 2025, roughly 1 in 31 web visits on its network came from an AI bot. At the start of the same year, that figure was closer to 1 in 200.
Cloudflare has seen a similar shift from another angle. In its 2025 Radar Year in Review, the company said traffic from AI bots accounted for an average of 4.2% of HTML requests across the year, with the share moving sharply across different periods.
For many site owners, this is easy to miss.
Analytics dashboards may show traffic growth, but not all of that traffic is human. Some of it is traditional search crawling. Some of it is AI training. Some of it is retrieval activity. Some of it is badly behaved automation that provides little obvious value to the site being crawled.
That creates a new problem for marketers and publishers. The old question was whether bot traffic was good or bad. The new question is more specific: which bots should be allowed, which should be restricted and which parts of a site should they never touch?
The WordPress Problem: Bots Are Hitting Expensive URLs
The most striking part of the Kinsta report is not about content scraping. It is about infrastructure strain.
Kinsta says it observed 7.67 million requests to add-to-cart URLs within 24 hours. ClaudeBot alone accounted for 3.75 million of those requests.
That is where “bot traffic” stops sounding abstract.
Add-to-cart URLs are not ordinary content pages. They are dynamic actions. They can bypass cache, trigger sessions and force server-side processing. A crawler hammering those URLs is not just reading a page. It is asking the site to do unnecessary work again and again.
Kinsta also says one loop rule filtered 550 million requests in 30 days. That suggests the problem is not only volume. It is repetition.
A crawler may treat every URL variation as a new page, even when a human would see the same product page with a different filter, color, sort order or parameter. On modern websites, especially ecommerce sites, that can create a near-endless crawl space.
This is not only a hosting issue. It also connects directly to technical SEO.
Google has warned that faceted navigation and parameter-heavy URLs can create crawl inefficiency because crawlers may spend time exploring large numbers of low-value URL variations. Google’s documentation also notes that this can slow discovery of more useful URLs.
That old SEO problem now has a new AI-era cost attached to it.
This Is Not Always Malicious
One important point from the report is that much of this activity is not necessarily an attack.
Some crawlers may simply be inefficient. They follow links, encounter parameter-heavy URLs and keep moving through variations without understanding that they are wasting resources.
That distinction matters because it changes the response.
If all bot traffic is treated as hostile, site owners may block systems that help with search discovery or AI visibility. If all bot traffic is allowed because “AI is the future,” websites may pay the infrastructure cost for crawlers that provide no clear return.
Neither extreme is useful.
The better approach is more selective. Search crawlers may need access to product pages, category pages and important content. They do not need unrestricted access to cart URLs, checkout paths, internal search pages, wishlist actions or endless filtered URL combinations.
That is where simple controls still matter. Google’s robots.txt guidance explains that the file can tell crawlers which URLs they can access, mainly to avoid overloading a site with requests. It is not a complete content protection system, but it remains one of the basic ways site owners manage crawler access.
Why This Matters for SEO and AI Search
For SEO teams, the Kinsta report is a reminder that crawl behavior is still a technical issue, even in the AI search era.
The same URL structure that creates crawl waste for Googlebot can also trap AI crawlers, SEO tools and unknown bots. That means technical SEO, log analysis, robots.txt rules, canonical handling and parameter management may become more important again.
This also connects to the wider AI visibility debate. As The Query Post has covered before, Google says AI search optimization is still built on SEO fundamentals. But the wider ecosystem is more complicated. ChatGPT, Claude, Perplexity and other AI systems can create new discovery surfaces where visibility may not always map neatly to classic rankings.
That is why bot access is becoming part of the AI search conversation.
Blocking every AI crawler may reduce scraping and server load, but it could also limit how often a site is discovered, referenced or cited by AI systems. Allowing every bot may increase visibility potential, but it can also create costs that do not show up clearly in marketing reports.
This is the uncomfortable middle ground.
Website owners are no longer only optimizing for rankings and clicks. They are also deciding how much access they want to give to systems that may summarize, reuse or cite their content without always sending traffic back.
That fits with a broader problem The Query Post recently covered around AI search visibility gaps that traditional SEO tools struggle to explain. If AI systems become another layer between users and websites, then crawler behavior, citation visibility and actual business outcomes all start to overlap.
Bot Traffic Can Also Distort Analytics
The Kinsta report also points to a less obvious problem: measurement.
If automated traffic inflates visits, pageviews or request counts, website owners may think a site is growing when part of that growth is not human demand at all.
That is dangerous for publishers and marketers because traffic numbers often influence business decisions. Teams use them to judge content performance, audience growth, campaign success and even company value.
But raw traffic becomes less useful when more of it comes from bots.
The better signals are the ones that still connect to real demand: branded search, direct visits, engaged sessions, newsletter signups, qualified leads, sales and revenue. If those numbers rise alongside traffic, the growth is more meaningful. If only request volume rises, the site may simply be absorbing more automated load.
This also matters for publishers trying to understand the value of AI visibility. A site may receive more bot visits without receiving more human visitors. It may be scraped more often without being cited more often. It may be cited more often without getting more clicks.
That is why AI citation visibility needs to be judged carefully. Being visible inside AI systems may matter, but the industry still has a measurement problem when it comes to tying citations to business results.
The Real Question Is Control
Kinsta frames the issue less as a simple block-or-allow decision and more as a control problem.
A WooCommerce store does not have the same risk profile as a media site. A local service business does not have the same crawling needs as a large ecommerce catalog. A staging environment should not be treated like a public website.
The right bot policy depends on the site.
For ecommerce sites, the priority may be protecting cart, checkout and filtered URLs. For publishers, the concern may be AI scraping and content reuse. For SaaS or web apps, the concern may be automation hitting app-like functionality. For agencies, the challenge is building a repeatable framework that can be adapted across different clients.
The practical message is simple: site owners need visibility before they make decisions.
They need to know which bots are hitting the site, which paths they are requesting and whether those requests create value or just load.
This is also where performance and SEO start to meet. Core Web Vitals, server response times and crawl efficiency are not separate from the bot problem. If automated traffic is hammering expensive URLs, performance issues can spill over into the real user experience. For site owners already working on speed, The Query Post’s Core Web Vitals guide covers the user-facing side of that performance equation.
The Bigger Picture
The rise of AI bot traffic is part of a wider shift in how the web works.
For years, publishers and businesses optimized for human users arriving through search engines, social platforms, newsletters and direct visits. Now, more automated systems are sitting between content and users. Some crawl for search. Some crawl for training. Some crawl for AI answers. Some act on behalf of users. Others are poorly built tools running at scale.
The next stage may be even messier.
Google has already introduced a separate Google-Agent user agent to identify some AI agent activity. That points to a future where automated systems do not just crawl pages. They may browse, compare, fill forms, trigger workflows and take actions on behalf of users.
For website owners, that means bot strategy cannot stay limited to blocking bad scrapers. It has to include decisions about agents, AI crawlers, search bots, analytics noise, content reuse and infrastructure costs.
The connection to security is also growing. As The Query Post recently reported, AI chatbot results are already becoming a target for malware campaigns. The same AI-driven discovery systems that can surface useful information can also become part of more complicated abuse patterns.
The Query Post View
Kinsta’s report is useful because it moves the AI bot discussion away from panic and toward practical trade-offs.
The answer is not to block everything. That may protect server resources, but it can also damage search visibility and reduce the chance of being discovered by useful systems.
The answer is also not to let everything in. That may sound open and future-friendly, but it can leave websites paying for crawlers that hit expensive URLs, distort analytics and provide little obvious value.
This is not only a theoretical issue. We recently saw the same thing on a completely new website. There was no big audience yet, no viral article and no real traffic breakthrough. But the bot traffic was already high enough to push the site into a hosting upgrade much earlier than expected. That is when the problem stops being abstract. You suddenly pay more for traffic that is not even real users.
Bot traffic can also make performance data harder to trust. If automated visits are included in analytics, metrics such as engagement time, session quality and pageview growth can look healthier or stranger than they really are. A site may appear to be getting more activity, while the actual human audience has barely changed.
For site owners, the first step is to check hosting logs or server reports instead of relying only on analytics. The important question is not just how much traffic came in, but which user agents requested which URLs.
Start with the expensive paths. Crawlers usually do not need access to ?add-to-cart= URLs, checkout pages, cart actions, login pages, internal search results, wishlist actions, staging URLs or endless filter combinations. These are the areas where bots can create server load without creating search or AI visibility value.
Robots.txt can help guide crawlers away from low-value areas, but it should not be treated as real enforcement. For aggressive bots, unknown crawlers or repeated hits to dynamic URLs, the stronger controls usually need to happen at the hosting, CDN or WAF level through blocking, challenging or rate limiting.
Search crawlers need more care. Googlebot and Bingbot should still be able to reach important articles, product pages, category pages and commercial landing pages. The goal is not to cut off discovery. The goal is to stop crawlers from wasting resources on URLs that should never matter.
AI crawlers should be handled separately. If a site wants visibility in AI-generated answers, it may make sense to allow access to useful editorial and commercial content. But there is little reason to let AI training crawlers hit cart flows, checkout paths, search pages or other high-cost endpoints.
A practical cleanup should start with these checks:
- Review the top user agents in hosting logs.
- Find the most requested URLs and separate content pages from dynamic paths.
- Block or restrict cart, checkout, login, search, staging and filter URLs where appropriate.
- Keep Googlebot and Bingbot open for important indexable pages.
- Challenge or rate-limit aggressive AI crawlers and unknown bots at the CDN or WAF level.
- Compare traffic growth with branded search, direct visits, leads, sales, newsletter signups and real user engagement.
The key is not to block the most traffic. The key is to understand which automated traffic has value and which traffic is only creating cost, noise and unreliable data.
