- AI crawlers like GPTBot and ClaudeBot do not execute JavaScript, meaning most sites built with tools like Cursor, v0, or Lovable return empty shells to the bots feeding ChatGPT and Perplexity.
- A December 2024 Vercel and MERJ server-log study found zero JavaScript execution across hundreds of millions of AI crawler fetches, confirming the gap between Google ranking and AI search visibility is structural, not content-related.
You handed an AI tool a brief, received a clean site in minutes, and launched with confidence. Three weeks passed. The traffic didn’t come. Most people blame the writing.
Wrong layer. Caleb Ulku, a seven-figure local SEO agency owner and the voice behind the AI SEO Mastery community with more than 3,000 members, says the real breakdown sits somewhere most site owners never think to check. In a video published on his YouTube channel, Ulku walks through four structural failures shared by most AI-built sites. The first failure is the kind that quietly stops everything else from working: AI search engines may have never read the site at all. “Nobody told the AI what Google actually needs to crawl your site, to understand it, and to trust it,” Ulku said.
A December 2024 server-log study by Vercel and MERJ put hard numbers behind what many had suspected. GPTBot alone sent 569 million requests across Vercel’s network in a single month, yet researchers recorded zero JavaScript execution across every major AI crawler they tracked. GPTBot downloaded JavaScript files in roughly 11.5% of its requests and never ran a single one. ClaudeBot fetched JavaScript in about 23.84% of requests, with the same result: nothing executed.
The practical consequence is direct: every headline, pricing table, case study, and service description that loads client-side simply doesn’t exist when GPTBot, ClaudeBot, or PerplexityBot arrives. They pull raw HTML, read whatever’s already in it, and move on. As of June 2026, that behavior covers every major AI crawler, including OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, Meta-ExternalAgent, and Bytespider. Google’s Gemini sits outside this group because it shares Googlebot’s rendering infrastructure. That exception doesn’t extend to the crawlers feeding ChatGPT or Perplexity.
Google’s bot will at least try to look at JavaScript. ChatGPT’s, Claude’s, Perplexity’s, they read the raw code, and that’s it.” Founders are spending months shipping AI-built products, then optimizing for AI search citations, from sites the AI systems can’t actually read. The solution recommended has no novelty to it: statically generated HTML, pages rendered fully on the server before any browser request lands. Google reads them without delay. AI crawlers read them the same way. Sites worked like this 20 years ago, and that architecture is quietly becoming the right answer again.
Four Technical Gaps That AI Builders Leave in Every Site
Getting the HTML right doesn’t finish the job. We have identified three more failure points that surface on almost every AI-generated site, each invisible to the owner, each actively limiting search visibility.
- Sitemaps and robots.txt. AI builders generate a sitemap at launch and never touch it again. Content gets added, but the sitemap remains unchanged, and the crawlers are not informed of this. The robots.txt file, which lets bots know where they can go, is often configured improperly, keeping entire chunks of the website away from crawlers and nobody the wiser. An audit of 6,944 websites carried out by SearchScore in July 2026 revealed 6.9% of websites were blocking one or more AI crawlers.
- Canonical tags and ghost pages. A homepage can technically sit at four separate addresses: with www, without it, with a trailing slash, without one. With no canonical tag naming the authoritative version, link equity splinters across all four URLs. Deleted pages that send the wrong status codes accumulate as ghost entries inside the index. Together, these two problems make Google’s and AI crawlers’ picture of the site progressively blurrier.
- Schema markup. Schema lives in the page code, invisible to visitors, built entirely for machines. It states facts explicitly: business name, address, phone number, hours, service types, location category. Without it, AI systems reconstruct what a business does whatever the marketing copy implies. That reconstruction is sometimes wrong. We have measured the limits here: schema is not a confirmed ranking factor, and no AI company has publicly confirmed it drives citations. Google and Microsoft both confirm it helps them understand pages accurately. The argument for adding it is clarity, not a hack.
- Content that serves visitors but not search engines. AI builders open pages with mission statements. Search engines and AI crawlers reward pages that answer the query in the first paragraph. Title tags like “Solutions for your home comfort needs” match nothing anyone who searches for it. Internal linking carries its own problem: AI builders typically wire everything through the navigation menu, producing a flat mass of links rather than a structured hierarchy that shows crawlers how pages relate to each other.
Google Rankings and AI Visibility Are Now Two Different Problems
Most SEOs haven’t fully processed this yet. Google eventually renders JavaScript through its headless Chromium pipeline. A page can rank well in Google while being completely invisible to every AI crawler that doesn’t execute JavaScript. SEO consultant Glenn Gabe’s August 2025 testing confirmed this gap directly: fully client-rendered sites appeared blank to ChatGPT, Perplexity, and Claude during his analysis. Traditional search performance and AI search visibility now have separate technical requirements. Success in one no longer implies anything about the other.
GPTBot accounted for an estimated 12% of worldwide bot traffic and ClaudeBot for 9.2% according to Cloudflare Radar data for the period of twelve months before early 2026, putting ClaudeBot on the same level as Bingbot. The same source showed that large AI bots followed robots.txt files 100% of the time. A misconfigured robots.txt isn’t a minor technical debt. It’s a full block on AI visibility, enforced automatically and consistently on every crawl.
What Most Coverage Gets Wrong About This Problem?
A well-sourced, clearly written page inside a client-rendered React application is still invisible to GPTBot and ClaudeBot. The quality of the prose becomes moot when the crawler gets fed an empty shell. The technical GEO analysis conducted by SearchScore in July 2026 puts robots.txt, sitemaps, schema, and rendered HTML into the four-layer foundation upon which AI visibility depends. The content strategy comes later. If the crawler cannot access the page, no content on it will get read.
Does AI Visibility Even Matter?
However, Google ranking and visibility through AI technology seem to be developing as two distinct technical problems; however, there is growing evidence of the business importance in addressing them. For instance, Shopify recently discovered that users coming through AI technologies have almost 50% higher conversion rates compared to those coming from organic searches, as well as spending more on average per purchase. That makes technical accessibility more than an indexing issue. If GPTBot or ClaudeBot cannot read a site’s content, businesses may miss the opportunity to appear in AI-generated recommendations that increasingly influences purchase decisions. Read our analysis of Shopify Says AI-Referred Shoppers Convert Better and Spend More.
Three Tests That Take Under Two Minutes Each
Start with the raw page source, not the DevTools Elements panel, which shows content after JavaScript runs. Right-click the page in Chrome, select “View Page Source,” and search for a sentence from your main copy. If it’s in the source, AI crawlers can read it. If the body is an empty div and a script tag, they can’t. Run a curl fetch against your URL to replicate exactly what an AI bot receives. Then open ChatGPT or Perplexity and ask what your company does. If the answer is vague or pulls no detail from your actual site, your content isn’t reaching those systems.
Sites already built in tools like Cursor, v0, or Lovable have three realistic paths: migrating to Next.js with server components, adding a prerendering layer through a service like Prerender.io, or rebuilding the deploy pipeline around static site generation. Each path carries its own cost and development time. The cleaner long-term answer is a platform that ships server-rendered HTML without requiring any of those workarounds.
Schema, canonical tags, and a properly configured robots.txt are not optional refinements. In 2026, they are the floor for any site that intends to appear in AI-generated answers. Most AI builders omit all three. Owners don’t catch it because nothing looks broken in the browser.
The distance between a site that looks finished and a site that AI search systems can actually process is larger than most people expect. Closing it starts with one honest question: what does an AI crawler see when it arrives? On most AI-built sites right now, the answer is an empty page.
