The Technical SEO Checklist for Getting Found in AI Search

Your next customer might never see a page of Google results. They will ask ChatGPT, Claude, Perplexity or Google’s AI Mode a question, get a written answer with a handful of cited sources, and shortlist from those. If your website is one of the citations, you are in the conversation. If it is not, you do not exist.

The good news is that getting cited is not a dark art. Under the surface, AI search leans heavily on the same boring, unglamorous technical foundations that classic SEO has always relied on. Most businesses fail at AI visibility not because they lack clever “AI optimisation”, but because a crawler turned up, found a slow site full of JavaScript, and left.

One thing before we start. We are an agency that sells SEO and AEO services, so yes, we benefit if this post convinces you the topic matters. We would rather say that plainly than pretend otherwise. Everything below is checkable for free, and most of it you can action without ever speaking to us.

How does AI search actually decide who to cite?

It helps to understand the machinery, because it explains every item on the checklist that follows.

When you type a question into a Google search box, Google matches it against an index it has spent twenty years building, then ranks the results. AI assistants work differently. When you ask ChatGPT a question that needs fresh information, it does not run one search. It breaks your question into several sub-queries, a process often called query fan-out. Ask “who’s the best accountant for a small construction firm in Bristol?” and behind the scenes it might search for construction industry accountants, Bristol accountancy firms, CIS tax specialists and small business accounting reviews, all at once.

Each sub-query pulls back candidate results, often a hundred or more. The retrieval is semantic: it works on topics and entities rather than exact keyword matches. The model then re-ranks that pool against what an ideal answer to your question would contain, and builds its response, citing the sources that survived.

The practical point is this: ranking third on Google for one keyword and being cited by an AI assistant are related but different games. The AI is assembling an answer from many angles at once, and it rewards sites whose content is easy to retrieve, easy to read by machine, and clearly about something.

Does ChatGPT actually crawl your website?

Mostly, no. This is the single most misunderstood fact in AI search, and it reshapes everything.

Crawling the entire web is enormously expensive, and it is not the AI companies’ business model. So when an assistant needs live information, it typically queries the big search engines’ infrastructure and gets back what a search API returns: a title, a description and a short extract for each result. If that snippet is enough to support the answer, the assistant can cite your website without ever visiting it. Only when the snippet does not answer the question will it send its own crawler to fetch the page, and that crawler is far simpler than Googlebot.

Two consequences follow, and they anchor the whole checklist:

1. If Google and Bing cannot crawl and index you properly, you effectively do not exist to AI search. The assistants are drinking from the search engines’ well. Classic technical SEO is not the old game being replaced by AEO; it is the foundation AEO stands on.
2. Your titles, meta descriptions and opening extracts are doing more work than ever. They are frequently the only part of your page an AI system reads before deciding whether to cite you.

With that understood, here is the checklist. It is written for business owners and marketing managers, not developers. For each item there is a plain-English way to check it yourself, and every item can be handed straight to whoever looks after your website.

1. Are you actually indexed by Google and Bing?

Everything else is decoration if the answer is no. Search site:yourdomain.co.uk on both Google and Bing and see what comes back. It should look roughly like your site: your key pages present, no strange duplicates, no staging versions. Better still, get access to Google Search Console and Bing Webmaster Tools, both free, and look at the indexing reports. Pages listed as “crawled, not indexed” or “discovered, not indexed” are pages AI search cannot cite.

Do not skip Bing because “nobody uses Bing”. ChatGPT’s browsing has long leaned on Bing’s index. A site that is healthy on Google but invisible on Bing has a real AI search problem.

2. Is your robots.txt blocking AI crawlers?

Your robots.txt file (at yourdomain.co.uk/robots.txt, anyone can view it) tells crawlers what they may and may not fetch. We have seen businesses block AI crawlers without ever deciding to: a blanket disallow left over from a site launch, a security plugin being overzealous, or a developer who added the blocks in 2023 when “stop the AI scrapers” was the fashionable advice and nobody has revisited it since. We have also seen the genuinely painful version: someone writing Disallow where they meant Allow.

Open yours and look for these user agents:

If any of these appear next to a Disallow: /, that is a decision someone made, deliberately or otherwise, to be invisible to that system. And to be clear about the default: an empty or absent robots.txt blocks nobody. You do not need to add allow rules; you need to make sure nobody has added block rules.

3. Is your important content in the actual HTML?

This is the big technical one, so bear with the one paragraph of jargon. Websites can deliver content two ways: baked into the HTML the server sends (server-side rendering), or assembled in the visitor’s browser by JavaScript after the page loads (client-side rendering). Human visitors cannot tell the difference. Crawlers can. Googlebot will usually execute the JavaScript eventually. AI crawlers will not. An analysis by Vercel and MERJ of hundreds of millions of crawler requests found no evidence that GPTBot, ClaudeBot or PerplexityBot execute JavaScript at all. They read the raw HTML, and anything that is not in it does not exist for them.

The check takes ten seconds and no technical knowledge: open your key page, right-click, choose “View page source”, and Ctrl+F for a distinctive sentence from the page. If it is there, good. If the source is a near-empty shell of script tags, your content is invisible to the crawlers that matter here, and “make our key content server-rendered” is the instruction to give your web company. It is a solved problem on every mainstream platform; a properly built WordPress site does this by default.

4. Is your site fast and reliable when the bots call?

AI crawlers are impatient. They arrive with tight time budgets, and a page that responds slowly may simply be skipped. Research shared with us covering a couple of thousand websites found the faster-responding sites were cited roughly 1.4 times more often than slower equivalents, and while we would hold any single number loosely, the direction matches everything we see: speed is a citation factor, not just a conversion factor.

There is a blunter version of the same point. If your server happens to be down, or crawling at a snail’s pace, at the moment an AI crawler comes to verify a page, you lose that citation. Nobody retries on your behalf.

Run your key pages through PageSpeed Insights, which is free. Passing Core Web Vitals is a reasonable proxy for “fast enough”, and the server response time figure near the top of the report is the one AI crawlers feel most. Cheap, overloaded shared hosting is where this check most often fails.

5. Is 80% of your crawled site junk?

Search engines give every site a crawl budget: a rough allowance of how many pages they will bother fetching. If most of that budget is burned on duplicate pages, thin tag archives, endless filtered URLs and parameter bloat, two bad things happen. Your good pages get crawled less often, and the whole domain’s quality assessment sags, because individual pages are judged partly on the company they keep.

The upside of fixing this can be dramatic. One case we heard discussed recently involved a heavy JavaScript site that was rebuilt to be server-rendered and lighter: its crawl rate went from around 8,000 pages a day to over a million, and its AI visibility improved simply because far more of the site became available to cite.

You will not audit crawl budget yourself, and that is fine. The question to put to your web company is: “What proportion of the URLs Google crawls on our site are pages we actually want found?” If they cannot answer it, that is an answer too.

6. Do your server logs show AI bots visiting?

Your server logs record every visit to your site, including every bot, and they are the only place you can see AI crawler behaviour directly rather than guessing. Ask whoever hosts your site to check the logs for the user agents in the table above, looking for four things:

• Important pages with zero bot visits. Absence of signal is the finding here. A service page no AI bot has ever fetched is a page AI search cannot quote from.
• Orphan pages that get bot traffic but are linked from nowhere, or vice versa.
• Bots hitting 404s and broken pages, which wastes their patience and your crawl budget.
• On JavaScript-heavy sites, whether bots fetch the HTML but never the content it depends on, which confirms the rendering problem from item 3.

One platform note, offered factually: hosted platforms such as Shopify generally do not give you access to raw server logs, so this whole diagnostic layer is closed to you. A WordPress site on your own hosting exposes it fully. We build on WordPress, so make of our motives what you will, but the logs point stands either way.

7. Are your sitemaps healthy, for the right reasons?

Here is a myth worth killing: AI companies do not crawl your XML sitemap to build their own index of your site, because they are not building an index. Submitting a sitemap “to ChatGPT” is not a thing.

Keep your sitemaps clean anyway, because Google and Bing do use them, and theirs are the indexes AI search leans on. A sitemap full of 404s, redirects and noindexed URLs quietly tells the search engines your site is carelessly run. Search Console and Bing Webmaster Tools will both flag sitemap errors for free.

8. Should you bother with llms.txt?

llms.txt is a proposed standard: a plain-text file summarising your site for AI systems. Our honest read is that there is no solid evidence the AI companies respect it, and as the pipeline described earlier shows, it is not how their retrieval actually works. They are querying search indexes, not reading courtesy files.

Implement it anyway. It takes half an hour, carries zero risk, and if the standard does get adopted you are already there. Just do not let anyone sell you an “llms.txt optimisation package” as if it were the strategy. It is a lottery ticket, priced accordingly.

9. Is your site ready for AI that does things, not just reads things?

The forward-looking one. The next phase of AI search is agentic: assistants that do not just recommend a supplier but complete the task, filling in the enquiry form, making the booking, adding to the basket. Google, Shopify, Stripe and PayPal are all building agent payment and commerce protocols right now.

Agents interact with your site through its underlying structure, much as screen readers do. A real HTML <button> announces itself as something clickable. A styled <span> that merely looks like a button announces nothing, so an agent cannot press it, and the sale walks. The same goes for real forms, labelled fields and properly marked-up navigation. Semantic, machine-readable HTML has gone from an accessibility nicety to a revenue question, and it is far cheaper to build in than to retrofit. If a rebuild is on your horizon anyway, put “semantic HTML throughout” in the brief.

What should you do with this list?

Run the free checks yourself this week: the site: searches, robots.txt, view-source, PageSpeed Insights. Then hand the rest, crawl quality, server logs, sitemap health, to whoever built your website, and ask for findings in plain English. None of this is exotic. It is competent technical SEO, done with an eye on who is reading the results now.

And if you would rather someone ran the whole thing for you, that is a service we offer. We are Grizzly, a Bristol web design and SEO agency. We have been building and optimising websites for over ten years, we hold 100+ five-star Google reviews, and we rank #1 for “web design bristol”, which at least suggests we follow our own advice. Our Answer Engine Optimisation guide covers the content side of getting cited; this checklist is the technical side.

Ask us for an AI search visibility check and we will run this list against your site, show you exactly what the crawlers see, and give you a straight answer on what is worth fixing. The customers asking ChatGPT for recommendations are already out there. The only question is whose website gets quoted back to them.