How AI crawlers find and cite your brand

If you've checked your server logs lately, you've probably noticed traffic from names you don't recognise: GPTBot, ClaudeBot, PerplexityBot, Google-Extended. These are AI crawlers, and there are more of them visiting your site every month than there were a year ago. The question worth asking isn't just "are they visiting", though. It's whether any of that crawling ever turns into your brand actually being named the next time someone asks ChatGPT or Perplexity for a recommendation in your category.

Those are two different things, and mixing them up is the most common mistake we see. A bot visiting your page is a crawl. Your brand appearing, correctly, in an AI-generated answer is a citation. You can have plenty of the first and almost none of the second. This post is about how the pipeline between them actually works, so you can tell which part of it you're missing.

Crawling, training and retrieval aren't the same process

AI assistants get information about the world through three distinct routes, and each one behaves differently.

Training data is the broadest and slowest-moving. Large language models are trained on a snapshot of the web collected up to some cutoff date, often built on top of Common Crawl and supplemented with the model provider's own crawl. If your page existed and was crawlable before that cutoff, it might be baked into the model's underlying knowledge. If you launched or changed a fact afterwards, the model simply doesn't know it, no matter how well-optimised the page is.

Live retrieval is what actually powers most of the AI answers people get today. Tools like Perplexity, ChatGPT with browsing turned on, Copilot and Gemini's grounded responses don't just recall training data, they run a live web search at the moment you ask, pull back a handful of pages, and have the model summarise from those pages in real time. This is the route where fresh, well-structured content has the fastest path to actually being cited, because it's read and used within seconds of the query, not baked in months ago.

Product-side retrieval-augmented generation (RAG) is a variant of the above that companies build for their own tools: a search index the model queries before answering, rather than the open web. It behaves like live retrieval but over a narrower, curated set of sources.

The practical takeaway: if you only think in terms of "getting crawled for training", you're optimising for the slowest and least controllable route. Most of the near-term opportunity is in the second one, being a good candidate for live retrieval, because that's the loop that runs every time someone asks a question, not once a year at a training cutoff.

The AI crawlers worth knowing by name

Each major AI provider runs its own crawler, and most respect robots.txt, so the first thing worth checking is whether you're accidentally blocking them. As of this year, the main ones to know are OpenAI's GPTBot (training) and OAI-SearchBot and ChatGPT-User (used for search and live browsing), Anthropic's ClaudeBot and Claude-User, Perplexity's PerplexityBot and Perplexity-User, Google's Google-Extended (a separate opt-in from classic Googlebot, governing whether Gemini and AI Overviews can use your content), and Microsoft's Bingbot, which underpins Copilot. Amazon and Apple run their own too (Amazonbot, Applebot-Extended). Providers add and rename these periodically, so if this matters to you, it's worth checking each provider's current published list rather than treating this one as permanent.

A quick, free check: open your site's robots.txt and see whether any of these are explicitly disallowed. Plenty of sites block them by accident, often because a boilerplate robots.txt was copied from a generic SEO guide written before these bots existed, or because a well-meaning developer blocked "all bots except the ones I recognise" and never updated the list. Blocking them is a legitimate choice if you genuinely don't want AI systems using your content, but it's a trade: no crawl access means no citation, full stop, in that route. Babel42's own robots.txt allows every user agent everywhere except /api/ and /mocks/, on the view that the upside of being citable outweighs the cost of being crawled.

Crawling is necessary, citation is the actual goal

Getting crawled is a low bar. Search engines and AI bots alike will happily crawl thin, outdated or badly-structured pages, they just won't have anything worth citing when they get there. Citation is a separate, higher bar: the retrieval step has to judge your page relevant to the specific question, and the generation step has to trust it enough to lift a fact or a quote from it into the answer.

What seems to raise that trust, based on what we and others have observed watching real AI-generated answers:

• Plain, quotable statements beat vague ones. "Free plan: 500 mentions a month, 2 monitors, 5 networks" is something a model can lift whole into an answer. "Generous limits on our free tier" gives it nothing concrete to cite, so it either paraphrases vaguely or skips you for a competitor who stated a number. We've written before about why numbers beat adjectives when it comes to being cited rather than just mentioned.
• The fact has to be readable without extra steps. A price buried in a PDF, gated behind a "book a demo" form, or rendered client-side in a way that a crawler can't easily parse, is effectively invisible to this whole pipeline, even though a human visitor would find it fine.
• A dedicated llms.txt file gives retrieval a clean summary to work from. This is a plain-text file at the root of a domain, an emerging (not yet universal) convention that spells out who you are and what you do in a few short, factual lines, aimed at machines rather than search-engine ranking. It doesn't replace your normal pages, but it's a low-effort way to hand a model an unambiguous summary instead of making it infer one from marketing copy. Ours is public if you want to see the shape of it: what Babel42 does, the free-plan numbers, the pricing range, and links to the pages that back each claim up.
• Freshness matters more for live retrieval than for training. A page that was accurate a year ago but hasn't been touched since is a weaker retrieval candidate than one with a recent, visible update, particularly for search-grounded tools that lean on recency as a relevance signal.

Why this is the same problem as AI search visibility, from a different angle

Everything above is really the technical half of a question we've written about from the buyer-behaviour half before: what AI search visibility actually measures. Crawling and citation are the plumbing; appearance rate, share of AI voice and recommendation sentiment are what that plumbing produces when you measure the output instead of the mechanism. A site that's fully crawlable but says nothing concrete about itself will score badly on both. A site that's specific and citable but accidentally blocks half the crawlers that matter will also score badly, for the opposite reason.

That's also why the citation data is worth watching directly, not just inferring from traffic. Babel42's AI Visibility product tracks exactly this: which sources get cited when an AI Buyer's shopping journey reaches an answer, and how big a share of those citations point back to you versus a competitor.

In the example above (a demo workspace shopping for email marketing platforms), you can see the gap between two of the dashboard's metrics: the brand under test appears in 100% of journeys, but holds only a 24% share of mentions across the category (341 of 1,397) and just a 10% share of citations (93 of 896). Its own domains do show up in the top-cited list, just behind independent review sites like emailtooltester.com and omnisend.com. That drop from appearing everywhere, to a modest share of mentions, to a smaller share of citations still, is the crawl-versus-citation distinction above playing out as a number: closing it means making your own pages the most quotable version of the facts that independent sites are currently getting cited for instead.

What to actually check this week

If you want a short, practical list rather than the theory:

1. Open robots.txt and confirm you're not blocking GPTBot, ClaudeBot, PerplexityBot, Google-Extended or Bingbot by accident.
2. Pick your three or four most important factual pages (pricing, what you do, key differentiators) and check that the core facts are in plain, crawlable text, not locked behind a form or rendered in a way a bot can't read.
3. Rewrite the vaguest sentence on your pricing page as a specific number.
4. Consider publishing a short llms.txt, ours is a reasonable template to start from.
5. Then measure whether any of it is moving the needle: ask the same buyer questions across a couple of AI assistants before and after, and watch whether your own domain starts showing up in the citations instead of a competitor's.

None of this replaces the slower work of earning genuine third-party coverage, that's still the biggest lever for the sources you don't control. But it's the part you can fix yourself, this week, without waiting on anyone else.

See how Babel42 tracks appearance rate, citations and share of AI voice, or start with what AI search visibility actually measures if this is your first stop.