A material study in bright daylight. A half circle fan of clear turquoise glass blades spreads from a small point at the lower right, bound along its outer edge by a polished pearl silver rim.
  1. Home
  2. Insights
  3. Search and AI Visibility
  4. Getting cited by AI

Technical explainer

How a business getscited by AI assistants.

An assistant cites a page it can fetch, parse, and lift a self contained answer from. Four things decide it, in order: whether the crawler is allowed in, whether the fact is in the HTML the crawler receives, whether the answer sits directly under the question, and whether the page is specific enough to be worth quoting at all.

Every crawler name in this article is published by its operator in that operator own crawler documentation, and every check described can be run against your own site today without a tool. Written 23 September 2026. Crawler names change; confirm against the operator.

HUREAL / Material studies

The distinction

Being cited is notthe same as ranking.

When an assistant answers a question, it does one of two things. It answers from what it absorbed during training, in which case no page is involved and no citation appears. Or it retrieves: it searches, fetches a small number of pages, and writes an answer grounded in what it found, naming the sources.

The second case is the one worth designing for, and its selection criteria are not the same as a search ranking. A search engine orders every page that is about a subject. A retrieval system picks a handful of passages that answer a specific question and that can be quoted without surrounding context. Authority still matters, and it matters less than it does in a ranking, which is why a small site can be cited on a narrow question it answers precisely while never appearing on the broad term above it.

That asymmetry is the whole opportunity, and it has a corollary worth saying out loud: none of this is under anybody's control. These systems change, vary between runs of the same prompt, and answer differently to two people on the same day. What follows influences inputs and measures outputs. It does not promise a citation, and nothing should.

First, and it dominates

Let the right crawlersthrough the door.

Each operator runs more than one crawler, and they do different jobs. The most consequential distinction on this page is between the crawlers that feed search and answers, and the crawlers that gather training data. Blocking the second group costs nothing in visibility. Blocking the first group removes the site from the answer entirely, and every other technique below becomes irrelevant.

The crawlers that decide whether a page can be cited, as published by each operator
Crawler What it does What blocking it costs
Search and answersAllow these
Googlebot Builds the Google index, which is also the source behind AI Overviews. Everything. Search, the overview, and a large share of what other systems retrieve.
Bingbot Builds the Bing index, which several assistants retrieve from. Bing, and the assistants that sit on top of it.
OAI-SearchBot Surfaces pages in search results inside ChatGPT. Appearing as a source in ChatGPT.
ChatGPT-User Fetches a page because a person in a conversation asked for it. The live fetch. A person asking about your company gets nothing back.
Claude-SearchBot and Claude-User Search indexing, and a fetch initiated by a person in a conversation. Appearing as a source in Claude.
PerplexityBot and Perplexity-User Indexing for results, and a fetch initiated by a person. Appearing as a source in Perplexity.
Applebot Feeds Siri, Spotlight and Apple search. The whole Apple surface, which is easy to forget and is not small.
Training and groundingA policy choice, not a visibility one
GPTBot Gathers data for model training. Nothing in citation terms.
ClaudeBot Gathers data for model training. Nothing in citation terms.
Google-Extended A control over whether content is used to ground and improve Google's generative products. It does not affect Google Search indexing. Reach inside Google's assistant products, without affecting search.
Applebot-Extended The equivalent control on Apple's side. The same trade, on the Apple surface.

Scroll the table sideways to read it.

Two things that break this without anybody deciding to. The first is a hosting platform or a content plugin that ships a block all AI setting, which blocks the search crawlers alongside the training ones and is sometimes on by default. The second is a firewall or bot management product sitting in front of the site: it can refuse these crawlers regardless of what the robots file says, and the robots file will keep looking correct while it does. Check the edge, not only the file.

It is also worth remembering what the robots file is. The Robots Exclusion Protocol, standardised as RFC 9309, is a request that well behaved crawlers honour voluntarily. It is not access control. Anything that genuinely must not be public needs authentication, and a disallow line is the wrong tool for it.

Second

A fact that arrivesafter the page does not exist.

A retrieval crawler fetches a URL and reads what comes back. Most of them do not execute JavaScript, and those that do, do it later, less often, and less reliably. A fact that only appears after the page has hydrated, or after a tab is clicked, or after a scroll, is not in the document the crawler received, which means it is not in the corpus the answer is drawn from.

  • The test, which takes ten seconds

    Fetch the page with a plain HTTP request and search the response for the sentence you want quoted. If it is not in that response, it is not there as far as retrieval is concerned, no matter what the browser shows.

    curl -s https://example.ca/page/ | grep -i "the sentence you want quoted"

  • The places facts hide

    Content inside a tab panel that is only populated when the tab is selected. Answers inside an accordion that is not in the markup until it is opened. Specifications loaded from a separate request after the page renders. Prices fetched client side. Anything inside a third party widget.

    All four are invisible to a fetch, and all four look fine in a browser.

  • The special case that catches everyone

    An accordion whose content is in the HTML and hidden with CSS is fine, because the text is in the response. An accordion whose content is inserted by script when it is opened is not. They look identical to a visitor and they are opposite to a crawler.

    This is why frequently asked questions are best left open at rest.

Third

Write so a passagesurvives being cut out.

Retrieval does not work on pages, it works on passages. A document is split into chunks, chunks are matched against the question, and the best few are handed to the model. So the unit that has to make sense is not the article. It is the piece of the article that arrives on its own, with nothing above it and nothing below it.

That produces one structural rule and three formatting consequences. The rule: put the question in a heading, and put a complete answer of roughly forty to sixty words directly underneath it, before any elaboration. A chunk that contains a question and its answer is quotable. A chunk from the middle of an argument that circles a subject for three paragraphs before reaching a point is readable and useless.

  1. Tables survive chunking, prose about tables does not

    A table carries its own structure: the column headings travel with the values. Three paragraphs describing the same comparison lose the correspondence the moment they are split, which is why a comparison written as prose is rarely quoted and the same comparison as a table often is.

  2. Give every heading a stable id

    An identifier that does not change lets an engine link the exact passage it quoted rather than the page it came from. That turns a page citation into a section citation, which is both more useful to the person reading the answer and more likely to be clicked.

  3. Say the thing before the context

    The most common structural failure in business writing is arriving at the point in the last paragraph. That order is fine in a memo, where the reader reaches the end. It is fatal here, because the chunk that gets retrieved is the one that matches the question, and the one that matches the question is the one that states the answer.

Structured data helps at the margin and is not a substitute. Marking up a question and answer helps a machine see which text answers which question, and it must describe what is visibly on the page. Markup that says something the page does not say is the fastest way to lose the feature it was added to win, and it is the kind of thing that gets a site distrusted rather than merely ignored.

Fourth, and it is the hard one

Be worth quoting,or be redundant.

A model composing an answer already knows the general shape of most subjects. What it does not have is the specific: the number, the date, the named statute, the named crawler, the measured result, the thing that is true in Canada and not elsewhere. A page that says what a competent generalist would have said contributes nothing to the synthesis, and it is not cited because there was nothing in it to cite.

This is why the most common content strategy fails at this surface. Producing large volumes of broadly correct material about a subject is precisely the thing a model can already do, and doing it at scale produces a site that is redundant with the answer rather than a source for it.

The test worth applying to every page before it is published

Does this page contain at least one thing that only somebody who has actually done the work would know? A section number. A failure mode with a name. A threshold. A date something changed. A result of a measurement, with the method attached.

If the answer is no, the page is not weak, it is unnecessary, and publishing it makes the site slightly harder to distinguish from every other site on the subject. That is a cost rather than a neutral outcome.

The corollary is uncomfortable for anybody planning a publishing schedule: the achievable cadence is set by how fast the specifics can be produced, not by how fast the words can be. Evidence is the bottleneck, and a schedule that outruns it produces exactly the material that does not get cited.

The other half

Being quotedis not being recommended.

There are two different outcomes here and they have different causes. A page can be cited as the source of a fact without the company behind it being understood as a company at all. Being recommended requires the second thing: a system has to have a resolved idea of what this organisation is, where it operates, and what it does.

That resolution is built out of consistency, and it is the least interesting work in this entire subject. One canonical description of the organisation, with a stable identifier, referenced from every page. Links out only to profiles that actually exist. The same legal name, the same address, the same phone number on the website, in the structured data, and on every directory and profile anywhere. Named people, with what they are knowledgeable about, pointing at profiles that are real.

Every one of those is boring, none of them is clever, and collectively they are the difference between a system stating what a company does and a system hedging around it. The most common cause of the hedge is not absence of information. It is two versions of the information, and a system with no way to tell which one is current.

Measuring it

There is no reportfor this. Run a panel.

No free, reliable interface exists that answers whether an assistant named a given company this month. The options are a paid monitoring vendor or a dated prompt panel run by hand, and the panel is worth doing regardless because it produces something a vendor report does not: the actual answers, with dates, that can be read.

  1. Write prompts in the buyer's language, not the vendor's

    Not the name of a service category. The question a person would actually type, phrased as a decision: who should rebuild this kind of company's website in this province, which suppliers do this particular thing in Canada, what does this cost here.

  2. Run each prompt three times, signed out, with the location set

    These systems vary between runs of an identical prompt. A single run is an anecdote. Running three and recording only what appeared consistently is the difference between a measurement and a screenshot.

  3. Record the date and the exact prompt, every time

    Without both, the result cannot be compared to the next one, and a series of uncomparable results is a habit rather than a measurement.

  4. Record who was named instead

    This is the finding that actually changes what gets published. A prompt that names three other suppliers tells you what the system currently believes about the category, and usually why: one of them has a page that answers the question directly and the others do not.

  5. Report what was consistent, not the best run

    The temptation in month one is to keep the run where the answer was flattering. A panel that does that is a marketing asset rather than an instrument, and it will produce a decision nobody should act on.

Alongside the panel, the ordinary instruments still apply. Server logs show which of the crawlers in the table above have actually visited, how often, and what response they received, and that is a measurement rather than an inference. If a search crawler has never appeared in twelve weeks of logs, no amount of work on answer shape is the problem to solve first.

Questions

Questions peopleactually ask.

  • Is this different from search engine optimization?

    It overlaps substantially and it is not identical. Crawler access, server rendered content, clean structure and internal linking serve both. What is additional is answer shape, which is writing so a passage can be lifted intact, and specificity, which is giving a synthesiser something it could not have produced on its own. A page can be cited without ranking, and can rank without being cited.

  • Does blocking the training crawlers cost us anything?

    Not in citation terms, on the current published behaviour of the major operators, because the crawlers used for training are separate from the ones used for search and for user initiated fetches. Blocking a training crawler is a policy choice with no visibility cost. Blocking a search or fetch crawler removes you from the answer, and the two are easy to confuse because they belong to the same company.

  • Should we publish an llms.txt file?

    It costs almost nothing, so yes, with the caveat stated plainly rather than buried. No answer engine has confirmed it as a retrieval or ranking input. It is a reasonable convention that may become useful and it should never be sold as a mechanism, to a client or internally. Anything presented as an llms.txt strategy is presenting a text file as a plan.

  • How long does it take to be cited?

    It can be faster than ranking, because retrieval favours clarity and a precise answer to a narrow question is not gated on link equity in the same way a competitive term is. It is also unreliable and varies between runs of the same prompt, which is why the measurement below runs each prompt several times and records only what was consistent.

  • Can we pay to be included in an answer?

    There is no mechanism to pay for an organic citation in any of the major assistants. Advertising is appearing in some of these surfaces and where it does it is labelled and sold separately. Anybody promising placement in an AI answer is offering something that does not exist to sell.

Read next

Three that followfrom this one.

The service behind this article

Every check hereis in the audit.

The checks in this article are the machine readable half of Search and AI Visibility, and they are run against a named URL rather than described. What comes back is what each crawler can reach, what is in the server response, where the answer shape holds and where it does not.

A performance audit reports which crawlers can actually reach your site, and what they receive when they do. The robots file, the response headers, what is in the HTML before any script runs, the answer shape per page, and what the structured data claims against what the page says. It is a document about your company and it is yours either way.

Request a performance audit

More on this subject in the Search and AI Visibility index, and everything else at Insights.

A material study photographed in bright daylight. A tall fan of clear turquoise glass fins rises on the right of the frame with a polished pearl silver ribbon curving through it, standing in a shallow film of still water. The left of the frame is empty pale mint.

Software you own from day one.

HUREAL / Material studies