Skip to content
BOOSTD
Free SEO audit

AI search

How do AI assistants choose sources, and what can you influence?

Published by Zubair Afzal, responsible editor

First published Updated 8 min readWe check this page every three months. Next check: .

Start here

Three kinds of claim about how AI assistants choose sources

Work out which group a claim belongs to before you act on it. Most disagreements go away once you do.

  • Documented

    Published by the company that runs the system: which crawlers fetch pages and how to allow or block each one, which robots and snippet rules are respected, and Google’s statement that its AI features need no special markup. The only group you can check against a primary source.

  • Observable

    Things anyone can see from outside without knowing why: which sites appear in a list of citations, whether a changed page starts being used, whether assistant visits appear in your analytics. Evidence of a pattern, not a cause, from small samples that are easy to overread.

  • Unknown

    How possible sources are scored, what makes one of three near-identical pages the one that gets cited, and whether any of this stays the same between model versions. Nobody outside these companies knows; the confident articles are guessing.

Documented

What AI companies have documented about crawling and citations

AI companies have documented how their crawlers reach your site, and almost nothing about how they pick a source. They publish the names of the crawlers that fetch pages for them and how to allow or block each one in robots.txt.

Each crawler is a separate decision. One collects training data, another builds a search index, and a third fetches a page live because a user just asked a question. A site can easily allow one and block another without meaning to.

Google documents that its AI features use the same index as ordinary search. No special markup is needed, and the normal snippet controls apply: nosnippet, a max-snippet limit, and data-nosnippet on a single element. They are the one documented way to reduce what gets shown, and a site that set them years ago is applying them to AI features too.

Microsoft is the most open case, because Copilot answers from Bing. Bing Webmaster Tools shows what was crawled, indexed and rejected, and IndexNow lets you tell Bing about a change as soon as you publish it. That makes Copilot the best place to find access problems that probably affect the other assistants too.

No AI company publishes what makes a source more likely to be chosen. The documented controls all limit what gets shown; everything about getting picked is guesswork.

The usual first finding

Who decides whether AI crawlers can read your site

Several people decide whether AI crawlers can read your site, and most of them are outside marketing. AI companies publish each crawler and how to allow or block it. Each is a separate setting, so a site can allow one and block another by accident.

Who controls whether AI crawlers can reach a site.

Decided by marketing

  • The lines your team wrote in robots.txt
  • A chosen setting for each crawler, where one was made

Decided elsewhere, by another team

  • A bot rule in the CDN, set for security reasons
  • A snippet limit set years ago to protect content
  • A robots file inherited from a website template
  • A default setting nobody in marketing has seen

What to do

Five steps to become a source AI assistants can cite

Each step is worth doing whatever the AI tools do next. If a change only makes sense as a way to sway a model, it isn’t here.

  1. Make sure your pages can be fetched

    Check robots.txt, then the bot rules in your CDN or firewall, and your server logs for the published crawler names. Then check that each page includes its content in the HTML, not only built in the browser afterwards. A page that can’t be fetched is ruled out before anything else is considered.

    You get: A written record of which crawlers are allowed, with a reason for each

  2. Give each fact one page

    Pick one page for every question you want to be the source on. State the full answer in the opening paragraph, and make the heading say the fact clearly. If the answer is split across four pages, bring it together on one. It helps human readers whatever any machine does.

    You get: One main page per fact, with the answer at the top

  3. Fix the conflicting descriptions you control

    Your website, directory listings, social profiles, Companies House or registry entry, Google Business Profile and press release boilerplate should all describe what you do, who for and where in the same way. A system that can’t match you to one consistent business tends to describe a competitor.

    You get: One agreed description, used everywhere you control

  4. Publish something only you have

    A survey of your customers, data from your own operations, a documented method or a price index for your category. A page that repeats what four other pages say can be swapped for any of them; a page that holds the only version of a fact can’t.

    You get: At least one publishable fact that only you can offer

  5. Keep a dated log of prompts and answers

    Run a fixed set of buyer prompts monthly in the assistants that matter to you. Record the date, the exact wording, the assistant and the full response, with no score. It is noisy sampling, but the only fair way to see whether the way AI describes your business is changing.

    You get: A running log of full responses with dates and wording

Unknown

AI citation tactics with no documented evidence

No AI company documents any of these, and nobody has isolated a measurable effect from them. Some are harmless; some aren’t.

  • Publishing a file that tells assistants how to read your site. No major AI company says it reads one, so it is a theory, not a deliverable.
  • Writing instructions to a model inside your page, hidden or not. AI search doesn’t work that way. Where it is spotted, it is treated as manipulation.
  • Markup sold as AI-specific or "answer engine" schema. No such vocabulary exists. Structured data is worth using for reasons that came long before AI search.
  • Adding a question-and-answer block to every page, whether or not anyone asks those questions. Pages get longer and harder to use, and the facts don’t get easier to quote.
  • Buying mentions on sites that exist only to be scraped. These mentions are easy to spot as coordinated. The risk sits with you, and none of it with the AI company.
  • Any product that gives a share-of-voice figure for an assistant without saying how many prompt runs it rests on. That number is sampling noise with a decimal point.

What lasts is being reachable, correct and consistent. Tactics that only make sense as a way to manipulate a model are undocumented, can’t be measured and don’t last.

The steps that survive a model update are the ones that would be worth doing if AI assistants didn’t exist. To apply this to your own site, find out what AI assistants say about you now, then trace where those descriptions came from.

Questions

Common questions about how AI assistants cite sources

Does adding an llms.txt file to my site do anything?

No major AI company has documented support for llms.txt, so anyone who says it works is reporting a belief. It costs almost nothing to publish, but it shouldn’t appear on an invoice as a deliverable.

The general test: a tactic with no documentation from the company it is meant to influence, and no effect you could measure, is a guess. Guessing is fine. Charging for guesses isn’t.

Does structured data make an assistant more likely to cite us?

No documented link exists between structured data and citations, and Google says its AI features need no special markup. Structured data does help a machine work out what a page is about and which business it belongs to, which is useful for a different reason.

Use it because it makes your pages clear and eligible for documented search features. Don’t use it because someone called it a citation lever.

How do I know whether AI crawlers can reach my site?

Check three places, in this order: your robots.txt file, the bot rules in your CDN or firewall, and your server logs for the crawler names the AI companies publish. The surprises are usually in the CDN or firewall, often set by an infrastructure team for security reasons.

Blocking AI crawlers is a fair choice for some publishers. The problem is finding out two years later that a default setting made that choice for you.

Why does a smaller competitor get cited when we don’t?

Often their answer is easier to find. They state the fact clearly on one page; you may have it split across a case study, a services page and a PDF. AI search tends to pick a clear, complete statement over a strong brand with a vague page.

Another reason is conflicting information. If your site, profiles, directory listings and old pages describe your business three different ways, a system has no stable version of you to use, and describing someone else is the safer choice.

Can we pay to appear in an assistant’s answer?

Not in the organic answer. Some AI products carry advertising, but those ads are a separate, labelled placement. Anyone offering to arrange a place in the answer itself is selling something that doesn’t exist.

That leaves one route in: being a source the assistant can use. It takes longer and is much harder to fake.

How would we ever know if any of this worked?

By sampling the answers regularly. Run a fixed set of buyer prompts on a schedule in the assistants that matter to you, and record the date, the exact wording and the full response. Over months, that shows a direction.

It doesn’t show a fixed position, because two runs of the same prompt can give different answers. A single score that doesn’t say how many samples it rests on is noise dressed up as a measurement.

References

Sources for how do AI assistants choose sources

The official documents and research this page is based on. Platform rules change, so check the source before you act on a detail.

  1. Google Search Central: AI features and your website
  2. Google Crawling Infrastructure: overview of Google crawlers and fetchers
  3. Google Crawling Infrastructure: Google common crawlers (including Google-Extended)
  4. Google Search Central: robots meta tag and inline directives
  5. OpenAI: overview of OpenAI crawlers
  6. Perplexity: Perplexity crawlers
  7. Bing Webmaster Tools: Bing Webmaster Guidelines
  8. IndexNow documentation

Related

Keep reading on how do AI assistants choose sources

Start here

Check which AI crawlers can reach your site

Our specialists will look at your robots file, your bot rules and how your pages render, and give you a list of the AI crawlers that can read your site today.