Indexing

How does indexing work?#

Indexing works in stages after a URL is discovered: the engine crawls the page, processes it — rendering it, reading the text, title and links, and grouping duplicates to pick a canonical — and then decides whether to store it in its index, the database it searches when someone asks a question.

1. 1

   Discovery

   The engine learns the URL exists — from a link on a page it already knows, from a sitemap, or from a push such as IndexNow.

   Your lever: Link new pages from pages that already get crawled, and list them in your sitemap.

2. 2

   Crawling

   A bot fetches the page, if robots.txt allows it and the server answers. Googlebot reads only the first 2 MB of the HTML.

   Your lever: Keep the page fast, unblocked and light.

3. 3

   Processing

   Google renders JavaScript and analyzes the text, title and alt attributes; most AI crawlers skip rendering. Duplicates are clustered and one canonical is chosen.

   Your lever: Serve the main content in the HTML — see server-side rendering — and send one clear canonical tag.

4. 4

   The index decision

   The page is stored, or it isn’t. Google: “Indexing isn’t guaranteed; not every page that Google processes will be indexed.” Low quality, noindex rules and duplication are the usual reasons.

   Your lever: Give every page a job no other page on your site already does.

5. 5

   Serving

   Only indexed pages can rank, or be retrieved when an AI engine grounds an answer. For Google’s AI features the page must also be eligible to show a snippet.

None of it can be bought. Google says it “doesn’t accept payment to crawl a site more frequently, or rank it higher,” and “doesn’t guarantee that it will crawl, index, or serve your page” even when the page follows every guideline.

Why is my page not indexed?#

A page usually isn’t indexed for one of five reasons: crawlers can’t reach it, it carries a noindex, Google treats it as a duplicate of another URL, it adds too little unique value, or Google hasn’t got to it yet. Search Console’s Page indexing report says which one applies.

Google adds that you “should not expect all URLs on your site to be indexed, only the canonical pages” — so a report full of alternates, redirects and parameter URLs is normal. Focus on the pages you’d be upset to lose, and on patterns: a whole template stuck in one status is a site problem, not a page problem. If the pattern is URL bloat, see crawl budget.

How do you check if a page is indexed?#

Check indexing with each engine’s own tools: URL Inspection in Google Search Console shows whether a URL is on Google and which canonical Google chose, and Bing Webmaster Tools does the same job for Bing. A site: search is only a rough spot check, not a complete list.

1. Google: URL Inspection in Search Console shows the indexing status, the Google-selected canonical and the rendered HTML, and its live test shows what Googlebot gets right now. Request indexing there for your most important URLs; it’s rate-limited.
2. Bing: verify the site in Bing Webmaster Tools, inspect key URLs and submit your sitemap. Bing’s index feeds Copilot, and Microsoft is a named ChatGPT search provider.
3. Brave: there’s no console. Search your page title on search.brave.com, and after big changes use the re-fetch form at search.brave.com/submit-url, which asks Brave to re-crawl a page without guaranteeing it.
4. Perplexity: no console and no submission. Discovery runs through its own crawler and links from sites it already trusts.

# Status code and any X-Robots-Tag header
curl -sI https://www.example.com/blog/launch-checklist | grep -i -E "^HTTP|x-robots-tag"
 
# A stray robots meta tag in the HTML (a leftover staging noindex is common)
curl -s https://www.example.com/blog/launch-checklist | grep -i -o '<meta[^>]*robots[^>]*>'
 
# Google spot check, incomplete by design:
#   site:example.com/blog/launch-checklist

Does indexing matter for AI search?#

Indexing is the entry ticket to AI search: AI Overviews, AI Mode and Gemini ground answers in Google’s index, ChatGPT draws on Bing and OpenAI’s own index, Claude on Brave’s and Perplexity on its own — so a page missing from an engine’s index can’t be cited by that engine.

Being indexed isn’t the same as being cited. In Ahrefs’ March 2026 data only 37.9% of pages cited in AI Overviews ranked in the top 10 for the typed query, because the engine retrieved them for narrower fan-out sub-queries. But every one of those pages had to be in the index first: retrieval-augmented generation can only retrieve what’s stored.

Common mistakes with indexing#

The most common indexing mistakes are blocking a page in robots.txt to keep it out of the index, leaving a staging noindex in place after launch, resubmitting pages Google has already crawled and declined, and publishing near-identical pages that get folded into one.

Related terms#

• Technical SEOCrawl budgetThe number of URLs a search engine is willing and able to crawl on a site in a given period, set by how much load the server can take and how much the engine wants the content — a real constraint mainly for large or fast-changing sites.Read the entry
• Technical SEOXML sitemapA machine-readable file listing a site’s canonical URLs, optionally with last-modified dates, so search engines can discover pages and prioritize recrawling the ones that changed — a discovery aid, not a ranking factor or a guarantee of indexing.Read the entry
• Technical SEOCanonical tagAn HTML link element (rel="canonical") that tells search engines which URL is the preferred version of a page reachable at several addresses, consolidating its ranking signals onto that one URL — a strong hint, not a command.Read the entry
• GSCMeasurementGoogle Search ConsoleGoogle’s free tool for site owners that reports how a site performs in Google Search — queries, impressions, clicks, position and indexing — and since 2026 it also reports impressions in Google’s generative AI features and lets owners opt out of them.Read the entry
• RAGHow LLMs answerRetrieval-augmented generationA technique in which an AI system first retrieves relevant documents from an index and then gives them to a large language model to write its answer, so the response can cite current sources instead of relying only on what the model memorized in training.Read the entry
• Technical SEOIndexNowAn open protocol that lets a website notify participating search engines, led by Microsoft Bing, the moment a URL is added, updated or deleted, instead of waiting to be recrawled — Google does not use it.Read the entry