On this page8 sections
Why it matters for founders and small teams
For a founder without a content team, SEO is still the one acquisition channel that compounds: a page written this quarter can keep bringing in buyers for years without a media budget. It now pays twice, because ChatGPT, Gemini, Claude and Google’s AI Overviews all retrieve their sources from a search index, so the pages you get indexed and ranked are the pages AI can cite. Skip it and you’re missing from both places at once.
How does search engine optimization work?#
Search engine optimization works on the three stages every search engine runs — crawling, indexing and ranking — by making each page easy to fetch, easy to understand, and more useful and trusted than the alternatives for its query.
Google lays out the stages in its guide to how Search works — crawling, indexing and serving results ranked by “hundreds of factors” — and notes it “doesn’t accept payment to crawl a site more frequently, or rank it higher.” SEO is the work at each stage.
- 1
Crawling: the engine finds and fetches the page
Crawlers follow links and sitemaps to discover URLs, then download them — if robots.txt, the server and any firewall let them in.
Your lever: Link to every important page, keep an honest XML sitemap, and never block the search bot in robots.txt.
- 2
Indexing: the page is analyzed and stored
The engine renders the page, works out what it’s about and picks a canonical version. Crawled doesn’t mean indexed: thin or duplicate pages are often left out.
Your lever: Serve the main content in the HTML, consolidate duplicates with a canonical tag, and check indexing in Search Console.
- 3
Ranking: the best answers are ordered for each query
Relevance to the search intent, content quality, links and mentions from other sites, and page experience decide the order.
Your lever: Write the most useful page for one clear question, and earn backlinks from sites in your niche.
- 4
Retrieval for AI answers
AI answer engines search the same kind of index and cite the pages they retrieve, so indexed, well-ranked pages form the candidate pool for AI citations.
Is SEO still worth it with AI search?#
SEO is still worth it because nearly every AI answer engine retrieves its sources from a search index — Google’s, Bing’s, Brave’s or its own — so a page that can’t be crawled, indexed and ranked can’t be cited either.
| AI engine | Where it retrieves sources from |
|---|---|
| Google AI Overviews and AI Mode | Google’s Search index, through its core ranking systems |
| Gemini app | Google’s Search index (Grounding with Google Search) |
| ChatGPT search | Third-party providers (Microsoft named) plus OpenAI’s own index |
| Claude | Brave Search |
| Perplexity | Its own index of more than 200 billion URLs |
Google says its generative AI features “are rooted in our core Search ranking and quality systems,” so “the best practices for SEO continue to be relevant.” What changed is that ranking is necessary but no longer sufficient. Engines split each prompt into narrower sub-queries (query fan-out): Ahrefs found only 37.9% of pages cited in AI Overviews rank top 10 for the typed query, and about 8% of ChatGPT’s citations rank in Google’s or Bing’s top 10 for the original prompt.
Rankbox benchmark
The SEO Hard-Limits Sheet
The handful of SEO thresholds search engines actually publish — lines where crossing them means content is ignored or a page experience is rated poor. Every value comes from Google’s own documentation; popular rules like ideal word counts or keyword density have no published threshold at all.
- First 2 MB
Googlebot HTML fetch
Googlebot fetches the first 2 MB of a URL, headers included; anything past the cut-off is not fetched, rendered or indexed. Keep the title, canonical and main content early. Source: Google Search Central Blog, March 2026.
- ≤ 2.5 s
Largest Contentful Paint
Loading: the main content should render within 2.5 seconds, measured at the 75th percentile of real page loads on mobile and desktop. Source: web.dev.
- ≤ 200 ms
Interaction to Next Paint
Responsiveness: the page should react to taps, clicks and key presses within 200 milliseconds, at the same 75th percentile. See Core Web Vitals.
- ≤ 0.1
Cumulative Layout Shift
Visual stability: content shouldn’t jump around as the page loads. A CLS of 0.1 or less counts as good. Source: web.dev.
- 50,000 URLs / 50 MB
XML sitemap file size
Per sitemap file, uncompressed; larger sites split into several files under a sitemap index. Google uses
lastmodonly when it’s consistently accurate and ignorespriorityandchangefreq.- 500 KiB / 24 h
robots.txt size and cache
Google ignores anything past 500 KiB and generally caches the file for up to 24 hours, so a fix can take a day to apply. OpenAI and Perplexity quote about 24 hours as well.
How to read it: Treat these as floors, not goals. Passing them doesn’t earn rankings, but failing the fetch, sitemap or robots.txt limits means content simply isn’t read, and failing Core Web Vitals marks the page experience as poor. Check the mechanical limits once per template; watch Core Web Vitals monthly in Search Console.
Free to use and adapt. If you cite it, link to rankbox.xyz/glossary/search-engine-optimization.
SEO vs GEO vs AEO: what's the difference?#
SEO earns a ranked link on a results page, AEO earns the single extracted answer, and GEO earns a named citation inside an AI-written response — three goals built on the same crawlable, indexed, trustworthy pages.
| SEO | AEO | GEO | |
|---|---|---|---|
| Wins | A ranked position | The one answer shown or spoken | A citation or mention in a generated answer |
| Typical surface | Blue links on a SERP | Featured snippets, voice assistants, chatbots | ChatGPT, Perplexity, Claude, AI Overviews |
| Unit that competes | The page | The passage that answers | The passage, across many sub-queries |
| Main metric | Rankings and clicks | Snippet and answer ownership | Citation rate and AI share of voice |
In practice they’re layers, not rivals. GEO and AEO can’t work on a page search engines never indexed, and the habits they add — answering in the first sentence, covering follow-up questions, earning mentions off-site — are good practice for classic search too. Treat SEO as the foundation and the other two as how you write and distribute on top of it.
How do you measure SEO?#
Measure SEO in Google Search Console — impressions, clicks, average position and indexing status per page and query — then follow those visits into analytics to see which pages produce signups or pipeline, not just traffic.
- Indexing first: the page indexing report and URL Inspection in Google Search Console show whether key pages are in the index at all. Nothing else counts until they are.
- Visibility: impressions and average position by query show where you’re close to page one — usually the fastest wins.
- Clicks and click-through rate: a falling CTR at a steady position usually means an AI Overview or another feature is answering on the results page.
- Business outcome: organic sessions, signups and pipeline by landing page in your analytics.
- The AI layer: Search Console’s Generative AI report, rolled out worldwide by 31 August 2026, shows impressions in AI Overviews and AI Mode by page — but no clicks. Pair it with prompt tracking.
Common mistakes with SEO#
The most common SEO mistakes for small teams are chasing head terms they can’t win, publishing many thin pages instead of fewer useful ones, and hiding content behind JavaScript or the wrong robots.txt rule.
Myth
Go after the biggest keyword in the category.
Reality
A young site rarely wins high-difficulty head terms. Specific long-tail questions are where small teams rank first — and where AI fan-out searches too.
Myth
More pages and more keywords means more traffic.
Reality
Google treats pages made mainly to manipulate rankings as scaled content abuse, whether written by AI or by hand. A tight topic cluster of useful pages beats a pile of thin ones.
Myth
Keyword stuffing and a keywords meta tag still help.
Reality
Google “doesn’t use the keywords meta tag,” and keyword stuffing is against its spam policies. The original GEO research paper found stuffing did little for AI visibility either.
Myth
If it looks fine in my browser, crawlers see it too.
Reality
Google renders JavaScript, but OpenAI’s, Anthropic’s and Perplexity’s fetchers read raw HTML. Use server-side rendering for anything you want ranked and cited.
Myth
Google is the only index that matters.
Reality
ChatGPT draws on Bing and OpenAI’s own index, Claude on Brave, Perplexity on its own. Verify your site in Bing Webmaster Tools and check that non-Google crawlers can read your pages.
Related terms#
- GEOAI search & GEOGenerative engine optimizationThe practice of making content easy for AI answer engines such as ChatGPT, Perplexity and Google AI Overviews to retrieve, quote and cite — where SEO competes for a ranked link, GEO competes to be a named source inside the generated answer.Read the entry
- AEOAI search & GEOAnswer engine optimizationThe practice of structuring content as direct, extractable answers so that systems which return one answer — featured snippets, voice assistants and AI chatbots — use it as the answer itself rather than listing it as one link among ten.Read the entry
- Technical SEOIndexingThe step in which a search engine processes a crawled page and stores it in its searchable database; only indexed pages can rank, or be retrieved for AI answers built on that index, and being crawled does not guarantee being indexed.Read the entry
- Content & relevanceSearch intentThe goal behind a query — to learn something, reach a specific site, compare options or buy — and matching it decides which kind of page, from a guide to a product page to a comparison, search engines and AI answers treat as relevant.Read the entry
- Authority & off-siteBacklinksA link from one website to another, which search engines treat as a vote of confidence that passes authority to the page it points to — and the relevance and quality of the linking site matter far more than the number of links.Read the entry
- AI search & GEOAI OverviewsAI-generated summaries shown at the top of some Google search results, written by a Gemini model from pages in Google’s search index and linked to their sources — so they are won through ordinary Google eligibility, not a separate submission or special markup.Read the entry
Go deeper
Sources
- 1.In-depth guide to how Google Search worksGoogle Search Central · developers.google.com ↗
- 2.Search Engine Optimization (SEO) Starter GuideGoogle Search Central · developers.google.com ↗
- 3.Optimizing your website for generative AI featuresGoogle Search Central · developers.google.com ↗
- 4.Inside Googlebot: demystifying crawling, fetching, and the bytes we processGoogle Search Central Blog · developers.google.com ↗
- 5.Web Vitalsweb.dev · web.dev ↗
- 6.Build and submit a sitemapGoogle Search Central · developers.google.com ↗
- 7.How Google interprets the robots.txt specificationGoogle Search Central · developers.google.com ↗
- 8.How many AI Overview citations rank in the top 10?Ahrefs · ahrefs.com ↗
- 9.AI search overlap with Google and BingAhrefs · ahrefs.com ↗
- 10.Google users are less likely to click on links when an AI summary appearsPew Research Center · pewresearch.org ↗
