On this page8 sections
Why it matters for founders and small teams
For most small sites, crawl budget is a problem you don’t have — Google says sites whose new pages are crawled the day they’re published can skip the topic. It becomes yours when a CMS, faceted filters or tag archives quietly turn a few hundred real pages into thousands of URLs, or when AI crawlers add enough load to slow down the server every bot depends on.
How does crawl budget work?#
Crawl budget is set by two things: the crawl capacity limit, meaning how many connections a crawler can use without straining your server, and crawl demand, meaning how much it wants your URLs based on popularity, freshness and quality. Google defines a site’s budget as the set of URLs it can and wants to crawl.
| Part | What raises it | What lowers it |
|---|---|---|
| Crawl capacity limit (hostload) | Fast, stable responses and time to first byte | Slower responses, 5xx server errors, 429 rate-limiting |
| Crawl demand | Popular URLs, content that changes, site moves | Duplicate and unimportant URLs, stale pages nobody links to |
Three details from Google’s guide matter. Budget is set per hostname, so www.example.com and shop.example.com are budgeted separately. Every site starts at the same conservative capacity limit, which rises only if the site stays healthy under more crawling. And that limit “is shared across all crawlers,” so heavy demand from one Google crawler, such as Shopping or AdsBot, leaves less for Googlebot.
Crawled doesn’t mean indexed: after crawling, each page is still evaluated and consolidated before it earns a place — see indexing.
When does crawl budget matter?#
Crawl budget matters for sites with more than about a million unique pages that change weekly, sites with 10,000 or more pages that change daily, and sites where Search Console lists many URLs as “Discovered - currently not indexed”; Google says everyone else can simply keep their sitemap current.
| Site profile, per Google's rough guide | Crawl budget a concern? |
|---|---|
| 1 million+ unique pages, changing about weekly | Yes |
| 10,000+ unique pages, changing daily | Yes |
| A large share of URLs in “Discovered - currently not indexed” | Yes — Google wanted to crawl them but expected to overload the site |
| New pages crawled the day they’re published | No — keep the sitemap current and watch the Page indexing report |
Google calls these numbers rough estimates, “not exact thresholds.” The third row is the one small sites run into. Search Console defines “Discovered - currently not indexed” as a URL Google found but postponed because crawling it “was expected to overload the site” — on a small site, that often points to a slow server or a URL explosion rather than too little budget.
Worked example
The Crawl Waste Ratio
An estimate of how much crawler attention goes to URLs you never wanted crawled, from one month of server logs. The inputs are illustrative, for a fictional project-management app called Plannora — the same steps work on any access log.
- 1
Count the URLs you want crawled
Plannora’s sitemap: product pages, blog posts, docs and a public gallery of 300 project templates.
900 URLs
- 2
Count the URLs crawlers requested
One month of access logs, filtered to Googlebot, GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, deduplicated by URL.
7,400 URLs
- 3
Sort the extras
Gallery filter and sort combinations (
?category=,?sort=,?page=): 5,100. Old docs URLs returning 404 after a migration: 1,050. Tag archives andutmduplicates: 350.6,500 unwanted
- =
Crawl waste ratio
6,500 ÷ 7,400 — the share of distinct URLs crawled that Plannora never wanted crawled.
88%
The result: An 88% ratio on a 900-page site isn’t a budget crisis yet, but it’s a warning: one more filter or a second migration could make it one, and meanwhile every bot burns fetches on dead ends. The fixes are cheap — disallow the filter parameters, 301 the migrated docs to their new URLs or return 410 for the ones that are gone, and canonicalize the tag duplicates. Re-run the ratio a month later.
Free to use and adapt. If you cite it, link to rankbox.xyz/glossary/crawl-budget.
What wastes crawl budget?#
Crawl budget is wasted on URLs that shouldn’t be crawled at all — faceted filters, sort orders, session and tracking parameters, endless calendars, soft 404s, long redirect chains and duplicate versions of one page — because every fetch spent on them is one not spent on new or updated content.
- Consolidate duplicates with redirects and canonical tags, so crawlers spend time on unique content rather than unique URLs.
- Block true junk in robots.txt — endless filter combinations, internal search results. Google advises against
noindexfor this, because it still requests the page before dropping it. - Return 404 or 410 for pages that are gone. A 404 is a strong signal not to crawl again; a blocked URL stays in the queue much longer.
- Fix soft 404s, error pages that return 200 and keep getting crawled.
- Keep your XML sitemap current, with
lastmod, and avoid long redirect chains. - Make pages cheap to fetch: fast responses, plus
304 Not Modifiedfor unchanged pages, which lets Google reuse its cached copy.
How do AI crawlers affect crawl budget?#
AI crawlers don’t draw on Google’s crawl budget, but they share your server: in Vercel’s network data, OpenAI’s, Anthropic’s, Apple’s and Perplexity’s bots together made about 28% as many fetches as Googlebot, and when load slows responses or triggers errors, Google lowers its own crawl capacity for your site.
569M
GPTBot fetches in one month across Vercel's network, against 4.5 billion for Googlebot
370M
fetches by Anthropic's crawler over the same month
34.82%
of ChatGPT's crawler fetches hit 404 pages (Claude's: 34.16%)
| Crawler owner | Rate controls it honors |
|---|---|
Google — Googlebot | No Crawl-delay; slows down on its own when responses slow or return 5xx and 429 errors |
Anthropic — ClaudeBot, Claude-SearchBot, Claude-User | robots.txt, including the non-standard Crawl-delay |
Perplexity — PerplexityBot | robots.txt rate limits, and backs off when a site struggles |
OpenAI — GPTBot, OAI-SearchBot | robots.txt per bot; when both are allowed, one crawl may serve search and training |
Mind the trade-off. Blocking search bots such as OAI-SearchBot or PerplexityBot to save load also removes you from those engines’ answers, while training-only bots like GPTBot and ClaudeBot can be blocked without that cost. Before blocking anything, fix the 404s and parameter URLs the bots are wasting fetches on — that helps every crawler at once.
Common mistakes with crawl budget#
The most common crawl budget mistakes are worrying about it on a small site, using noindex or robots.txt as a budget-shifting trick, leaving parameter and tag URLs crawlable by the thousand, and blocking AI search crawlers to cut load when the real drain is 404s and duplicates.
Myth
Every site needs to optimize its crawl budget.
Reality
Google’s guide targets very large or fast-changing sites. If new pages are crawled the day they’re published, keep your sitemap current and move on.
Myth
noindex saves crawl budget.
Reality
Google still requests a noindex page before dropping it. Use robots.txt for URLs you never want crawled, and 404 or 410 for pages that are gone.
Myth
Submitting more URLs gets more of them crawled.
Reality
Sitemaps and IndexNow pings suggest; they don’t raise capacity. Every IndexNow submission counts toward your crawl quota, so ping only what changed.
Myth
More crawling can be requested or bought.
Reality
Google “doesn’t accept payment to crawl a site more frequently.” The two levers it names are more server capacity and better content — popularity, uniqueness and user value.
Related terms#
- Technical SEOIndexingThe step in which a search engine processes a crawled page and stores it in its searchable database; only indexed pages can rank, or be retrieved for AI answers built on that index, and being crawled does not guarantee being indexed.Read the entry
- Technical SEOXML sitemapA machine-readable file listing a site’s canonical URLs, optionally with last-modified dates, so search engines can discover pages and prioritize recrawling the ones that changed — a discovery aid, not a ranking factor or a guarantee of indexing.Read the entry
- Technical SEOCanonical tagAn HTML link element (rel="canonical") that tells search engines which URL is the preferred version of a page reachable at several addresses, consolidating its ranking signals onto that one URL — a strong hint, not a command.Read the entry
- AI crawlersrobots.txtA plain-text file at a site’s root that tells crawlers which paths they may fetch, rule by rule for each user agent — a voluntary standard that reputable search and AI bots follow, which controls crawling but not whether a URL gets indexed.Read the entry
- AI crawlersAI crawlersAutomated bots run by AI companies that fetch web pages for one of three jobs — training models, building an AI search index, or retrieving a page live for a user’s question — and because each job uses its own user agent, each can be allowed or blocked separately.Read the entry
- Technical SEOIndexNowAn open protocol that lets a website notify participating search engines, led by Microsoft Bing, the moment a URL is added, updated or deleted, instead of waiting to be recrawled — Google does not use it.Read the entry
Go deeper
Sources
- 1.Crawl budget managementGoogle Crawling Infrastructure · developers.google.com ↗
- 2.Page indexing reportSearch Console Help · support.google.com ↗
- 3.How Google interprets the robots.txt specificationGoogle Crawling Infrastructure · developers.google.com ↗
- 4.In-depth guide to how Google Search worksGoogle Search Central · developers.google.com ↗
- 5.The rise of the AI crawlerVercel · vercel.com ↗
- 6.Does Anthropic crawl data from the web, and how can site owners block the crawler?Anthropic Help Center · support.claude.com ↗
- 7.Perplexity crawlersPerplexity Docs · docs.perplexity.ai ↗
- 8.Overview of OpenAI crawlersOpenAI · developers.openai.com ↗
