glossary · crawl budget
Crawl budget: the ceiling most sites never reach
Every site has a limit on how much of it gets fetched, and most sites never touch it. What sets the number, how to read it in Search Console, and the cases where it really is the thing holding you back.
Crawl budget is the number of URLs a search engine will fetch from a site over a period, set by how much crawling the server can absorb and how much the engine wants the site's pages.
reviewed 2026-09-02 · by the IT Master editorial team · how we check facts
What crawl budget is in practice
Crawl budget is not a setting, and there is no dial for it. It is the result of two judgements a search engine makes about your site.
- Crawl capacity. How fast it can fetch without hurting your server. Quick, stable responses let the rate rise; slow pages, timeouts and 5xx errors pull it down within days.
- Crawl demand. How much it wants your pages at all. Popularity, how often they change, and its own read of how much of the site is worth keeping.
Multiply the two and you get the fetches a day your site actually receives. You can see the number: the Crawl stats report in Search Console shows requests per day, average response time, what was fetched and by which crawler.
What spends the budget is rarely the pages you care about. Filter combinations, session parameters, sort orders, endless pagination, redirect chains, soft 404s and near-duplicate variants of one product can absorb most of the fetches on a large site. The pages that earn money get visited least, because they sit behind thousands of URLs nobody asked for.
Crawl capacity is what a server can absorb; crawl demand is how much a search engine wants the pages it finds. Crawl budget is the two of them multiplied, not a setting anyone can raise directly.
Why it matters in 2026
For most sites it does not, and that is the honest starting point. Google's own documentation says a site of a few thousand URLs is usually crawled efficiently without the owner managing anything. If you have 400 pages and they are indexed, crawling is not your problem.
It starts to bite in three situations, and all three got more common.
- URL explosion. A catalogue that generates combinations — filters, sort orders, variants, bundles — can address tens of thousands of URLs while the shop sells a few thousand products. That is index bloat, and the fetches it absorbs come out of the same pool as your product pages.
- Slow pages. A category page that renders cold in several seconds costs a crawler far more per fetch than a cached one, so it comes back less often.
- Publishing at volume. A new article needs discovery before it needs ranking, and discovery is a crawl. Programmatic SEO sets fail here first.
There is a second class of fetcher now. Assistant crawlers pull pages for training and, in some products, at the moment a question is asked. They add load without adding rankings, which is a capacity question your server feels even when Google is comfortable.
How IT Master fits into this
Narrowly, and it is worth being plain about it. Crawl budget is decided by your site's architecture — facets, parameters, response times, robots rules — and IT Master publishes into your CMS rather than owning any of that.
What the engine does control, it keeps tidy. Each published article arrives with JSON-LD schema and internal links into the articles already on the site, so a crawler reaches it through the existing structure instead of finding an orphan. IndexNow pings search engines on publish, so a new URL is announced rather than left to be rediscovered. The nightly loop refreshes decaying articles in place, updating the page that already ranks instead of adding a second one beside it. And only drafts that clear the validation gauntlet publish at all, so the URL count grows with pages that were checked.
The limit worth stating: if Google is already fetching your site and choosing not to index what it finds, publishing more articles will not change that. That is a selection problem. Run the site check first, and see how it works for the rest of the pipeline.
Refreshing an article in place, at the URL it already has, adds nothing for a crawler to fetch; publishing a second version at a new URL spends crawl budget and splits the signals between two pages.
Common misunderstandings
"Crawl budget is why my page is not indexed." Usually not. "Crawled, currently not indexed" means the page was fetched and passed over — a selection decision about quality or duplication, not a shortage of fetches. "Discovered, currently not indexed" is the one that points at crawling.
"Blocking URLs in robots.txt frees the budget." It does stop the fetches, which is the point, but a blocked URL can still be indexed from links alone, and the crawler can no longer see a noindex tag on it. Use noindex for pages that must leave the index; use robots.txt for URL patterns that should never be fetched.
"A bigger sitemap gets more crawling." Sitemaps help discovery. They are not an entitlement, and a file padded with every URL the site can generate teaches the engine to trust it less.
"Crawl-delay will pace Googlebot." Googlebot ignores the crawl-delay directive. Fix response times instead.
"More publishing pulls more crawl." Crawl demand follows what the engine values, not what you upload.
questions people ask
Does crawl budget matter for a small site?
Rarely. A site of a few thousand URLs that responds quickly is generally crawled as much as it needs, and Google's documentation has said as much for years. If pages are missing from the index on a small site, the cause is almost always discovery or selection: nothing links to the page, or the engine fetched it and decided against keeping it. Both are fixed by internal links and by the page being worth keeping, not by chasing crawl rate. Check the Crawl stats report before assuming a budget problem.
How do I check my crawl budget?
Open the Crawl stats report in Search Console, under Settings. It gives total requests per day over 90 days, average response time, and a breakdown by response code, file type, purpose and crawler. Two patterns are worth acting on. Requests flat while average response time climbs means capacity is throttling you, and the fix is server-side. Requests concentrated on parameter URLs, redirects or 404s means demand is being spent on pages you do not want indexed, and the fix is architectural.
Do AI crawlers use up crawl budget?
Not Google's budget, which is separate, but they use your server's capacity, and that is what sets the crawl rate Googlebot settles on. Assistant crawlers can be heavy on large catalogues, and some fetch pages live when a user asks a question rather than only for training. Watch them in your access logs by user agent rather than guessing. Blocking them saves load but also removes you from the answers they build, so decide that deliberately per crawler rather than with one blanket rule.
What actually improves crawl budget?
Two levers, in this order. Make responses fast and stable, because capacity is measured, not negotiated: cut cold render times, remove redirect chains, and stop returning 200 for pages that are really missing. Then remove URLs that should not exist — parameter combinations, sort orders, duplicate variants, expired listings — so demand lands on pages worth having. Adding sitemap entries, pinging harder or raising publishing volume does not create budget. Fewer, faster, non-duplicate URLs does.
related
See what search engines and AI assistants find on your site
Free, no account. Type your address and we show you what is missing and what we would write first.