Glossary · SEO & AI searchTOFU
Crawl budget
The short answer
Crawl budget is the number of URLs a search engine will crawl on your site within a given period, shaped by how fast your server responds and how much the search engine wants to crawl your content.
For most small sites it's not a concern. For large sites — eCommerce stores with filters, marketplaces, classifieds, publishers — it becomes critical: if crawlers spend their time on thousands of low-value URLs, important pages get crawled less often.
What wastes crawl budget
- Unlimited filter and sort parameter combinations
- Infinite calendar or pagination loops
- Duplicate URLs without canonical control
- Redirect chains
- Slow server responses
- Large numbers of thin or soft-404 pages
How to improve it
- Control faceted navigation
- Consolidate duplicates
- Fix redirect chains
- Improve server speed
- Keep XML sitemaps clean and current
- Use log file analysis to see what bots actually crawl
Signs of crawl budget problems
- Important new pages take a long time to be indexed
- Search Console shows many "Discovered – currently not indexed" pages
- Crawlers spend time on parameter URLs, filters or old pages
- Server errors or slow responses during crawl spikes
See technical SEO.
Crawl budget in practice
A marketplace with millions of filter URLs finds that important new products take weeks to be indexed. Blocking low-value parameter URLs and improving internal links helps search engines focus on important pages.
How to manage crawl budget
- Reduce duplicate and low-value URLs
- Keep sitemaps clean
- Fix server errors and slow responses
- Link to important pages
Common crawl-budget mistakes
- Letting filter and tracking URLs multiply
- Slow server responses
- Sitemaps full of redirected or noindexed URLs
Frequently asked questions
Does my small site need to worry about crawl budget?
Usually not. Focus on content quality and basic technical health.
How do I see crawl activity?
Search Console's crawl stats report and server log analysis.
Does an XML sitemap help crawl budget?
It helps search engines discover important URLs and see updates; it doesn't guarantee crawling.
Do redirects waste crawl budget?
Long redirect chains do. Point links directly to final URLs.
Do small sites need to worry about crawl budget?
Rarely — it mainly matters for very large sites.
How do we see how Google crawls our site?
Search Console's crawl stats report shows crawl activity and response codes.
Related terms
Full glossaryFree growth audit
Find out where your budget is leaking.
Send us your ad account, analytics or site. Within 72 hours you get a written audit: tracking gaps, wasted spend, and the three moves we'd make first. No pitch deck.