Being Freed
Being Freed
Being Freed

Your website’s crawl budget

A website crawl budget is the resources search engines dedicate to exploring and indexing the site. You can tell them what to map first.
Your Website's Crawl Budget -- a graphic of a sitemap with various items greyed out, not connected to the chart.

What is a crawl budget?

For Googlebot, crawl budget is the amount of time and energy it’s willing to spend cataloging your site.  When you optimize your site for crawl budget, you’re telling Google and other search engines which of your pages are most important.  This is important because here are millions of sites out there, and search engines can only spend so much time on each one.  It’s like with your household budget: when it’s gone it’s gone.  So, it’s best to choose where you want to spend it.  That’s where crawl budget optimization comes in.

What does optimizing a crawl budget do?

Optimizing your website for SEO crawl budget does two things.  Firstly, it helps the environment by cutting down on the amount of energy it takes to crawl your site.  Secondly, tells search engines which pages to look at and which to ignore, so the engine knows which pages are most important to you.  Win win!

There are plenty of page types that have no SEO value such as login pages, media pages, and thank you pages.  You wouldn’t want Google to crawl those pages and not your blog posts, so why not tell Google to skip them?

How do I choose what to exclude?

WordPress is a powerful tool, with options for many website scenarios.  Of course, not all websites need all the tools.  In fact, most websites don’t use most of WordPress’s tools, so you can hide the code you’re not using and search engines won’t spend their time on it.  Shortlinks, various types of feeds, and media pages are a few things you can (normally) safely disable.  Of course, if you’re a visual artist and your images are the most important part of your site, you definitely want those to be catalogued.  Every site is different, and knowing what your site needs is the key to crawl budget optimization.

How to optimize your crawl budget

Yoast SEO has powerful tools to help you optimize your crawl budget, and this is what I use.  With Yoast, you can remove feeds, exclude metadata and coding that WordPress includes but your site does not use. 

To select various feeds, code, and other sundries, start at the WordPress dashboard.  Select Yoast SEO >> Settings >> Advanced >> Crawl optimization.  Educate yourself about what your site needs.

A screenshot of the Yoast SEO Advanced > Crawl Optimization Panel. Arrows point to the menu items, the toggle buttons, and the Save Changes button.
In the Yoast SEO Dashboard go to Advanced >> Crawl optimization. Once you've made your choices make sure to save the changes.

To optimize an entire page or post, start in the page or post’s Yoast SEO panel, scroll down to the Advanced tab.  From there, you can tell engines not to show it as a search result.

A screenshot of Yoast SEO's Advanced tab
In the page's Yoast SEO panel, scroll to the Advanced tab and in the dropdown menu for "Allow search engines to show this content in search results?", select no.

Why it’s worth optimizing for crawl budget

The bigger your site is, the more you need to pay attention to crawl budget.  Maybe you have posts (like I do), but you only have one author and type of post, and you take no comments (like I do).  It would be good to turn off post authors feeds, post type feeds, and comment feeds.

Even if you think your site isn’t big enough for this to matter, it just may be.  And besides, good practices are always worth following.

Vocabulary

Crawl budget – the amount of time/resources search engines put into crawling your website

Crawlability – the ease with which search engines can scan your page and understand its purpose.  Learn more here.

Data center – a building where servers are stored which, in turn, store data

SEO value – how important a thing is to search engine results

Shortlinks – a condensed version of a URL; a shorthand of sorts

Like my posts? Toss me a tip!

digital nomad hiker drysuit diver queer woman sober ukulele player scorpio Tough Mudder goofball

Freed scuba diving with her ukulele

All content © by Freed

Want to know more about what I do?  When I have a new post?  Where I’ve been lately?  (I do get around.)  Sign up for my mailing list below.  I promise to not fill your inbox with junk.