Googlebot Web Crawler Update: What It Means for Your B2B SEO

When Google recently updated a help document, they accidentally revealed information that could change how B2B marketers approach SEO. It has to do with Googlebot, the generic name for Google’s web crawler. Googlebot is responsible for simulating both desktop user experience and mobile user experience and plays a crucial role in where your website ranks for a search engine result page (SERP). After gathering information to make an SEO assessment, Googlebot then provides this information to Google in order to decide whether or not to index a page.

So, what’s the secret? It has been reported that Google’s web crawler only fetches the first 15 MB of a web page, and the rest of the webpage after that cutoff point is not used to calculate search engine rankings. 

Specifically, Google revealed this:

“Googlebot can crawl the first 15MB of an HTML file or supported text-based file. Any resources referenced in the HTML such as images, videos, CSS, and JavaScript are fetched separately. After the first 15MB of the file, Googlebot stops crawling and only considers the first 15MB of the file for indexing. The file size limit is applied on the uncompressed data. Other crawlers may have different limits.”

What About Your B2B SEO Marketing?

What does this mean for you? There has been a lot of speculation and also some misinformation on this development. Apparently, the 15MB cutoff has been around for many years, it has just never been publicly disclosed. 

For starters, it’s important to understand that most web pages don’t even come close to being 15 MB in size. Most webpages exist as HTML files, where the median file size currently sits at 30 kilobytes (KB), or 500 times smaller than 15 MB. Unless you’re publishing an entire encyclopedia of text on a given webpage, you probably don’t have a page that even comes close to being 15MB in size. This means that you still need to optimize for SEO on an entire webpage, and none of your hard work optimizing for SEO has gone in vain. In fact, if you do have a web page that’s 15MB or greater, you likely have a website that’s hardly usable and offers a very poor user experience. 

But what if your webpage has lots of images? Don’t worry, images are fetched separately. Whenever an image is defined through URL, it does not account for the 15MB crawl limit. This is because the image is not encoded within the HTML file itself, rather, it is referenced from a different URL. Whenever possible, images and videos should be compressed and not encoded directly into a webpage. 

Regarding SEO best practices, it is recommended to keep web pages below 100KB, and HTML for eCommerce sites typically want to stay below 200KB. But how can you check your site’s HTML? It’s possible to right-click on a page and use the ‘inspect’ feature, but you can also simply plug in your URL at DebugBear’s HTML size analyzer

More than anything, you shouldn’t be intimidated by Google’s new announcement. Very few websites will be affected in any way, but it’s a great reminder to stay on top of core SEO best practices.

As a quick reminder, remember to optimize for the following on a web page:

  1. User experience: Google ranking considers the end user’s experience. For you, this means a clean, functional, and easy-to-use web page. However, this also means that you should be offering deep, tactical content to your end user. With B2B SEO, your visitors will be CEOs and business owners. They aren’t searching for fun, they’re in need of vital services to keep their businesses humming. Every webpage should be useful and efficient – all while moving your prospect further down the sales funnel.
  2. Site speed: Related to user experience – make sure your pages load quickly. Don’t choose a bloated design and be wary of using too many banners or pop-ups. That will slow down your page and result in a worse user experience. Pages that load more quickly also have lower bounce rates. To improve your page speed, optimize your code, enable compression, reduce redirects, and compress images to the proper size. You can test your page speed here
  3. Unique, engaging content: Content is always king. If you’re creating great content, then you’ll attract followers over time. If you continue to put out helpful information, your audience will continue to grow and your engagement will too. This may be the most difficult part of SEO, but it’s the most beneficial for the long term. You can create a blog, a podcast, or even a Youtube channel to attract a following.

Optimizing for SEO can feel like a game of cat and mouse, always chasing Google and waiting for instructions on what to do next. However, it’s a worthwhile investment in order to gain traffic organically and increase your sales. As a business owner, we know that you cannot devote all your time to staying on top of SEO – this is why you rely on us.

Please contact us today for all of your digital marketing needs!