The Strange Case of WebP Images Reversed to the Crawl Queue

Following the recent rollout of the August Core Update, my blog seodepths.com has been welcomed with a rocketing rise in pages crawled-not indexed.

The page indexing report in Google Search Console pinned the spike on featured images and post infographics that were in WebP format.

After a week of deep digging, I realised I was just opening Pandora’s box and making things worse, like opening a right can of worms.

In this post, I’ll walk you through the root causes and some advanced fixes I tried to get my WebP images back on Google search.

The Inshees of WebP Banshees

We’re not alone in this.

WebP images have reportedly been classed as Crawled, Not Indexed pages on the Search Console properties of plenty of other webmasters or SEO struggling with fixes.

💡 Tip

I even tried to funnel the chaos causing pages to get crawled and not indexed in an old post I recommend you reading.

It’s written in the stars: Google identifies WebP images based on the HTTP header from the server and then they index WebP images as image files, not HTML pages.

Regardless of how Google indexes WebP, they may add up to the crawl queue especially if they are linked to unclear pages (e.g., .php URLs) – although I am wary of considering .xml as an unclear format.

Ok, but what’s the point of dumping .webp images in the Page Indexing report on Search Console, if the tool is designed to report exclusively on HTML pages ?

You might argue it’s simply one of the reporting bugs that have been normalised from our old friend Google Search Console.

So, should you stick with WebP images or play it safe with those hefty JPGs and PNGs?

WebP’s got some positive sides. For example, it can reduce bandwidth needs and support lossy and lossless compressions.

However, it could increase the disk space needed on the server side because these files will be added on as additional images

💡Tip

Be careful with WebP images, as not all browsers (like IE) support them—especially consider your audience’s OS and browser preferences, particularly for mobile users.

I touched on this when discussing images optimization for eCommerce

Thankfully, WebP is a broad condoned format across most operating systems and browsers as of now.

WebP Reversed to “Crawled – Not Indexed”

Shit happens all the time and here’s how it all started.

Starting from August 14th, Googlebot kicked my WebP images out of the index, sending them back to the crawler. Funny enough, this happened just a day before the August Core Update was rolled out.

The moment I saw a spike in ‘Crawled, not indexed’, I knew something was off. But it wasn’t down to quality or a sudden shake-up in rankings

Turns out all my WebP images were linked to XML sitemaps that Google couldn’t sort out, even though they were actually pointing to valid and indexable sitemaps!

Google should definitely improve the accuracy of the error labels issued on the Search Console, but that’s not the point of this post.

There’s more.

This is like leaving your kettle on the hob, nipping to the shop, and coming back to find your whole flat up in flames. No matter how much you blame something else, your place is burnt to the ground, and deep down, you know it’s on you.

I ended up shooting myself in the foot by keeping too many unused plugins on my back end, like Jetpack and RankMath, which both handled XML sitemaps.

I even had a WebP image compressor plugin installed, but it was pointless since I was already manually converting images before publishing each post.

To top it all off, I found out that the images were still hanging around in the index cache, easily spotted with a site: search on Google Images.

This drove me up the wall, but at least it confirmed the issue was a false positive. The WebP images have vanished from the live index, but they’re still lingering in Google Image search’s cached version—for who knows how long.

In short, the images are still technically indexed and could show up as long as Google’s cache for certain searches doesn’t get cleared.

A couple of Plugins for a One-Size-Fits-All purpose

Not only did I rely upon Jetpack and RankMath to load my XML sitemaps but I’ve been using a WebP compressor plugin despite pre-compressing them manually before publishing content.

For the records:

• https://seodepths.com/sitemap_index.xml is RankMath’s XML sitemap that returned HTTP 404 as of the 14th of August.
• https://seodepths.com/sitemap.xml is Jetpack’s sitemap, as in the one containing the WebP image files and remained fully available to Googlebot throughout August.

Again, what’s the point of reversing WebP files to the crawler despite being stored in a fully accessible sitemap?

So, I started wondering if I was sending mixed signals by using two different XML sitemap plugins.

Someone on LinkedIn got me thinking I might be messing things up. For instance, my blog posts had data-permalinks with non-existing URLs that were just redirecting to the homepage

To make things even blurrier, when checking out the cached version of the homepage, there’s an image <div> with a duplicate of the image with the loading="lazy" feature turned on, coming from cdn-gbphn.nitrocdn.com.

Nitropack-powered images have a ‘Link’ header saying the proper version of the image is the WebP one.

This mishmash of images is proper confusing, and I reckon it’s just throwing a bunch of mixed signals at search engines for no good reason.

WebP vs i0.wp.com HTTP headers

On my site, images are hard-coded with the typical i0.wp.com URI structure for all images hosted on WordPress. Of course, this is not the case for the crawled not indexed WebP images as they present a more ordinary URL with the WebP format extension.

The least I could do was compare the HTTP headers of a sample image wrapped in two different file formats.

While both headers serve WebP images and have similar core functionality, the i0.wp.com header confirms resources can be cached for a longer duration and includes SEO elements, while the .webp response emphasizes security with Strict-Transport-Security.

I found three main breakpoints from the response headers:

1. Strict-Transport-Security header
   WebP leverage the Strict-Transport-Security: max-age=31536000 response header, just like any traditional HTML file to ensure the browsers only access the site over HTTPS for one year. The i0.wp.com header does not include this security feature.
   
2. Link header.
   Images on i0.wp.com include a Link header, which points to the WebP image URL as the canonical version of the image file. This is crazy, Google is being served a hint as to what picking up as the definitive version.
   
3. X-ac vs X-nc* start caching in different ways.
   In the case of the i0.wp.com image, a cache hit indicates efficient caching and fast delivery, while the .webp image had to go through the slower process of fetching the original file. For WebP images this means the server had to retrieve the content from its origin every single time, causing cumulative delays in the server response.

💡 *FYI

The X-ac and X-nc headers indicate how content is served from a cache.

• X-ac (MISS) in the .webp response shows the request was routed through Atomic caching but resulted in a cache miss, meaning the image wasn’t found in the cache and had to be retrieved from the origin server.
• X-nc (HIT) in the i0.wp.com response indicates a cache hit, meaning the content was found in the cache and served quickly, reducing latency and server resources.

Cache hits significantly improve performance by decreasing response times and bandwidth usage, whereas cache misses slow content delivery by requiring retrieval from the origin server.

Now, Google has often said that one reason their crawlers might skip over your files is due to your server’s capacity. When you scale this up to thousands of WebP images, it makes sense why Googlebot might avoid crawling them or sending pages back to the crawler.

It’s all about managing resources, and if your server goes down for a while or has trouble resolving your DNS, this could be one of many things messing up your WebP indexing.

Was there even a Cure?

The first thing I did was to get rid of the WebP compressor plugin. Then, I purged the cache with Nitropack and attempted to disable Jetpack’s XML sitemaps generator.

Nothing happened.

In the meantime, I discovered Google couldn’t access the sitemap files via GSC. So, I reset RankMath to ensure the sitemaps returned a proper HTTP 200 status code and I was served XML sitemap files with X-Robots-tag set to noindex.

💡Tip

It’s not a big deal if your XML sitemaps are set to noindex as long as it is accessible to search engines for collecting all the important links within the file. Applying noindex via HTTP header is a precaution to minimize security breaches and scraping

To speed things up, I added the missing sitemaps to the robots.txt file.

After 3 days, Googlebot finally came back—only to crawl /wpautoterms/ AKA the most useless sitemap of the bunch!

Fair play, Google.

💡Tip

I am not the best man to suggest plug-ins as you’ve noticed, but trust me Bot Logger is just legit to have installed if you have a WordPress website. The plugin offers algorithms designed to record all logs from Googlebot and Bingbot.

After a whole week, Googlebot resumed access to the XML sitemaps.

To sum up, I did everything I could think of:

• Reset RankMath’s XML sitemap and restored access to the XML files
• Flushed the website’s cache
• Removed WebP compressor plugins
• Disabled and re-enabled Jetpack’s XML sitemap generator

And the results were pretty delusional.

Over the next 10 days, Googlebot kept refreshing its sample of WebP pages, still marking them as Crawled, not indexed

Even though the referring page had no reason to be invalid anymore.

More Doubts than Answers

It seems like WebP images piling up in the crawl queue might point to iffy links or unclear XML sitemaps positions, but evidence confirms this assumption only scratches the surface.

Long story short, the issue was caused by WebP images whose sitemaps Google couldn’t resolve, even though they were valid.

After ditching Jetpack sitemaps and resetting RankMath, I got Googlebot back on crawling. But the cherry on top was finding out the images were still hanging about in Google’s cache via a site: search.

After resetting the sitemaps, it took Googlebot a few days to crawl back on. I reckon the spike in crawled, not indexed pages was down to a classic Google Search Console reporting bug, especially since some of my XML sitemaps had returned 404 errors.

I guess the real kicker was that WebP images weren’t getting cached properly, unlike images on i0.wp.com, which were served faster.

• Could it be that storing original and WebP versions on your backend can compromise a website’s image management?
• Could the caching issues be stopping Googlebot from crawling WebP images?

I can’t prove it, but these are clues worth considering.