Zentury Spotlight – Google Introduces New Web Crawler

Google Officially Finished April 2023 Reviews Update

Google officially stopped rolling out the April 2023 Reviews Update. 

The update was significant for the fact that it no longer affects only products, but it examines websites that have recommendations for services, travel destinations, movies and tv shows, games, and other things.

Let us remind, what new changes Google introduced:

Apart from changing the name from “products reviews system” to “reviews system”, with the new update, product reviews now apply to all types of reviews. Meaning, it can evaluate reviews of any topic that is reviewable, including services and business, destinations and media.

The new documentation shares the signals of product reviews experience, which serve as an evidence that product has been used, measured and tested. The mentioned signals are:

• Audio & visual, and links to evidence of experience:

• Quantitative measurements:

With the end of the update, websites that were affected might need to reevaluate their content and strategies. Since this update was focused on reviews, if your websites do not contain reviews, something else might be an issue. 

Google Made an Update on Googlebot Verification Documentation

In association with GoogleUserContent.com, Google makes an update in the official Googlebot documentation.

Google added information about user-triggered bot visits that had been missing from earlier Googlebot documentation to their Search Central documentation on verifying Googlebot. This confusion remained for a long time as some publishers had blocked the IP ranges of the legitimate visits.

Google updated their documentation to include a classification of the three major bot types that publishers should be prepared for. So, now Google differs three categories of Google Bots:

• Googlebots (search crawlers)
• Special (case crawlers)
• User (triggered fetchers)

The last category “user” is associated with GoogleUserContent and until now Google did not possess any documentation for it, which often confused the publishers.

Google freshy updated its Google Crawlers page, and added a separate section for Triggered Fetchers. These crawlers are now listed as user-triggered fetchers:

• Feedfetcher: used for crawling RSS or Atom feeds for Google Podcasts, Google News, and PubSubHubbub.
• Google Publisher Center: fetches and processes feeds that publishers explicitly supplied through the Google Publisher Center to be used in Google News landing pages.
• Google Read Aloud: upon user request, Google Read Aloud fetches and reads out web pages using text-to-speech (TTS)
• Google Site Verifier: Google Site Verifier fetches upon user request Search Console verification tokens.

Google Introduces New Web Crawler – GoogleOther

Google worked on optimization of Googlebot’s performance and introduced a new web crawler – GoogleOther.

The new web crawler is designed to reduce strain on Googlebot, its first search index crawler. It is a generic web crawler that will help Google optimize its crawling operations. Google Search Central presented GoogleOther as a crawler that may be used by various product teams for fetching publicly accessible content from websites.

Google analysts, Illyes shared on Linkedin: “The new crawler uses the same infrastructure as Googlebot and so it has the same limitations and features as Googlebot: hostload limitations, robotstxt (though different user agent token), http protocol version, fetch size, you name it. It’s basically Googlebot under a different name.”

As for implications of GoogleOther on SEO and website owners, the new crawler should not leave any significant impact on websites, as it is more or less the same as Googlebot. However, it surely is significant development for Google in its efforts to optimize its web crawling processes.

If GoogleOther concerns you, there are ways in which you may monitor its visits on your sites. You may analyze server logs in order to identify requests made by GoogleOther. It may help you understand how often it crawls your website and which pages it inspects.

You may also update robots.txt to ensure you have the control in the access and crawling behavior of GoogleOther on your websites. By monitoring crawl statistics in Google Search Console you can notice any changes in crawl frequency and budget, or the number of indexed pages. 

One thing you should always keep an eye on is your website performance. So, make sure that you regularly track your metrics including load times, user engagement, bounce rates, and others. This can help you detect any unforeseen issues on your website.

Google Updates Helpful Content Creation Guidance

Google has added a section dedicated to page experience by updating its guidance on creating helpful content. This update has the goal to help owners of the websites to consider taking a more comprehensive approach to the page experience while creating content.

Google Search Central offers question to selfasses your content’s page experience, and if the question is YES to all of the below questions, you are on the right track:

• Do pages have good Core Web Vitals?
• Are pages served in a secure fashion?
• Does content display well for mobile devices when viewed on them?
• Does the content lack an excessive amount of ads that distract from or interfere with the main content?
• Do pages lack intrusive interstitials?
• How easily can visitors navigate to or locate the main content of your pages?
• Is the page designed so visitors can easily distinguish the main content from other content on your page

These questions are not the only factors you should take into consideration, but they may help you to get the idea of overall good page experience.