How to view websites as Googlebot using Chrome
Still struggling to ensure Googlebot crawls and indexes your website correctly? For technical SEO, rendering issues (especially on JavaScript-intensive websites) can lead to ranking drops and hidden content.
Why should I view websites as Googlebot?
In the past, technical SEO audits were relatively simple, with websites relying on HTML and CSS, and JavaScript limited to minor enhancements like animations. Today, entire websites are built using JavaScript, shifting the workload from the server to the browser. This means search bots, including Googlebot, must render pages on the client-side, a resource-intensive and often latency-prone process.
Search bots often struggle with JavaScript. For example, Googlebot processes the raw HTML first, which can take days or even weeks to fully render the JavaScript content, depending on the specific website. Some websites circumvent these challenges by using dynamic rendering, providing a server-side version for bots and a client-side version for users.
A brief rant
Generally, this setup makes websites overly complex and introduces more technical SEO issues than server-side rendered or traditional HTML websites. Thankfully, the adoption of dynamically rendered websites is declining.
While there are exceptions, I don't think client-side rendered websites are a good idea. Websites should be designed based on the minimum device specifications and progressively enhanced via JavaScript to improve the experience for users with devices capable of handling additional features.
My anecdotal evidence suggests that client-side rendered websites are often more difficult to access for users who rely on accessibility solutions like screen readers. Various studies support this view, although the research I've seen has been conducted by companies and charities dedicated to accessibility (and I think any bias is justified for the benefit of everyone). However, there are also instances where technical SEO and usability overlap.
Good news
Viewing a website as a Googlebot allows you to detect discrepancies between what the bot sees and what the user sees. While these views don't need to be exactly the same, key elements such as navigation and content must remain consistent. This approach helps identify indexing and ranking issues caused by rendering limitations and other quirks specific to search bots.
Can we see what Googlebot sees?
No, not entirely.
Googlebot uses a headless version of the Chrome browser to render web pages, but even using the techniques described in this article, its behavior cannot be perfectly replicated. For example, Googlebot's handling of JavaScript can be unpredictable.
A significant bug in September 2024 caused Google to fail to detect meta `noindex` tags in the client-side rendering code of many React-based websites. Such issues highlight the limitations of simulating Googlebot, especially regarding important SEO elements such as tags and main content.
However, our goal is to simulate Googlebot's mobile-first indexing as closely as possible. To achieve this, I used the following toolkit:
-
A browser for direct Googlebot imitation.
-
Screaming Frog SEO Spider mimics and renders as Googlebot.
-
Google tools, such as URL Inspection in Search Console and Rich Results Test for screenshot and code analysis.
It's worth noting that Google's tools, especially after switching to the "Google-InspectionTool" user agent in 2023, don't perfectly reflect what Googlebot sees. However, when used in conjunction with the Googlebot browser and SEO Spider, they are extremely useful for identifying potential problems and troubleshooting.
Why use a separate browser to view websites as Googlebot?
Using a dedicated Googlebot browser can simplify technical SEO audits and improve the accuracy of results. Here's why:
1. Convenience
A dedicated browser allows you to quickly emulate Googlebot without relying on multiple tools, saving you time and effort. Switching user agents within standard browser extensions is inefficient, especially when reviewing websites with inconsistent server responses or dynamic content.
Furthermore, some Googlebot-specific Chrome settings are not persisted across tabs or sessions, and some settings (such as disabling JavaScript) can interfere with other tabs you are working on. You can use a separate browser to circumvent these challenges and streamline the review process.
2. Improve accuracy
Browser extensions can unintentionally alter a website's appearance or behavior. A dedicated Googlebot browser minimizes the number of extensions required, reducing distractions and ensuring a more accurate simulation of the Googlebot experience.
3. Avoid mistakes
In standard browsers, it's easy to forget to disable Googlebot spoofing, which can lead to website malfunctions or being blocked. I was even blocked by a website for spoofing Googlebot and had to email them my IP address to get unblocked.
4. Maintain flexibility in the face of challenges.
For years, my Googlebot browser worked flawlessly. However, with the rise of Cloudflare and its stricter security protocols for e-commerce websites, I often had to ask clients to add specific IPs to their allowed lists so I could trick Googlebot while I tested their websites.
If whitelisting wasn't an option, I'd switch to alternatives like Bingbot or DuckDuckBot user agents. While these weren't as reliable as mimicking Googlebot, they could still yield valuable insights. Another alternative was inspecting the rendered HTML in Google Search Console. Although Google Search Console's user agent differed from Google's crawler and had some limitations, it remained a reliable way to simulate Googlebot behavior.
If I needed to audit a website that blocked non-Google Googlebots, and my IP address was allowed access, the Googlebot browser remained my preferred tool. It's more than just a user agent switcher; it offers the most comprehensive way to understand what Googlebot sees.
Which SEO audits are useful for Googlebot browser?
The most common use case for Googlebot is auditing websites that rely on client-side or dynamically rendered content. It can directly compare what Googlebot sees with what a regular visitor sees, highlighting differences that could impact your website's search result performance.
Given my recommendation to limit the number of browser extensions to a necessary few, it can also more accurately test the website experience for real Chrome users compared to browsers loading extensions, especially when using Chrome's built-in DevTools and Lighthouse for speed audits.
Even for websites that don't use dynamic rendering, you never know what Googlebot might discover by deceiving it. In over eight years of experience auditing e-commerce websites, I'm still amazed by the unique issues I've encountered.
What should you investigate during the Googlebot review process?
-
Navigation Differences: Is the main navigation consistent between the user's and the robot's views?
-
Content Visibility: Can Googlebot see the content you want it to index?
-
JavaScript Indexing Delay: If your website relies on JavaScript rendering, will new content be indexed quickly (e.g., for events or product launches)?
-
Server Response Issues: Is the URL returning the correct server response? For example, an incorrect URL might show Googlebot a 200 OK, but a 404 Not Found to visitors.
-
Page Layout Changes: I often see people tricking Googlebot by displaying links as blue text on a black background. This design is machine-readable but very user-unfriendly. If Googlebot cannot render your website correctly, it won't know which content to prioritize.
-
Geographic Redirects: Many websites redirect based on geographic location. Since Googlebot primarily crawls from US IP addresses, it's crucial to verify how your website handles such requests.
The level of detail in the audit depends on the specific circumstances, but the Chrome browser has many built-in tools for technical SEO audits. For example, I often compare data from the "Console" and "Network" tabs to identify discrepancies between visitor traffic and Googlebot's metrics. This process can capture files blocked by Googlebot or missing content that might be overlooked.
How to configure your Googlebot browser
Setting up the Googlebot browser takes about 30 minutes and makes it easier to view web pages as Googlebot. Here's how:
Step 1: Download and install Chrome or Canary
-
If Chrome is not your default browser, you can use it as the Googlebot browser.
-
If Chrome is your default browser, please download and install Chrome Canary.
Canary is a development version of Chrome that Google uses to test new features. It runs independently of the default Chrome installation and is easily recognizable by its yellow icon, which symbolizes the canary once used in mines to detect toxic gases.
Although Canary is marked as "unstable," I haven't encountered any problems using it as my Googlebot browser. In fact, it offers some beta features that are useful for review. If these features are also implemented on Chrome, you'll be ahead of the curve and impress your colleagues who don't use Canary.
Step 2: Install browser extension
To optimize your Googlebot browser, I recommend installing five key extensions and a bookmarking tool to improve my Googlebot experience. These tools emulate Googlebot and improve technical SEO audits, with three being particularly useful for JavaScript-intensive websites. Here they are:
An extension that simulates Googlebot:
-
User Agent Switcher: Switches your browser's user agent to mimic Googlebot's behavior.
-
Web Developers: Allows you to easily turn JavaScript on or off, providing insight into how Googlebot processes your website.
-
Windscribe (or your preferred VPN): Simulates Googlebot's location, typically in the US, ensuring location-based differences are taken into account.
Other collections:
-
Link redirect tracking: Quickly examine server responses and HTTP headers for technical SEO auditing.
-
View rendered source: Compare the raw HTML (content delivered by the server) with the rendered HTML (content processed by the browser).
Bookmark:
- NoJS Side-by-Side : Makes it easier to spot differences when comparing the appearance of web pages with and without JavaScript enabled.
Before moving on to step 3, I will break down the extensions I just mentioned.
User-Agent Switcher Extension
User-Agent Switcher works exactly as its name suggests: it switches your browser's user agent. While Chrome and Canary browsers have built-in user agent settings, these only apply to the currently active tab and reset when the browser is closed. Using this extension ensures consistency across sessions.
I retrieved the Googlebot user agent string from Chrome's browser settings, which is the latest version of Chrome at the time of writing (note that below I'm retrieving the user agent from Chrome, not Canary).
Configure the user agent switcher:
1. Obtain the Googlebot user agent string:
- Press F12 or go to More tools > Developer tools to open Chrome DevTools.
- Navigate to the "Network" tab.
- From the Network Burger menu in the upper right corner, select More tools > Network conditions.
- In the Network Conditions tab:
- Uncheck "Use browser default settings".
- Select “Googlebot Smartphone” from the list.
- Copy and paste the user agent from the field below the list into the User Agent Switcher extension list (see another screenshot below). If your primary browser is Chrome, remember to switch it to the default user agent.
- An extra tip for Chrome users:
- If Chrome is your Googlebot browser while you are here, check "Disable caching" in DevTools for more accurate results during testing.
2. Add the user agent to the extension:
- Right-click the "User Agent Switcher" icon in your browser toolbar, and then click "Options" (see screenshot below).
- The “indicator” is text in your browser’s toolbar that displays the user agent you have selected. Paste the Googlebot user agent string into the list and label it (for example, “GS” means Googlebot smartphone).
- Alternatively, add other user agents, such as Googlebot Desktop, Bingbots, or DuckDuckBot, for broader testing.
Why deceive Googlebot's user agent?
Web servers identify browsers using user agent strings. For example, the user agent for a Windows 10 device using the Chrome browser might look like this:
Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, such as Gecko) Chrome/131.0.0.0 Safari/537.36
If you're curious about the history of user agent strings and why other browsers' user agents appear in Chrome, you might find resources like the history of user agent strings interesting.
Web Developer Extensions
Web Developer extensions are essential tools for technical SEO, especially when auditing websites with abundant JavaScript content. In my Googlebot browser, I periodically turn JavaScript on and off to simulate how Googlebot processes web pages.
Why disable JavaScript?
Googlebot does not execute all JavaScript when it first crawls a URL. To understand what it sees before rendering JavaScript, disable JavaScript. This allows the raw HTML content to be displayed and helps identify key issues, such as missing navigation or content that relies on JavaScript for display.
By using this extension to toggle JavaScript, you can gain insights into how your website performs to search engines during the crucial first crawl.
Windscribe (or other VPNs)
Windscribe or any reliable VPN is very useful for simulating a typical US location for Googlebot. I use a Windscribe Pro account; their free plan includes up to 2GB of data per month and offers multiple US locations.
Tips for using a VPN in Googlebot browser:
- Location doesn't matter: Googlebot primarily scrapes data from the US, so any US location will do. For fun, I'm imagining Gotham City as real (and without villains).
- Disable unnecessary settings: Windscribe's browser extension blocks ads by default, which may interfere with webpage rendering. Please ensure that the two icons in the upper right corner are showing zero.
- Use the browser extension via the application: The VPN extension links location spoofing to your Googlebot browser, ensuring your standard browsing is unaffected.
These tools, when paired with a user agent switcher, enhance your ability to emulate Googlebot, revealing content discrepancies and potential indexing issues .
Why would you spoof Googlebot's location?
Googlebot primarily crawls websites originating from US IP addresses, and there are several reasons why it mimics this behavior during the review process:
- Location-based blocking: Some websites block US IP addresses, meaning Googlebot cannot crawl or index these IP addresses. Spoofing a US IP address ensures you access the website in a way that resembles Googlebot.
- Location-specific redirects: Many websites offer different content based on geographic location. For example, a company might have separate websites for Asia and the United States, with US visitors automatically redirected to the US website. In this case, Googlebot might never encounter the Asian version, resulting in it not being indexed.
Other Chrome extensions that can be used to audit JavaScript websites
In addition to basic tools such as User-Agent Switcher and VPN, I also rely on the following tools for technical audits:
- Link redirection tracking : Displays server responses and HTTP headers to help troubleshoot technical issues.
- View the rendering source : Compare the raw HTML (served by the server) and the rendered HTML (processed by the browser) to help you discover differences between what the user and Googlebot see.
- NoJS Side-by-Side Bookmarks : Allows you to compare web pages with and without JavaScript enabled, and display them side-by-side in the same browser window.
Okay, let's go back to step 3.
Step 3: Configure browser settings to simulate Googlebot
Next, we will configure the Googlebot browser settings to match content that Googlebot does not support when crawling websites.
Content not supported by Googlebot:
- Service personnel: Because users who click on search results may not have visited the page before, Googlebot does not cache the data for future access.
- Permission requests: Googlebot does not handle push notifications, camera access, geolocation requests, and similar functions. Therefore, any content that relies on these permissions will not be seen by Googlebot.
- Statefulness: Googlebot is stateless, meaning it does not retain data such as cookies, session storage, local storage, or IndexedDB. While these mechanisms may temporarily store data, this data is cleared before Googlebot crawls the next URL.
These key points were summarized from Eric Enge's interview with Martin Splitt of Google .
Step 3a: DevTools Settings
You need to adjust some settings in the developer tools (DevTools) to configure your Googlebot browser for accurate emulation.
How to open DevTools:
- Press F12, or open the hamburger menu in the upper right corner of Chrome or Canary, and then go to More tools > Developer tools.
- The DevTools window is docked in the browser by default, but you can change this setting. You can switch to the dock side or open it in a separate window using the second hamburger menu in DevTools.
Key configurations in DevTools:
- Disable caching :
- If you are using Chrome as your Googlebot browser, then you may have already done this.
- Alternatively, in DevTools, open the hamburger menu, go to More tools > Network conditions, and then check the "Disable caching" option.
- Prevent service personnel:
- Navigate to the Applications tab in DevTools.
- Under Service Workers, check the "Bypass Network" option.
Step 3b: General browser settings
Adjust your browser settings to reflect Googlebot's behavior.
- Block all cookies:
- Go to Settings > Privacy and Security > Cookies, or enter chrome://settings/cookies in the address bar.
- Choose "Block all cookies (not recommended)" – sometimes going against the grain can be fun!
- Adjust site permissions:
- In Privacy and Security , navigate to Site Settings or enter chrome://settings/content.
- With the permissions set, block location, camera, microphone , and notifications respectively .
- In the additional permissions section, disable background synchronization.
Step 4: Simulate a mobile device
Since Googlebot primarily uses mobile-first crawling , it is important to simulate a mobile device in the Googlebot browser.
How to simulate a mobile device:
- Open DevTools and click the device toolbar toggle button in the upper left corner.
- Select the device to simulate from the drop-down menu or add a custom device for more specific testing.
Key considerations:
- Googlebot does not scroll web pages; instead, it renders them using a window with a long vertical height.
- While mobile simulation is important, I also recommend testing in a desktop view, and if possible, on an actual mobile device to cross-check the results.
What's it like to view websites as Bingbot?
To create a Bingbot browser, use the latest version of Microsoft Edge and configure it using the Bingbot user agent .
Why consider Bingbot?
- Bingbot behaves similarly to Googlebot, supporting and not supporting certain content.
- Search engines such as Yahoo, DuckDuckGo, and Ecosia are all powered by or based on Bing, which makes its influence greater than many people imagine.
Abstract and Conclusion
Now you have your own Googlebot simulator. Setting up a browser to simulate Googlebot is one of the easiest and fastest ways to browse the web like a web crawler. Best of all, it's free if you already have a desktop device that can install Chrome or Canary.
While other tools, such as Google's Vision API (for images) and Natural Language API, offer valuable insights, the Googlebot browser simplifies website technical review, especially those that rely on client-side rendering.
To gain a deeper understanding of how to audit JavaScript websites and comprehend the nuances between standard HTML and JavaScript-rendered websites, I recommend reading articles and presentations by experts such as Jamie Indigo, Joe Hall, and Jess Peck. They...JavaScript SEO and its challenges offer excellent insights.
