Typically, you will see the Googlebot process being displayed as a simple flow:
Follow the link to discover the page
Read HTML and index
If JS is detected, render and update the index
Delving into this in more detail, here’s a more thorough look at the process from Google’s own documentation:
The key point here is that websites heavily reliant on JS can be subject to a delay in crawling and eventual indexing, given the resources required to parse and execute from Google’s point of view. This can lead to some problems, particularly around Google’s ability to crawl, render and index websites, understand site structure, internal linking, and various meta elements.
Typical JS Challenges for Googlebot
As we’ve outlined, a number of key challenges for Googlebot during this process can impact the website's ability to perform well in Google’s search results.
Here are several more common issues:
Differences in how users see content vs Googlebot
Google might not fetch every resource
Googlebot and its Web Rendering Service continuously analyse and identify resources that don't contribute to essential page content and may not fetch such resources, so not every resource may be requested. Additionally, if a page doesn’t change much after rendering, Google may decide to not render a page in the future to save on resources. This means that Google may ignore crucial content on your page, which will hinder its ranking opportunity.
Google still can’t see everything
Potential for pages on new sites to be seen as duplicates
With app shell models (like React), the HTML return on the initial request is often limited and can be the same across all pages. This can cause problems with pages being seen as similar, or as duplicates, and not sent to the rendering queue.
All of this means that it's not a straightforward and consistent process for Google to crawl, render, and index JS-based websites. The number of failure points increases significantly, which can impact organic visibility and rankings. Solutions for how to reduce these challenges are discussed below.
If you’re encountering a JS-heavy website, your first port of call is to test whether Google is processing it correctly. There are three tests to run to determine this, and we would recommend undertaking all to ensure all your bases are covered.
Google Search Console
It’s probably best to check what Google Search Console is saying. Inspect a URL in the tool by pasting it in the top search bar, in order to answer these questions:
Is the main content being loaded?
Is the navigation visible?
Is the design and layout of the page visible?
Is the page loading without any issues or errors?
Inspect the page in Google Chrome
Here, it’s crucial to check whether the links use the correct anchor tags with an href attribute and whether they are marked up correctly. If not, Google won’t be able to crawl or discover these pages. Google can follow links only if they are an <a> tag with an href attribute. Links that use other formats won't be followed by Google's crawlers. Here’s an example of what “good” and “bad” JS links look like:
Check if the content has been indexed in Google
Despite what Google Search Console and your source code may be telling you, it’s probably best to check what is showing up in Google Search itself. Use the Google site search command in Google Search to search whether the URL itself is indexed, and then repeat this process for several sentences across different templates to ensure all the content has been indexed.
It’s important to note that if the page has just been put live, then you may be in the second phase of indexing. If it has been several weeks since you launched the page, this is likely an indicator of crawling issues. If the content cannot be found, it is an indication that Google isn’t processing the JS correctly and further investigation is required.
The crawling, rendering, indexing, and ranking process for a JS website costs Google a lot more in terms of resources to process and understand than it does for general HTML and CSS resources
Even though Googlebot has improved its ability to render JS and understand how content is being pulled in, there are a number of failure points along the way. These include how Google may decide to prioritise a particular resource or even not render a page in the future
Changes to individual pages that are only effective with JS can take longer to be reflected in the SERPs due to a longer queue process.
Google can follow links only if they are an <a> tag with an href attribute. Links that use other formats won't be followed by Google's crawlers.
Generally, in order to ensure that your site is successfully and consistently crawled, indexed and rendered by Google, it’s recommended to find a solution that enables the fully-rendered HTML to be shown to Googlebot. This can be done by looking at options such as server-side rendering and dynamic rendering, which we’ll run through below.
What is dynamic rendering?
Dynamic rendering, detailed here, is also the preferred solution for Bingbot. It enables websites to do the hard work for search engines on the server-side and deliver the rendered version for them. Users still experience the JS version:
Dynamic rendering is not considered cloaking by the search engines and it ensures they can see all the content and links without having to go through an extensive rendering process. It is good for indexable, JS-generated content that changes rapidly. There are certain tools and/or services that can help with the implementation – Prerender.io, Puppeteer, or Rendertron along with Google’s implementation guidelines.
What about server-side rendering?
Server-side rendering (SSR) offers a number of options that can be explored, depending on the application. The chart below from Google highlights these options along with their pros and cons:
Depending on the framework, there are tools that can help with the implementation of server-side rendering:
React – Next.js
Angular – Angular Universal
Vue.js – Nuxt.js
On the plus side, with SSR, all search engines (not just Google) and users are delivered the full code with all the elements needed for SEO. It can also reduce the time taken for the First Contentful Paint, a metric measured by Google for page experience, although it can increase Time to First Byte as everything happens in real-time. However, this is a trade-off worth considering.
What can I do about page speed?
What about other search engines?
This guide has specifically focused on Googlebot, and rightly so given the market share that the search engine has. However, there are other regions where other search engines play a more prominent role and indexing JS needs wider consideration. In terms of rendering JS, you can’t assume what works for Googlebot will work for others – Google is a more advanced search engine with a more advanced crawl, rendering and indexing engine.
As of documentation running up until very recently (with the below chart from Moz from 2017), it remains the case that most other search engines are unable to process JS content.
Here are some points to take away:
Understand what your client’s understanding or plans are for serving fully-rendered HTML (DOM) content to browsers and/or search engines.
Get an understanding of the current framework setup and any changes that need to be made to ensure the website is discoverable by search engines. Is moving to server-side or dynamic rendering an option? How are things being discovered by Google as it stands?
How are internal links being discovered and are they being marked up using the correct href links?