Mobile-First Indexing: How Google Crawls Mobile Pages and What Can Go Wrong

Google indexes and ranks the mobile version of a site’s content, crawled with its smartphone crawler. The transition to this system is finished: on October 31, 2023, Google announced that “the trek to Mobile First Indexing is now complete” (Google’s October 2023 announcement). For a site that works on mobile devices, there is no switch left to prepare for. What remains is making sure the mobile version Google crawls contains everything you want indexed, because content, metadata or structured data missing from the mobile version is, for Google, missing from the page.

How it works

Google’s best practices page states that Google uses the mobile version of a site’s content, crawled with its smartphone crawler, for indexing and ranking (mobile-first indexing best practices). Having a mobile version is not required for inclusion in results, but Google calls it “very strongly recommended”.

Google’s 2023 announcement adds that a small set of sites that do not work on mobile devices at all are still crawled with the legacy desktop crawler “for the time being”, and that Google continues to reduce desktop crawling. The setting in Search Console that used to show which crawler indexed a site was turned off at the same time, because all sites that work on mobile are now primarily crawled with the mobile crawler.

Configuration determines where things go wrong

Google describes three ways to build a mobile-friendly site:

Google recommends responsive design “because it’s the easiest design pattern to implement and maintain”. Its parity guidance applies mainly to dynamic serving and separate URLs; with responsive design, content and metadata are identical by construction, though CSS can still hide things on small screens.

The parity audit checklist

Run these checks on representative templates (home, category, product or article, and any page type with unique components). Compare the mobile HTML that Googlebot Smartphone receives with the desktop HTML.

Access and robots

• [ ] The same robots meta tags on mobile and desktop. Google warns that a different tag on mobile, “especially the noindex or nofollow tags”, can make Google fail to crawl and index the page.
• [ ] Mobile resources (CSS, JavaScript, images) are not blocked by robots.txt, including resources on mobile-only URLs.

Content

• [ ] The primary content is equivalent. Google says that if a mobile page intentionally has less content, “you can expect some traffic loss”, and suggests accordions or tabs instead of removal.
• [ ] Headings are the same.
• [ ] Primary content does not load only after user interaction. Google states it “won’t load content that requires user interactions (for example, swiping, clicking, or typing)”.
• [ ] Lazy-loaded content loads when it enters the viewport, without interaction.

Structured data

• [ ] The same structured data types on both versions, validated on the mobile version.
• [ ] URLs inside the mobile version’s structured data point to mobile URLs where separate URLs are used.

Metadata

• [ ] Equivalent title elements and meta descriptions on both versions.

Images and video

• [ ] Images on the mobile version follow Google’s image best practices, which the mobile-first page points to explicitly, and carry the same alt text as on desktop.
• [ ] Video is embedded on the mobile version in a way Google can find.

Links

• [ ] Internal links available on desktop (including footer and navigation links) are present in the mobile HTML, not removed to save space.

Ads

• [ ] Ads do not dominate the top of the mobile page; Google’s page notes that ads at the top can take up too much room on mobile.

Separate URLs only

• [ ] Canonical and alternate annotations connect each desktop URL to its mobile URL.
• [ ] Each mobile URL redirects or serves the correct equivalent, not the mobile home page.
• [ ] hreflang on mobile URLs points to other mobile URLs.

What can go wrong

Google’s 2023 announcement names the error types it saw on sites that do not work on mobile: “the page shows errors to all mobile users, that the mobile version of the site is blocked with robots.txt while the desktop version is allowed for crawling, or that all pages on the mobile site redirect to the homepage”. Beyond those failures, the problems that cost visibility without breaking the site:

• Trimmed mobile templates. Reviews, specification tables or long descriptions removed from mobile to shorten the page are no longer indexed.
• Interaction-gated content. A “Read more” button that fetches the rest of an article only on tap leaves the remainder invisible to Google.
• Missing structured data. A mobile template built separately may omit Product, Article or breadcrumb markup present on desktop.
• Navigation reduced to a script-only menu. Links that exist only after a menu is opened by script may not be found.
• Mismatched robots rules. A noindex added to an m. subdomain to avoid duplicate content removes the indexed version entirely.

How to test

1. URL Inspection. Inspect a URL and open View crawled page to see the HTML and response Google recorded for the indexed version, which for a mobile-first site is the mobile version. Run the live test to see the current version; Google notes a screenshot of the rendered page is available only in the live test.
2. Rendered HTML comparison. Save the rendered HTML from URL Inspection and the desktop rendered HTML, and compare word counts, headings, structured data blocks and link lists. Differences point to parity gaps.
3. Lighthouse. On December 1, 2023 Google removed the Mobile-Friendly Test and the Mobile Usability report from its documentation “as they are going away” (Google Search documentation updates). Use Lighthouse in Chrome DevTools for mobile usability checks instead.
4. Rich result reports. If structured data counts in Search Console fall, check whether the mobile template lost markup.

Missing specifications on a dynamic-serving retailer

Say a retailer uses dynamic serving. The mobile HTML omits the product specification table to keep pages short, and loads customer reviews only when a user taps “Show reviews”.

• Observed: product pages rank for brand and product names but have lost impressions for specification queries such as “20 liter backpack with laptop sleeve”.
• Check: URL Inspection’s crawled HTML shows no specification table and no review text.
• Fix: include the specification table in the mobile HTML inside a collapsible section; render the first reviews in the HTML and load more on scroll rather than on tap.
• Verify: recrawl a sample of product pages; confirm the table and reviews in URL Inspection; watch impressions for specification queries over the following weeks.
• What to expect: recovery depends on recrawling and on ranking systems; parity removes the cause, it does not guarantee the previous positions.

Parity as an ongoing task

The work is a parity discipline: whatever must be indexed has to exist in the mobile version Googlebot Smartphone receives, with the same robots rules, content, headings, structured data, metadata and links. Responsive design makes that discipline easiest; dynamic serving and separate URLs need routine comparison.