How Google Crawls, Indexes and Decides Crawl Priority

Google finds a URL through a link or a sitemap, crawls it (mostly as a smartphone), renders it and decides whether to index it. When several URLs carry the same content, Google indexes one canonical version and shows that one. Which URLs get crawled first depends on two things: how much crawling your server can handle, and how much Google wants your pages.

Two common questions have short answers. The <priority> tag in your XML sitemap has no effect, because Google ignores it. And the version Google uses for indexing and ranking is the mobile version of your page.

None of this is advanced. SEO is built on simple fundamentals, and this is the first one: a page Google has not crawled and indexed cannot rank. It cannot be cited in AI Overviews either.

The six stages from URL to search result

Google’s guide to how Search works names three stages: crawling, indexing and serving. It helps to split them into six, because each step fails in its own way and shows up in a different report.

Discovery and crawling

There is no central registry of web pages. Google finds new URLs by following links from pages it already knows and by reading the sitemaps you submit. Google says every page you care about should have a link from at least one other page on your site. Use a normal <a href> link, because Google can generally only crawl those.

Googlebot then decides algorithmically which sites to crawl, how often and how many pages to fetch, while trying not to overload your server. Google does not accept payment to crawl a site more often.

For most sites, most crawl requests come from Googlebot Smartphone. That matches how Google indexes: it uses the mobile version of a site’s content, crawled with the smartphone agent, for indexing and ranking. Keep the same content, structured data, titles, meta descriptions and robots meta tags on mobile and desktop.

Rendering

During the crawl, Google renders the page and runs its JavaScript in an evergreen version of Chromium. Pages can wait in the render queue for a few seconds, sometimes longer. Google still calls server-side rendering or pre-rendering a great idea, partly because not every bot runs JavaScript. And if the initial HTML carries a noindex rule, Google may skip rendering altogether.

Indexing and canonicalization

At the indexing stage, Google analyzes the page’s text and key tags, such as the title element and alt attributes. It also groups pages with similar content and picks one canonical URL to represent the group. The canonical is the version that may appear in results, and it is crawled most regularly. Duplicates are crawled less often.

Some duplicate content is normal and is not a spam violation. Google chooses the canonical from signals such as HTTPS, redirects, rel="canonical" and sitemap inclusion. It calls redirects and rel=”canonical” strong signals, and sitemap inclusion a weak one. Whatever you declare is a hint, not a rule.

Duplicate versions also split your signals. For a large Gurugram real estate developer, my team at Adtric merged the www and non-www versions of the site in July 2025. Google stopped splitting links and rankings between the two copies (case study).

Indexing is not guaranteed. Google lists three common reasons it fails: low-quality content, robots meta rules that disallow indexing, and a site design that makes indexing difficult.

Serving

When someone searches, Google picks the indexed pages it judges most relevant and highest quality. An indexed page may still not appear if it does not match the query or is low quality. Crawling gets you in the door, nothing more. In 2017, Google’s Gary Illyes wrote that crawling is not a ranking signal.

How Google decides crawl priority

Google does not crawl every URL it knows about, or crawl them all equally often. Its crawl budget guide describes two inputs: crawl capacity and crawl demand. Together they make your crawl budget, which Google defines as the set of URLs it can and wants to crawl.

Crawl capacity: what your server can take

Google limits how much time your server spends holding connections open for its crawler. If your server responds consistently and response times stay stable or improve, the limit goes up. If the site slows down, or returns 5xx errors or 429 rate-limiting responses, the limit goes down and Google crawls less.

Crawl demand: how much Google wants your URLs

Even with capacity to spare, Google crawls less when demand is low. Google names these factors:

• Perceived inventory. Without guidance, Google tries to crawl all or most of the URLs it knows about on your site. Filter, sort and session URLs can soak up that effort.
• Popularity. More popular URLs tend to be crawled more often.
• Staleness. Google wants to recrawl pages often enough to pick up changes.
• Site-wide events. A site move can trigger an increase in crawl demand.

On staleness, Google learns how often content seems to change and revisits as needed. That decides how often Google checks, not how well you rank. I disagree with the common advice that you must publish three times a week to rank.

A domain move shows these factors at work. My team at Adtric moved a small finance bank to a .bank.in domain, live from 29 September 2025. Google showed the new domain in brand searches within two days. Most other pages started appearing in December, and weekly impressions rose from under 20,000 to over 250,000 by late January (case study).

Google does not explain the order for any single site. Its guide only says that popular URLs tend to be crawled more often, and that a site move raises crawl demand. If you are planning a move, my website migration SEO page explains how I approach it.

What does not change crawl priority

• The sitemap <priority> tag. Google ignores <priority> and <changefreq> values. Setting your homepage to 1.0 and your posts to 0.5 changes nothing.
• A <lastmod> date you cannot back up. Google uses lastmod only when it is consistently and verifiably accurate. Change it for significant updates to content, structured data or links, not for a new copyright year.
• A sitemap on its own. Google calls submitting one “merely a hint”. It does not guarantee that Google will download it or crawl every URL in it.
• Noindex as a crawl saver. Google still requests a noindexed page, then drops it.
• Asking again and again. Requesting a recrawl of the same URL many times will not get it crawled any faster.

Many articles on crawl prioritization, this one’s earlier version included, lean on a 2009 Google patent called “Managing URLs”. A patent shows an idea Google filed, not how Googlebot works today. This guide sticks to what Google documents.

What you can do to steer crawling

• Link to important pages from pages Google already crawls often, such as the homepage, menus and related articles.
• Keep filter, sort, session and calendar URLs under control. Overly complex URLs can waste Googlebot’s bandwidth and leave content unindexed. Block unimportant ones in robots.txt if you cannot consolidate them.
• Consolidate duplicates with redirects or canonical tags, and link internally to the canonical URL.
• Return a 404 or 410 for pages removed for good, and avoid long redirect chains.
• Keep the server fast and stable, and keep sitemaps current with honest lastmod dates.

Who actually needs to worry about crawl budget

Most site owners do not. Google wrote its crawl budget guide for three kinds of site, and calls the numbers a rough estimate.

• Large sites with more than 1 million unique pages whose content changes about once a week.
• Medium or larger sites with more than 10,000 unique pages whose content changes daily.
• Sites with a large share of their URLs marked “Discovered – currently not indexed” in Search Console.

If your pages seem to be crawled the same day they are published, Google says you do not need the guide. Keeping your sitemap up to date and checking the Page indexing report regularly is adequate. The Crawl Stats report carries a similar note: with fewer than a thousand pages, you should not need it.

So if a business site with a few hundred pages has pages missing from Google, crawl budget is unlikely to be the cause. Look at the fundamentals first: content quality, duplicates, internal links, the sitemap and broken links. My SEO checklist for new websites covers these at launch.

Fixing fundamentals alone can be enough. At Adtric, we started work on an education website in August 2022. Google was indexing the site incorrectly, the sitemap was faulty and there were broken links. We fixed indexing so Google saw the right pages, rebuilt the sitemap and fixed the broken links.

We added no new content. Organic traffic rose 163% in three months (case study).

“Crawled – currently not indexed” vs “Discovered – currently not indexed”

These two statuses in the Page indexing report look alike but point to different stages. “Discovered” means Google knows the URL but has not crawled it. “Crawled” means Google fetched the page and chose not to index it, at least for now.

Resubmitting an unchanged page does not help. The previous version of this article, at about 4,000 words, sat in “Crawled – currently not indexed”. Length did not save it, which is why this version is shorter and built on Google’s own documentation.

Neither status is always a problem. Google says it is fine for a URL not to be indexed for the right reasons. Do not expect every URL on your site to be indexed. Worry when the list includes pages you need, such as services, products, admissions or key articles.

How to check what Google crawled and indexed

When I audit a site, one of my first checks is a site:domain.com search on Google. Then I open Search Console and GA4 if I have access. Google says site: results are not always exhaustive, so treat that search as a quick view only.

URL Inspection for one page

The URL Inspection tool shows Google’s indexed version of a single URL. Check these fields:

• Whether the URL is on Google, and if not, the reason given.
• User-declared vs Google-selected canonical. If they differ, Google chose another URL to represent the page.
• Crawled as, which shows whether Google used the desktop or the smartphone crawler.
• Last crawl, page fetch and indexing allowed, which show whether Google reached the page and whether a noindex rule stopped it.
• Referring page and sitemaps, which show how Google found the URL.

The live test checks whether Google can access the page now, and only the live test shows a screenshot of the rendered page. You can request indexing, but there is a daily limit and no guarantee.

The Page indexing report for the whole site

This report shows how many URLs Google has crawled and indexed, and why the rest were not. Two habits make it far more useful:

• Use the sitemap filter. “All submitted pages” shows only URLs in your sitemaps. “Unsubmitted pages only” shows URLs Google found some other way. Pages you need in the second list are missing from your sitemap.
• Click Validate fix once. Google says validation typically takes up to about two weeks, and sometimes much longer.

If you run a large site, also open the Crawl Stats report under Settings. It shows crawl requests, average response time and host status, and splits crawls into discovery of new URLs and refresh of known ones.

Check more often after big changes

After a migration or relaunch, these checks become daily work. For the Gurugram developer, the relaunch changed every URL, and my team ran daily index checks in Search Console through March 2025. For the bank, we tracked brand searches daily in the first week and the rest of the site over the next three months.

Sitemaps also break quietly, and my own site is the proof. In October 2026, an audit of ujjwalganesh.com found that the sitemap index had stopped listing the book and essay sitemaps. Only 11 of 26 book pages and about half of the essays were indexed.

The missing sitemaps have been restored and resubmitted. A sitemap is only a hint, so this reopens one route back rather than guaranteeing a result. Google still has to recrawl those pages, and I cannot report the outcome yet.

AI search uses the same crawl and index

Google’s AI Overviews and AI Mode draw on the same index as regular results. Google says there are no additional requirements to appear in them. A page must be indexed and eligible to be shown in Google Search with a snippet.

Google’s guide to optimizing for generative AI features says they are rooted in its core Search ranking and quality systems. They retrieve pages from the Search index. If your site is not retrievable by Google, it is invisible to AI Overviews as well. I made that point in a post on AI search and SEO, and it still holds.

You will read that AI search needs a special “machine-readable” site structure. Part of that holds up. Google’s guidance supports:

• Crawlable content and links. Google can process content in JavaScript as long as it is not blocked.
• Readable URLs, with words rather than long ID numbers.
• One canonical URL for each piece of content, served over HTTPS.
• Clear paragraphs, sections and headings, which Google says people appreciate.
• A current sitemap, and a robots.txt that does not block what you want found.

The rest does not hold up. Google says you do not need llms.txt or other AI text files, special markup or Markdown. There is no requirement to chunk content into tiny pieces. Structured data is not required for generative AI features, though it can still make a page eligible for rich results.

Traffic from AI features is counted in the Search Console Performance report, under the Web search type. My view is simple: AEO and GEO are not separate disciplines. They are SEO, extended. My guide to ranking in AI search results covers the rest.

What to do next

Start with the pages that bring you enquiries or sales. Run your top ten through URL Inspection. Confirm that each one is indexed, was crawled as a smartphone, and that Google picked the canonical you declared.

Then open the Page indexing report, filter by “All submitted pages”, and read every reason listed against pages you need. Fix the fundamentals before you worry about crawl budget.

If the problem runs wider than a handful of URLs, a website SEO audit checks crawling, indexing, canonicals and sitemaps in one pass. It also tells you which fixes to make first.