What is the Google index: why a page isn't indexed

The question of what is the Google index usually gets asked in the middle of a problem: the page is live, the link works, but it does not show up on Google. At that point most teams go straight to looking for a fix and repair the wrong thing. Yet the answer to what is the Google index is what lets you name the stage the problem is at, and a problem without a name cannot be fixed. This article is not a diagnostic flow but a vocabulary exercise: what the index is, which statuses differ from which, and what the language Search Console uses actually means.

What is the Google index: a record, not a copy

The index is a large database in which Google stores the information it extracts from the pages it crawls. There is an important misunderstanding here: the index is not a backup or an exact copy of your site. The page is crawled, its content is parsed, and the result of that parsing is stored. When a user searches, Google searches that record, not the web. So "my page is live" and "my page is in the index" are independent statements; the first concerns your server, the second Google's record.

The practical consequence is this: being able to open a page in a browser does not show that it can be found in search. Equally, removing a page from your site does not instantly destroy the record in the index. There is always a delay and an intermediary between your site and the search results. We covered the whole chain earlier in our what is a search engine article.

Discovered, crawled, indexed: three separate states

Knowing that a URL can be in three different states as far as Google is concerned eliminates half of all problems at the outset. A discovered URL is an address Google knows exists but has not yet downloaded; it usually learned of it through a link or a sitemap. A crawled URL has been downloaded and its content seen. An indexed URL has been parsed and stored, and is a candidate to appear in search results.

These three states are sequential, and each transition is a separate decision. Google is not obliged to index every page it crawls; its own documentation states plainly that it does not guarantee a page will be crawled, indexed or served, even one that follows the rules. That is why the answer to "why isn't it indexed" is usually not a technical fault but the outcome of an assessment.

It is worth adding that the clients doing the crawling are not all of one kind. Google's documentation defines common crawlers, special-case crawlers and user-triggered fetchers separately. That distinction helps when reading server logs to work out what a given request means: a manual inspection request and routine crawling are not the same thing.

There is also the state of falling out of the index. A page that was previously indexed can drop out of the record when its content changes, when it stays unreachable for a long time, or when it is no longer selected as the representative version. Index status is therefore not a badge won once and kept, but a state that changes over time; assuming a page verified three months ago is still indexed today is a mistake.

Why a page is not indexed: the language of the statuses

Search Console's page indexing report gives a status for every URL, and those statuses are not randomly chosen phrases; each points to a different cause and a different owner. Looking at a few examples shows the logic of the language.

"Crawled - currently not indexed" says Google saw the page but chose not to store it. There is no technical obstacle here; it is the outcome of an assessment, and the fix is to make the page itself a better answer. By contrast, "Discovered - currently not indexed" says the page has not yet been downloaded; that is more about crawl priority and site structure.

"Excluded by 'noindex' tag" and "Blocked by robots.txt" are deliberate instructions and usually not problems at all. Rather than panicking and removing them, the question to ask is: who put this instruction there, when, and why? A noindex left on by accident and a thank-you page deliberately kept out of the index appear in the same report under the same phrase.

Another group concerns canonicals: alternate page with proper canonical tag, duplicate without user-selected canonical, or Google choosing a different canonical than the user did. These statuses do not report a fault; they report that the same content sits at more than one address and which one Google treats as the representative. The problem arises when the representative Google picked is not the one you wanted. We worked through the diagnosis and repair order for these in our indexing problems article, and which decision the report feeds in our Search Console guide.

Being in the index does not mean being visible

This is the second common confusion. A page being in the index does not mean it will appear high for any query. A page in the index that is not found relevant enough for any query may get almost no impressions. So before diagnosing "we have an indexing problem", look at the performance report: if the page is getting impressions it is in the index, and the problem is relevance or click-through rate, not indexing.

The reverse holds too: for a page getting no impressions, verifying whether it is in the index is faster than starting work on the content. A two-minute URL inspection can make two weeks of content work unnecessary. That order is worth turning into a team rule: when a visibility complaint comes in, the status is verified first and the content discussed second.

Volume enters the picture as well. On large sites it is unrealistic to expect every page to be crawled at the same rate; crawling has a priority, and that priority is affected by site structure, internal links and how often pages are updated. On a site with thousands of pages, a handful being outside the index is not in itself a sign of a fault; what matters is the proportion and which pages are left out.

What you control and what you do not

The most useful result of this vocabulary is that it draws the boundary of your responsibility. What you can control: that the page is reachable, that it returns the right status code, that it carries no unintended noindex, that its canonical tag points to the version you want, that it appears in the sitemap and is reachable through internal links. The only thing you cannot control is the decision to index a crawled page; that decision belongs to Google, and no method promising to speed it up changes that.

In practice this means splitting the work in two: removing the obstacles is your job, deserving the assessment is the content's job. Putting these infrastructure checks under a regular audit falls within our technical SEO service.

Sources

• Search Console Help — Page indexing report: reasons for non-indexing and status definitions.
• Google Search Central — Overview of Google crawlers: common crawlers, special-case crawlers and fetchers.
• Google Search Central — Search Essentials: technical requirements and key best practices.