What is a search engine? Crawl, index, rank

What is a search engine? The shortest answer is a system that finds, understands and stores pages on the web and presents the ones best suited to a query in ranked order. But for a website owner that definition is not enough on its own; what matters is knowing at which link in this chain the site can get stuck. Based on Google's own documentation, this article explains in plain language the three core stages of a search engine, namely crawling, indexing and serving results, and shows what you should check on your site at each stage. It was written to answer the question of what a search engine is for the person who makes decisions, not for a technical team.

What is a search engine: a three-stage chain

Google Search Central's guide to how search works defines the process in three stages. In the crawling stage, automated programs called crawlers download text, images and videos from pages found on the internet. In the indexing stage, the page's content is analysed and the information is stored in a large database. In the serving stage, when a user searches, pages relevant to the query are found and shown in ranked order.

These three stages are sequential: a page that is not crawled cannot be indexed, and a page that is not in the index cannot appear in results. So when a page is not showing in search, the first question should not be "why is it not ranking" but "at which link of the chain did it stop". Google also states clearly that it does not guarantee a page will be crawled, indexed or served; this holds even for a site that follows the rules.

Another point the guide underlines is that Google does not accept payment to crawl a site more often or to rank it higher. Ads are a separate system, and organic results cannot be bought. This also explains why SEO work takes time: ranking is not a payment but the result of a page successfully passing every stage of the chain.

Crawl: how a search engine finds pages

Crawlers mostly find new pages through links: a link from a known page to a new one is the most common route to discovery. Sitemaps also help with this discovery. According to Google's sitemap documentation, a sitemap is a file that gives the search engine information about the important pages and files on a site; it is especially useful for sites that are large, new or have few external links.

The main tool for controlling crawling is the robots.txt file. There is a common mistake here: robots.txt is not a way to hide a page from Google. According to Google's documentation, robots.txt blocks crawling but not indexing; a URL that receives links from other sites can appear in the index without being crawled. To keep a page out of search, you need to use noindex or protect the page with a password.

What to check on your site at this stage: do important pages have internal links, is the sitemap up to date and does it contain only the pages you want indexed, and is robots.txt accidentally blocking an important section? We also covered how to build sitemap architecture on multilingual sites in our multilingual sitemap article.

Index: indexing and the canonical version

A crawled page is analysed before it is indexed: the text, headings, images and structure of the page are examined. An important part of this stage is sorting out duplicate content. Google groups pages that are very similar to each other and selects one of them as canonical, that is, the main version to be represented in results. Having the same content at several different addresses on a site can lead the search engine to choose a version you did not prefer.

Indexing problems are often silent: the page is live, users can reach it, but it does not appear in search. The way to see this is the page indexing report in Search Console. We walked through reading the statuses in this report and the order of fixes step by step in our indexing problems article.

Rank: how results are ordered

In the last stage, when a user types a query, the search engine looks for matching pages in the index and shows the results it judges most relevant and highest quality. According to Google's guide, many factors play a role in this assessment, such as the user's location, language and device. The same query can give different results in different cities or on different devices; that is why a manual check from a single computer is not a reliable measurement of rankings.

Nor is the results page made up only of ten blue links. Depending on the intent of the query, different features such as local results, images, videos or AI-assisted summaries can appear. We discussed what search engine optimisation turns into in the AI search era in our article on what SEO is.

The full list of ranking factors and their weights is not public; Google shares only general principles and guidelines. So claims of the "fix this factor and you will reach first place" kind should be treated with scepticism. The reliable approach is to make sure the page genuinely answers the user's question, is technically accessible and clearly shows experience and expertise on the topic.

A practical way to understand the search engine's point of view is to examine your own site through its eyes. The URL Inspection tool in Search Console shows how Google sees a particular page, when it last crawled it and which canonical version it chose. This information lets you explain why a page is not showing in search with data rather than guesswork.

Being in the index and appearing for a particular query are also different things. An indexed page that is not judged relevant enough for any query may hardly ever appear in results. The performance report in Search Console reveals which queries a page earns impressions for; if there are no impressions, the problem is often not technical but a mismatch between the content and the query.

A practical checklist for site owners

Turning the chain into a checklist makes it quick to find which stage a problem is in. For crawling: are important pages reachable through internal links, and do robots.txt and server responses allow crawling? For indexing: are pages free of noindex, do canonical tags point to the right version, and are there unexpected exclusions in Search Console? For serving: does the page really meet the intent of the target query, and in what way does it differ from competing results?

The first two parts of this list concern the technical infrastructure and often need to be solved before content work; good content produces no results on a page that cannot be crawled or indexed. Building this technical foundation and auditing it regularly is the starting point of our technical SEO service.

Sources

• Google Search Central — How Google Search works: the crawling, indexing and serving stages.
• Google Search Central — Introduction to robots.txt: how robots.txt controls crawling, and its limits.
• Google Search Central — What is a sitemap: when sitemaps are useful.