Multilingual sitemap architecture: locales and crawl budget
Sitemaps for a multilingual site: one file per locale or a single index, moving hreflang into the sitemap, lastmod discipline and which URLs to leave out.
By Roozbeh Nazari · CEO
The sitemap is the least discussed and most often misbuilt part of technical SEO. On a single-language site a badly built sitemap is usually harmless: Google finds the URLs by other routes anyway. On a multilingual site, the sitemap is the one structural document in which you state that four languages are four separate URL sets, which URL is the canonical version for which language, and what you do not want indexed. This article describes how we build the sitemap architecture on a site publishing in Turkish, English, Arabic and Persian, and why each decision was taken.
The rules first: the protocol's limits
The sitemap protocol sets two hard limits: a sitemap file may contain at most 50,000 URLs and may not exceed 50 MB uncompressed. Sites above these limits must split the file and combine the parts with a sitemap index file. A multilingual corporate site rarely approaches these limits; a site of a few hundred or a few thousand URLs fits in one file. Still, the decision to split is driven by manageability, not size; we come to that below.
The second rule comes from Google's own documentation: Google ignores the priority and changefreq fields in a sitemap, and uses lastmod only when it is consistently accurate. This means that writing "every page changes every day" into the sitemap achieves nothing, while writing the date a page actually changed does.
One file, or one file per locale
On small and mid-sized multilingual sites we see three architectures. First, a single file: every locale's URLs in one sitemap.xml. Simple, easy to set up, fine up to a few thousand URLs. Its weakness is that you cannot read indexing rates per locale in Search Console; the whole site appears as one row. Second, one file per locale plus an index: sitemap-tr.xml, sitemap-en.xml, sitemap-ar.xml, sitemap-fa.xml and a sitemap.xml listing them. Search Console reports each file separately; you can see what percentage of Arabic pages are indexed by comparing against the Turkish pages. Third, one file per content type: pages, blog posts and services in separate files. This model helps when you want to track how quickly blog posts get indexed separately from static pages.
Our preference: a single file while the site is under a few hundred URLs, one file per locale once it grows past that. The decision point is not size but diagnostic need: the day you start having indexing problems in one locale, you want to see that locale in its own file. We laid out that diagnostic flow step by step in our article on indexing problems; splitting the sitemap is the first tool in that flow.
Moving hreflang into the sitemap
Google accepts three ways of declaring language and regional versions: link tags in the HTML head, HTTP headers, or xhtml:link entries inside the sitemap. Head tags are common on multilingual sites but have two weaknesses: they create a block that repeats on every page and inflates page size, and a template error propagates to the whole site at once. The sitemap method gathers that block into one file and makes verification easier: you can read the reciprocal declarations of four languages in one XML file and check them with a script.
The rule is simple but often broken: every URL entry must list all of its alternates, including itself, and the alternates must be reciprocal. If the Turkish page points to its Arabic version, the Arabic page must point to the Turkish version too. For a page that exists in only three languages, the entry lists only the three versions that exist; an hreflang pointing to a page that does not exist is one of the errors we see most often. The full list of these and related errors is in our hreflang errors article. If you use the sitemap and head tags at the same time, they must not contradict each other; in practice, picking one method is safer.
Arabic and Persian URLs: encoding
The sitemap wants URLs in their percent-encoded form. A URL with an Arabic or Persian slug may look readable in the browser, but in the sitemap each letter appears as six percent-encoded characters. This has two consequences. First, file size grows much faster than with Latin-alphabet URLs; you will not approach the 50 MB limit, but the file becomes unreadable to the human eye, so verification has to be done by script. Second, very long Arabic or Persian slugs cause trouble outside the sitemap too: some server and cache layers limit URL length. In our own practice we keep the percent-encoded length of each locale's slug under 240 characters and run that check automatically before publishing.
What does not go in the sitemap
A sitemap is not a list of "every URL on the site" but a list of "the canonical URLs I want indexed". The following stay out: URLs that redirect, URLs returning 404 or 410, pages tagged noindex, pages whose canonical tag points to another URL, parameterised or sort-filtered duplicates, pages requiring login, and interstitial pages that select or auto-redirect by locale. Any of these in the sitemap inflates the "Submitted URL not indexed" rows in Search Console and hides the real problems. The cleanliness of the sitemap is the readability of the indexing report.
Crawl budget: whose problem
Google's crawl budget documentation is clear: the topic matters mainly for very large or very frequently changing sites. On a clinic or agency site of a few thousand URLs, crawl budget is not the bottleneck. Yet we hear the word budget often on multilingual sites, because the symptoms look similar: new pages get indexed late, some locales lag behind. The real cause is usually not budget but signal confusion: hreflang contradictions, canonical errors, locale auto-redirects, or junk URLs in the sitemap. Clean up those four before entering a budget discussion; in most cases the discussion ends there.
Verification and maintenance
Building the sitemap is a day's work; keeping it correct is continuous. We automate three checks. First, a script that checks weekly that every URL in the sitemap returns 200 and points to itself as canonical. Second, a count comparing the URL totals of the four locales: if Turkish shows 60 posts and Persian shows 40, then 20 posts are missing a translation or have not been published; this count doubles as an audit of the content operation. Third, a monthly check that the sitemap report in Search Console shows "success" for each locale file and that the number of discovered URLs matches the number in the file. All three checks are in our technical SEO audit checklist.
One last note: notifying Google of a sitemap by "ping" is no longer necessary; that endpoint has been retired. Submit the sitemap to Search Console once and point to it with a Sitemap line in robots.txt; after that it comes down to lastmod discipline. How to build sitemaps on multilingual sites is one of the topics we cover in the first week of our technical SEO service, because every later indexing discussion rests on the correctness of this file.