Search engines and AI assistants look for a few standard files at the top of every website. Your Mount Olympus site makes all of them for you and keeps them up to date. There's nothing to upload or edit.
Where to find them
Replace yourdomain.com with the site's address (its own domain, or its free address):
- https://yourdomain.com/sitemap.xml: the list of pages you want in search results.
- https://yourdomain.com/robots.txt: the rules for which bots may read the site.
- https://yourdomain.com/llms.txt: a plain-text guide to the site for AI assistants.
- https://yourdomain.com/blog/feed.xml: the blog's RSS feed, when the site has a blog. Each Live language has its own, such as /es/blog/feed.xml.
They always use the site's main address. When the site has its own domain, the links inside use that domain.
How to submit the sitemap to Google
The easy way is Search Console setup:
- In the editor, click Settings, then Google Tools.
- Follow the Google Search Console steps. The last step submits the sitemap for you.
- Later, click Resubmit sitemap after big changes, such as many new pages.
See Connect Google Search Console. If you manage Search Console yourself, add sitemap.xml under Sitemaps there. Other search engines find the sitemap from the Sitemap: line in robots.txt.
What's in the sitemap
The sitemap lists:
- Every published page, with the homepage marked as changing weekly (priority 1.0) and other pages monthly (priority 0.7). Each entry has the date it last changed.
- One address per item of a dynamic (collection) page, instead of the dynamic page's own address.
- Every live blog post.
- On a multi-language site, each page and post in every Live language, at its address in that language. Each entry links to its other language versions (hreflang), plus an x-default pointing at the main language.
It leaves out:
- Folders, popups, blog layouts and the custom 404 page.
- Pages that are drafts (not published).
- Pages set to No index in their page settings, and blog posts set to No index.
- The generated privacy page.
- Items of a dynamic page set to No index, and items whose canonical URL points somewhere else.
- Pages and posts in Draft languages, and pages hidden in a language.
What's in robots.txt
For a live site it lets every search engine in, keeps them out of /api/ and /admin/, adds the AI crawlers rules from AEO Settings, and ends with the sitemap's address.
When the site isn't published, or No index for the entire site is ticked in SEO/AEO → Additional Settings, robots.txt blocks everything instead:
User-agent: *
Disallow: /
See AEO settings, llms.txt and AI crawlers for the AI crawler choices.
What's in llms.txt
The site's name and description, your optional introduction, then a list of pages (and dynamic items) with their titles and descriptions, a section per Live language, and the blog posts. You can turn it off, add an introduction, leave pages out and leave out the blog in SEO/AEO → AEO Settings.
What's in the blog feed
The 50 newest live posts, with title, link, date, author, tags, summary, main image and the full post. The feed's title, description and image come from the blog's settings. See Blog.
When they update
All four files are built from the live site each time they're requested. You never need to regenerate them. Search engines may see a copy that's up to an hour old (the blog feed, up to 15 minutes). New pages appear after you publish them.
When the site has its own domain, IndexNow is also on: Bing and Yandex are told about changed pages each time you publish.
Examples
- A new service page isn't showing in Google: publish it, check it's in yourdomain.com/sitemap.xml, then click Resubmit sitemap in Google Tools.
- A thank-you page shouldn't be in search: set it to No index in its page settings. It drops out of the sitemap and llms.txt.
- A site still being built on a live domain: tick No index for the entire site. robots.txt blocks all bots and the sitemap stops being served. Untick it at launch.
- A news site wants its posts on another site: give them the blog feed address, yourdomain.com/blog/feed.xml.
Limits and what it can't do
- There's no sitemap or robots.txt editor. You can't add custom robots rules or extra sitemap entries.
- You can't upload your own sitemap.xml, robots.txt or llms.txt as a root file. Those names are taken.
- The sitemap's change frequency and priority are fixed (weekly 1.0 for the homepage, monthly 0.7 for other pages).
- Robots rules are requests. Well-behaved bots follow them; they don't stop someone determined.
- The blog feed holds the 50 newest posts.
Who can use it
The files are public, so anyone can open them. Changing what goes in them needs the matching setting: page settings and Additional Settings for No index, AEO Settings for llms.txt and AI crawlers. Client people need SEO settings for those. Setting up Search Console is agency only.
Troubleshooting
- My sitemap is 404 (not found): the site isn't published, No index for the entire site is ticked, or there are no pages to list (for example, every page is No index or a draft). Publish the site or untick the setting.
- llms.txt is 404: the same reasons as the sitemap, or Publish an llms.txt file is unticked in AEO Settings.
- The blog feed is 404: the site isn't published or has no blog. A language's feed (/es/blog/feed.xml) is 404 while that language is a Draft.
- A page is missing from the sitemap: check it's published, not set to No index, not a folder, popup or 404 page, and not the generated privacy page. For a dynamic item, check the dynamic page isn't No index and the item's canonical URL isn't set to another address. Then wait up to an hour.
- robots.txt says "Disallow: /" on a live site: No index for the entire site is ticked in Additional Settings. Untick it and save.
- The sitemap shows the free address, not my domain: the domain isn't the site's main address yet. Finish connecting it in Settings → Domain and publish on it.
- A change isn't showing in the file yet: wait up to an hour. Search engines and browsers may keep the older copy until then.