Add generating sitemap index

[r3nic1e]

It would be nice to generate sitemap index that will reference sitemap.xml for all available versions. Right now you have to either specify sitemaps for each versions manually.

If that would help, I can implement it and send a PR.

[jimporter]

I'm not sure this is a good idea: it would make Google index results for all the versions (assuming Google listened), but that means that out-of-date revisions will get indexed too. On the contrary, I think the better behavior is to tell Google to prefer only the default/latest version. That's already supported in mike by setting canonical_version to whatever you prefer.

[r3nic1e]

I'm not an expert in SEO at all but I guess older versions won't dissapear from Google index as soon as newer version is published. Some sort of noindex meta should be used for that.
I have an example when having multiple versions indexed is useful - imaging supporting multiple major version (e.g. python 2 and 3 back in 2015). In that case I'd love to index all major versions alltogether.

I use canonical version and it produces latest directory with redirect pages. But there is no sitemap.xml there - https://github.com/helmwave/docs/tree/gh-pages/latest

[samuelcolvin]

It would be great if mike could generate a sitemap that covers all releases, in pydantic I've just added a root sitemap to point just to the latest release, see here, but including a link to all sitemaps would be great.

[jayqi]

@jimporter I have a use case for this and it's actually in fact complementing the canonical_version feature.

I previously was not using canonical_version on my site, and Google indexed all of my old versions of my docs. Recently, I enabled canonical_version, and I've also manually gone and edited all of the old deployments to have canonical refs to my stable version.

However, now I need to ask Google to reindex my site. Google Search Console's docs indicate that either you can (1) request that Google reindex a webpage one page at a time, which is wildly impractical, or (2) submit a sitemap. If there were a global sitemap for all releases, I'd be able to easily get Google to reindex everything with the fixed canonical refs.

See following comment.

[jayqi]

On further investigation, I don't think a combined sitemap index actually solves my problem. Based on Google's docs, it seems like you're only supposed to include canonical URLs in the sitemap. And in fact, if you're using canonical_version, all of the sitemap.xml files generated by mike will use the canonical URLs. So in the case where you're using canonical_version, having multiple sitemaps that contain the same canonical URLs do not help.