Add a Dynamic Sitemap example that works with the new Dynamic Routes

[leo-petrucci]

Example request

Up until now I've been using a custom Express server with next-routes to handle all my routing. I'm very excited to upgrade and finally move to the Dynamic Routing that was introduced with NextJS 9.

That said, a key part of my site is the Sitemap. At the moment it's very easy for me to have one thanks to my custom server.js and the sitemap package however I am completely lost on how to achieve this with dynamic routing.

I'm not sure if this would be considered out of scope for NextJS, but Sitemaps are essential for websites and its unfortunate that deciding to use a great feature like Dynamic Routing means that I'd have no obvious way of creating one.

An example showing how to use the existing Sitemap NPM Package would be fantastic, but any solution would work.

I've tried my best to get it to work but haven't gotten anywhere so far.

[danielr18]

@creativiii How are you currently using the sitemap package? Would it be possible to continue using it like you do but creating the sitemap files in the public folder with a script?

[leo-petrucci]

@danielr18 I am not generating a static Sitemap on build, which is the suggestion I see a lot when googling, so I don't think this solution would work if that's what you're suggesting.

Here's an amended version of my server.js that's generating my sitemap.

const express = require( 'express' )
const axios = require( 'axios' )
const next    = require( 'next' )
const cacheableResponse = require('cacheable-response')
const sm = require('sitemap')

// Import middleware.
const routes = require( './routes' )

// Setup app.
const app     = next( { dev: 'production' !== process.env.NODE_ENV } )
const handle  = app.getRequestHandler()
const handler = routes.getRequestHandler( app )

const ssrCache = cacheableResponse({
  ttl: 1000 * 60 * 60, // 1hour
  get: async ({ req, res, pagePath, queryParams }) => ({
    data: await app.renderToHTML(req, res, pagePath, queryParams)
  }),
  send: ({ data, res }) => res.send(data)
})

const createSitemap = (res) => {
  let urlRoutes = ['posts', 'pages'];
  let baseUrl = 'https://example.com/'

  let sitemap = sm.createSitemap ({
    hostname: baseUrl,
    cacheTime: 1
  });

  sitemap.add({
    url: baseUrl,
    changefreq: 'daily',
    priority: 1.0
  })

};

app.prepare()
  .then( () => {

    // Create server.
    const server = express();

    server.get('/sitemap.xml', function(req, res) {
      res.header('Content-Type', 'application/xml');
      createSitemap(res);
    });

    // Use our handler for requests.
    server.use( handler );

    // Don't remove. Important for the server to work. Default route.
    server.get( '*', ( req, res ) => {
      ssrCache({ req, res })
    } );

    // Get current port.
    const port = process.env.PORT || 8080;

    // Error check.
    server.listen( port, err => {
      if ( err ) {
        throw err;
      }

      console.log( `> Ready on port ${port}...` );
    } );
  } );

This generates a sitemap with my base url, I then would loop through my desired routes to add other pages. These pages are dynamically generated from the information being pulled from Wordpress so they change fairly often.

[leo-petrucci]

Also since I posted this a user on Spectrum shared their own solution.

I never got around to testing it out since I realised that I also need a way to use cacheable-response which would be impossible with the new dynamic routes. 🤷‍♂️

[dohomi]

I am making use of the new rewrites feature of NextJS v9.1.4

//next.config.js
 experimental: {
      modern: true,
      async rewrites () {
        return [
          {source: '/sitemap.xml', destination: '/api/sitemap'},
        ]
      },
      catchAllRouting: true
    },

Inside of pages/api/sitemap.ts I run the npm module sitemap and build a dynamic sitemap on the fly based on content from a headless API:

import { SitemapStream, streamToPromise } from 'sitemap'
import { IncomingMessage, ServerResponse } from 'http'

export default async function sitemapFunc(req: IncomingMessage, res: ServerResponse) {
  res.setHeader('Content-Type', 'text/xml')
  try {
    const stories =  await fetchContentFromAPI() // call the backend and fetch all stories
    
    const smStream = new SitemapStream({ hostname: 'https://' + req.headers.host })
    for (const story of stories) {
      smStream.write({
        url: story.full_slug,
        lastmod: story.published_at
      })
    }
    smStream.end()
    const sitemap = await streamToPromise(smStream)
      .then(sm => sm.toString())
    res.write(sitemap)
    res.end()
  } catch (e) {
    console.log(e)
    res.statusCode = 500
    res.end()
  }
}

[likesan]

I am making use of the new rewrites feature of NextJS v9.1.4

//next.config.js
 experimental: {
      modern: true,
      async rewrites () {
        return [
          {source: '/sitemap.xml', destination: '/api/sitemap'},
        ]
      },
      catchAllRouting: true
    },

Inside of pages/api/sitemap.ts I run the npm module sitemap and build a dynamic sitemap on the fly based on content from a headless API:

import { SitemapStream, streamToPromise } from 'sitemap'
import { IncomingMessage, ServerResponse } from 'http'

export default async function sitemapFunc(req: IncomingMessage, res: ServerResponse) {
  res.setHeader('Content-Type', 'text/xml')
  try {
    const stories =  await fetchContentFromAPI() // call the backend and fetch all stories
    
    const smStream = new SitemapStream({ hostname: 'https://' + req.headers.host })
    for (const story of stories) {
      smStream.write({
        url: story.full_slug,
        lastmod: story.published_at
      })
    }
    smStream.end()
    const sitemap = await streamToPromise(smStream)
      .then(sm => sm.toString())
    res.write(sitemap)
    res.end()
  } catch (e) {
    console.log(e)
    res.statusCode = 500
    res.end()
  }
}

I wonder why my fetchContentFromAPI gets error Cannot find name 'fetchContentFromAPI'. . Do you know What should I do to find that function? Thank you in advance!

[dohomi]

@sj-log fetchContentFromAPI is a dummy function. You would need to write your own function to call your backend/service to fetch all links you would like to expose as a sitemap

[likesan]

@dohomi Which function did you use for your case? Should I use 'getInitialProps()' at there?

[dohomi]

@sj-log I think you misunderstand: you can do whatever suits your usecase in this function. It is nothing specific to NextJS, it simply returns an array of items and I just gave it the name fetchContentFromAPI

[likesan]

@dohomi Well Thank you for the answer, but I really have no idea how to fetch backend or the service as you mentioned.

[dohomi]

@sj-log what backend do you need to call? I am using a headless CMS (Storyblok). You either can use the native fetch or Axios, whatever suits your need

[MihaiWill]

@dohomi Perfect!!! Thanks a lot! What about:

Warning: You have enabled experimental feature(s).
Experimental features are not covered by semver, and may cause unexpected or broken application behavior. Use them at your own risk.

[dohomi]

@MihaiWill you will see this warning as soon you enable any of the experimental features of NextJS. As it states: the experimental features are not considered stable and used at your own risk. I am running them very stable so far :-)

[likesan]

@dohomi Dear friend, Just I've found another sitemap-generator for nextjs.
so I solved this site mapping issue! because I couldn't use any of Story book or Axios. So sorry if I made you a bit frustrating! just I wanted to grab my goal instantly.

• If there is someone who is struggling to build a sitemap in nextjs, please refer to this one as well. I got this.

dohomi's way to implement is quite simple and good, However, If you still feel 'I don't know what to do' status, please check out the above link. The way I linked, is for a total beginner as easy as I can follow up. Good days.

[leerob]

I've spent some time looking into this. I didn't want to mess with a custom server, or utilize rewrites. Here's where I landed.

// pages/sitemap.xml.js
import React from 'react';

const EXTERNAL_DATA_URL = 'https://jsonplaceholder.typicode.com/posts';

const createSitemap = (posts) => `<?xml version="1.0" encoding="UTF-8"?>
    <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
        ${posts
          .map(({ id }) => {
            return `
                    <url>
                        <loc>${`${EXTERNAL_DATA_URL}/${id}`}</loc>
                    </url>
                `;
          })
          .join('')}
    </urlset>
    `;

class Sitemap extends React.Component {
  static async getServerSideProps({ res }) {
    const request = await fetch(EXTERNAL_DATA_URL);
    const posts = await request.json();

    res.setHeader('Content-Type', 'text/xml');
    res.write(createSitemap(posts));
    res.end();
  }
}

export default Sitemap;

More info here.

[perryraskin]

@leerob took a look at your blog post, really helpful! do you know what I would do to make a dynamic sitemap if my app creates blog post pages on the fly via [slug].tsx? For example, right now https://raskin.me/blog/init(blog) goes to my blog post, but it uses the template [slug].tsx to generate it. Currently the sitemap grabbed the actual "[slug]" name.

[leerob]

@perryraskin I think you would need to slightly modify your script to look at your /posts folder via something like posts/*.md.

const pages = await globby(['posts/*.md', 'pages/**/*.tsx', '!pages/_*{.jsx,.tsx}']);

That will fetch all your Markdown files. Then, you'd need to probably replace posts with blog in the URL you output to your sitemap.

Give that a shot!

[perryraskin]

@leerob nice, that worked great. the sitemap is still showing the [slug] though, should i remove it? i tried doing sitemap.replace but it does not seem to be having any effect

[leerob]

@perryraskin You'll probably want to exclude the blog folder since you're handling it via posts. This should get it working for ya!

const pages = await globby(['posts/*.md', '!pages/blog', 'pages/**/*.tsx', '!pages/_*{.jsx,.tsx}']);

Another idea to consider: utilize getStaticPaths and getStaticProps that will be released soon to generate a static-site (SSG) instead of server-side rendering (SSR) your blog posts. Here's an example.

Once SSG support lands, I'll probably update the blog post to include an example for that. This isn't directly related to your question about the sitemap, but I just figured I'd mention it 😄

#10437 (comment)

[pspeter3]

getStaticPaths and getStaticProps just launched. Can you use them to generate non HTML content (eg RSS / Sitemap)?

[leerob]

@pspeter3 After reading the new docs, it seems like getStaticProps generates HTML/JSON. To create non-HTML content, I believe you'd need to use getServerSideProps.

export async function getServerSideProps({res}) {
    const request = await fetch(EXTERNAL_DATA_URL);
    const posts = await request.json();

    res.setHeader('Content-Type', 'text/xml');
    res.write(createSitemap(posts));
    res.end();
}

[pspeter3]

@leerob Thanks! I'm able to get that to run but I end up with admin/config.yml.html (I'm trying to generate configuration for Netlify CMS)_. Do you how I can not generate the .html suffix?

[leerob]

@pspeter3 The way I've gotten around that myself is using replace for file extensions. I used this when mapping directly over the pages directory looking at MDX files. I believe a similar approach could work for you to filter out .html files after talking to Netlify CMS.

const sitemap = `
    <?xml version="1.0" encoding="UTF-8"?>
    <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
        ${pages
            .map((page) => {
                const path = page
                    .replace('pages', '')
                    .replace('.js', '')
                    .replace('.mdx', '');
                const route = path === '/index' ? '' : path;

                return `
                    <url>
                        <loc>${`https://yoursitehere.com${route}`}</loc>
                    </url>
                `;
            })
            .join('')}
    </urlset>
`;

[pspeter3]

Got it. I was actually commenting on something different. I was trying to use Next to generate a static file that does not have a .html ending. I think that really I want the ability to generate static files using the Next infrastructure and be able to set the permalink. Maybe I should open a new issue?

[leerob]

@pspeter3 Ahh, sorry for the confusion there – my bad. I agree, I would open up a new issue and try and provide a solid example 👍

[pspeter3]

Added #11115 as the new issue

[t-lochhead]

Thank you @leerob for putting this together. The exact code wasn't working for me, but got it to work through some small edits (found here). Sharing in case anyone else ran into the same problem.

// pages/sitemap.xml.js

const EXTERNAL_DATA_URL = "https://jsonplaceholder.typicode.com/posts";

const createSitemap = (posts) => `<?xml version="1.0" encoding="UTF-8"?>
    <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
        ${posts
          .map(({ id }) => {
            return `
                    <url>
                        <loc>${`${EXTERNAL_DATA_URL}/${id}`}</loc>
                    </url>
                `;
          })
          .join("")}
    </urlset>
    `;

// remove component
export async function getServerSideProps({ res }) {
  const request = await fetch(EXTERNAL_DATA_URL);
  const posts = await request.json();

  res.setHeader("Content-Type", "text/xml");
  res.write(createSitemap(posts));
  res.end();
}

// add component here
export default () => null;

[pspeter3]

This is a cool hack! Will it work for exporting too?

[t-lochhead]

@pspeter3 Not sure! First time I tried this.

[xaun]

@leerob what's the go with generating a sitemap like that for thousands, or even millions of posts? Would that be recommended or beneficial?

[leerob]

@xaun If there are millions of posts, there are likely multiple sitemaps stitched together.

[BrunoBernardino]

@xaun have you tried #9051 (comment) ? It should work for your case as well.

[dbertella]

I have tried something very similar to what you are suggesting and succeded to get a sitemap working in my local machine for both dynamic and static routes but in prod it looks like static routes aren't generated in getServerSideProps. I'm using globby package to get the list of the pages but I guess it's not possible to use it in production?

This is my attempt if it can help someone else, dbertella/cascinarampina@efdfea0#diff-2e0db62b38c8e7637d13c40a1a3034cb516d8899478112624a04b6f1f5c0abde
I for tonight added the pages list as a static list for now.

I was thinking will this work in a function at build time and write a static xml file instead of having a ssr page?

[BrunoBernardino]

@dbertella that's what I suggest in #9051 (comment) if you want to try that out.

[dbertella]

Hey @BrunoBernardino yours is a good idea. I'm not super familiar with the whole node environment but if I understand correctly what you are doing the only difference with my script is that you generate the list of pages at build time and that's cool indeed. Regarding the lamda it seems you are using an older version of next, would it work the same if you put the function in the api folder? Also @now/node seems to be deprecated in case will it work the same with @vercel/node? The real question is if you move the lamda in the api folder would you still need to install the additional package?

I will try to generate the list of static pages at build time btw, see if it work fine with my current implementation and let you know in any case.

EDIT. That seems to works fine! Thanks for the tip. I'm running globby in a prebuild script and generate a page.json static file. Other than that the approach with a ssr page seems to work fine I made a PR with the needed changes here

[BrunoBernardino]

Awesome to hear it worked! I'm sure newer versions might need some tweaks, but should still work fine.

[dbertella]

I'm actually thinking at a different approach now and going static by default. The issue I have is that I'm trying to use es6 imports in a node.js script and it doesn't like it very much, but I'd probably go back to it when I have time to try to figure it out this is my next attempt https://github.com/dbertella/cascinarampina/pull/9/files

[lstellway]

Hi all 👋🏼

I am currently exploring the possibility of using the getStaticProps method to build a static page/sitemap.xml.tsx page and leverage the revalidate property for incremental static regeneration.

My site has a lot of dynamic content, and it would be a heavy / expensive job to dynamically generate a sitemap.xml on every request.

From what I am finding, this will probably require a custom server and / or a custom document.

**My first attempt writing a custom document**

I exported a nonHtmlDocument property in my pages/sitemap.xml.tsx page to be referenced. I created a custom pages/_document.tsx with custom render and renderDocument methods to override the base functionality.

pages/sitemap.xml.tsx

import { GetStaticProps } from "next";
...

/**
 * Get static page properties
 *
 * @param GetStaticPropsContext props
 * @return Object
 */
export const getStaticProps: GetStaticProps<SitemapProps> = async () => {
    return {
        revalidate: 60 * 60 * 24, // 1 day
        props: {
            nonHtmlDocument: true,
        },
    };
};

...

pages/_document.tsx

import Document from "next/document";

class MyDocument extends Document<{ nonHtmlDocument?: boolean }> {
    static renderDocument(document, props) {
        return props?.__NEXT_DATA__?.props?.pageProps?.nonHtmlDocument ? (
            <>{props?.html}</>
        ) : (
            super.renderDocument(document, props)
        );
    }

    render() {
        return this.props?.__NEXT_DATA__?.props?.pageProps?.nonHtmlDocument ? (
            <>{this?.props?.html}</>
        ) : (
            <Document {...this.props} />
        );
    }
}

export default MyDocument;

Issues with my attempt:

• I still see the <!DOCTYPE html> rendered
• The XML is still encoded
• I need to change the Content-Type header to application/xml

Does anybody have any ideas?

[lkbr]

Is there a particular reason why you've chosen getStaticProps over an /api route like was suggested here: #9051 (comment)?

We've been happy with an /api/sitemap.ts route + a /sitemap.xml => /api/sitemap.xml rewrite + https://vercel.com/docs/edge-network/caching#stale-while-revalidate.

It feels less of a hack than trying to turn a react page into a sitemap.

[lstellway]

@lkbr
Thank you for your response -
I agree, the API route is a much cleaner solution.

Is there a particular reason why you've chosen getStaticProps over an /api route like was suggested here: #9051 (comment)?

My main concern was with incremental static regeneration, but I was oblivious to how that was implemented with the Cache-Control header.
(I assumed Next.js implemented some sort of additional logic for this)

Knowing this, I see that my CDN does support the stale-while-revalidate header and this approach should work great for me.

Thank you 🙏🏼

[KishokanthJeganathan]

tips from everyone here really helped me. Here is how I did this with contenful CMS. This works with incremental static generation as well

import React from 'react';

const client = require('contentful').createClient({
  space: process.env.CONTENTFUL_SPACE,
  accessToken: process.env.CONTENTFUL_TOKEN,
});

const createSitemap = (
  allBlogPortfolioServiceEntries
) => `<?xml version="1.0" encoding="UTF-8"?>
    ${allBlogPortfolioServiceEntries
      .map((post) => {
        return `
      <url>
          <loc>https://kishokanth.com/${post.fields.category}/${post.fields.slug}</loc>
          <changefreq>daily</changefreq>
          <priority>0.7</priority>
      </url>
  `;
      })

      .join('')}
    </urlset>
    `;

export async function getServerSideProps({ res }) {
  const allEntries = await client.getEntries();

  const allBlogPortfolioServiceEntries = allEntries.items.filter(
    (item) =>
      item.sys.contentType.sys.id === 'servicesOffered' ||
      item.sys.contentType.sys.id === 'portfolioCaseStudies' ||
      item.sys.contentType.sys.id === 'blogPosts'
  );

  res.setHeader('Content-Type', 'text/xml');
  res.write(createSitemap(allBlogPortfolioServiceEntries));
  res.end();
  return {
    props: {}, // will be passed to the page component as props
  };
}

const sitemap = () => {
  return <div></div>;
};

export default sitemap;

[samos123]

I implemented @leerob solutions and it works however it's a bit hacky. Is there any plan for NextJS to have a better sitemap solution? There should be an official recommendation on sitemap generation in the docs because it's quite common. I've moved to NextJS mostly because of SEO. So supporting SEO use cases like sitemap generation are very important to me.

[lstellway]

I was recently browsing through the NextJS examples directory and came across the with-sitemap example.

The approach implements a sitemap generation script that outputs a static sitemap.xml file to the public directory. The script is included in the next.config.js webpack extension (see related documentation).

I plan to implement a similar sitemap generation script in my project and run it via a cron job (vs on build). This way, I will have more control over when the sitemap is generated & do not have to rely on any cache headers (like in my previous approach).

[BrunoBernardino]

@loganstellway that's an interesting idea. If you need dynamic content (that might not exist at build time), you could consider the solution I have in this example repo.

[jupardo]

Tried using functions to generate sitemap, however my generated sitemap is huge so i cannot generate it with lambda. Is there other option of generating sitemap at build / call? Any help would be appreciated!

[BrunoBernardino]

@jupardo have you tried the solution I've suggested above?

[amala-james-cko]

@leerob, I tried this for both dynamic and static pages. but it looks like I have two sitemaps. Is there a way to combine both together?

[leerob]

We are adding official docs on sitemaps, please leave any feedback there! 🙏

P.S. I was going to clean this up and remove outdated solutions, but I'll leave it for posterity. Again, please share any and all feedback on the PR for adding this to the docs - Thank you!