Enabling Web Crawling rules on Docusaurus(.69) website

[divyabhushan]

Hello all,

I would like to share how I added robots.txt file to my Docusaurus2 v.69 site.

Have you faced the robots.txt file not found error after adding the file to your root directory?

I discovered that the robots.txt file must not be placed in the root directory as suggested in the official document, but in the static folder.
Doing so will copy the file from the static folder to the build directory and added to your Docusaurus website.
I have a written blog on the same and would welcome your thoughts on the same.

Thanks,
Regards,
Divya

[slorber]

thanks

yes, anything in /static will be deployed as part of your site, so you can put whatever you want here

[divyabhushan]

Got it, thank you.