Adding basic robits.txt and sitemap.xml files for crawlers

pdweinstein · Feb 3, 2021 · df3a7a2 · df3a7a2
1 parent 965727b
commit df3a7a2
Show file tree

Hide file tree

Showing 2 changed files with 1,205 additions and 0 deletions.
diff --git a/www/robots.txt b/www/robots.txt
@@ -0,0 +1,11 @@
+User-agent: *
+Disallow: /auth/
+Disallow: /js/
+Disallow: /media/
+Disallow: /files/
+
+Disallow: /blog/wp-content/plugins/
+Disallow: /blog/wp-admin/
+Disallow: /blog/readme.html
+
+Sitemap: https://www.weinstein.org/sitemap.xml