factorcode: add gitweb.cgi to robots.txt

we are getting crawled by: Bingbot: 7249 Applebot: 5228 Googlebot: 1675 Serpstatbot: 1047 ClaudeBot: 580558 Amazonbot: 334936 MJ12bot: 29222 Discordapp Bot: 20
factor · May 12, 2024 · 9b71e8a · 9b71e8a
1 parent 11a494a
commit 9b71e8a
Showing 1 changed file with 3 additions and 0 deletions.
diff --git a/extra/websites/factorcode/robots.txt b/extra/websites/factorcode/robots.txt
@@ -3,3 +3,6 @@ Disallow: /bg_header.jpg
 
 User-Agent: *
 Disallow: factorcode.factor
+
+User-Agent: *
+Disallow: /gitweb.cgi