Skip to content

Commit

Permalink
block more spam tools, sort for maintainability
Browse files Browse the repository at this point in the history
  • Loading branch information
pushcx committed Feb 15, 2024
1 parent df2ac93 commit 79a8eb9
Showing 1 changed file with 13 additions and 16 deletions.
29 changes: 13 additions & 16 deletions public/robots.txt
Original file line number Diff line number Diff line change
Expand Up @@ -5,29 +5,26 @@ Disallow: /page/
Disallow: /comments/page/

# https://lobste.rs/s/ybowdq/great_gpt_firewall
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: Google-Extended
User-agent: anthropic-ai
User-agent: Bytespider
User-agent: CCBot
Disallow: /
User-agent: ChatGPT-User
User-agent: FacebookBot
User-agent: Google-Extended
User-agent: GPTBot
User-agent: Omgili
User-agent: Omgilibot

# SEO/spam tools
User-agent: BLEXBot
User-agent: SemrushBot
Disallow: /
User-agent: SiteAuditBot
Disallow: /
User-agent: SemrushBot-BA
Disallow: /
User-agent: SemrushBot-COUB
User-agent: SemrushBot-CT
User-agent: SemrushBot-SI
Disallow: /
User-agent: SemrushBot-SWA
Disallow: /
User-agent: SemrushBot-CT
Disallow: /
User-agent: SiteAuditBot
User-agent: SplitSignalBot
Disallow: /
User-agent: SemrushBot-COUB
Disallow: /
User-agent: BLEXBot
Disallow: /


0 comments on commit 79a8eb9

Please sign in to comment.