PSA Fedi/Masto admins, useragent allowlist, edit to deny.d conf regarding blocking hidden files also blocking .well-known. Also, .dump is a good contender to add for denied extensions

[jwbjnwolf]

Glad for this existence, and used this put on my proxy server I shield my upstream servers behind, one of which being a Mastodon server. (see important allow list below: #509 (comment))

Not necessarily a bug report or anything as you do disclaim big caution with the deny.d config, but few things as notes to others:

The location block for denying hidden file requests as it is: location ~ /\. { return 444; }, that also blocks .well-known requests (though excluding the acme challenge which has a location block setting the root to /tmp/letsencrypt.

Fediverse instances for federating purposes make strong use of the .well-known directory. Therefore excluding .well-known is a good idea, so change that location block to be like: location ~ /\.(?!well-known).* { return 444; }.

Also, adding dump to the list of denied file extensions such as .sql, .conf etc, is something that should be good too, so if you dump your database as a .dump and have it stored in a web folder accidentally, then you're shielded from that being leaked out.. even though you should obviously NOT let that happen in the first place.

Lastly, image hotlinking, denying that as is can cause your images with your setup to break (and personally I choose to not to deny this. (Years back on shared hosting the admin had to make me a new account because disabling hotlinking both in cpanel and cloudflare completely irreversibly bugged my account lol).

[jwbjnwolf]

Another note to folks hosting fediverse instances such as Mastodon:

A good contender to also deny is the api calls for your instance's instance blocks and known peers. This is exclusively used by blocklist scrapers which still attempt to access the peers and blocks regardless if you have disable these in the software directing attempts to 401s.

location ~* /(api/v1/instance/blocks|api/v1/instance/peers) { return 444; }

[jwbjnwolf]

And if you are like me and want to block all at once outdated instance versions, such as like any version that's 2+ years old, which at time of writing goes are Pleroma v2.3.0 and older versions and Mastodon v3.3.3 and older, though personally at time of writing I've stopped at Mastodon v3.2.2, you can do the following in your bad useragents conf:

"~*(?:\b|)Mastodon/3\.2\.2(?:\b|)"      3;
"~*(?:\b|)Mastodon/3\.2\.1(?:\b|)"      3;
"~*(?:\b|)Mastodon/3\.2\.0(?:\b|)"      3;
"~*(?:\b|)Mastodon/3\.1\.5(?:\b|)"      3;
"~*(?:\b|)Mastodon/3\.1\.4(?:\b|)"      3;
"~*(?:\b|)Mastodon/3\.1\.4(?:\b|)"      3;
"~*(?:\b|)Mastodon/3\.1\.3(?:\b|)"      3;
"~*(?:\b|)Mastodon/3\.1\.2(?:\b|)"      3;
"~*(?:\b|)Mastodon/3\.1\.1(?:\b|)"      3;
"~*(?:\b|)Mastodon/3\.1\.0(?:\b|)"      3;
"~*(?:\b|)Mastodon/3\.0\.2(?:\b|)"      3;
"~*(?:\b|)Mastodon/3\.0\.1(?:\b|)"      3;
"~*(?:\b|)Mastodon/3\.0\.0(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.9\.4(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.9\.3(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.9\.2(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.9\.2(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.9\.1(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.9\.0(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.8\.4(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.8\.3(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.8\.2(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.8\.1(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.8\.0(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.7\.4(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.7\.3(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.7\.2(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.7\.1(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.7\.0(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.6\.5(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.6\.4(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.6\.3(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.6\.2(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.6\.1(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.6\.0(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.5\.2(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.5\.1(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.5\.0(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.4\.5(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.4\.4(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.4\.3(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.4\.2(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.4\.1(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.4\.0(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.3\.3(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.3\.2(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.3\.1(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.3\.0(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.2\.0(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.1\.3(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.1\.2(?:\b|)"      3;
"~*(?:\b|)Mastodon/2\.1\.0(?:\b|)"      3;
"~*(?:\b|)Pleroma\ 2\.3(?:\b|)"         3;
"~*(?:\b|)Pleroma\ 2\.2(?:\b|)"         3;
"~*(?:\b|)Pleroma\ 2\.1(?:\b|)"         3;
"~*(?:\b|)Pleroma\ 2\.0(?:\b|)"         3;
"~*(?:\b|)Pleroma\ 1\.1(?:\b|)"         3;
"~*(?:\b|)Pleroma\ 1\.0(?:\b|)"         3;
"~*(?:\b|)Pleroma\ 0\.9(?:\b|)"         3;

And if you want to also outright block Soapbox instances, and the fba blocklist scraper, you can do

"~*(?:\b|)Ruby\,\ mastodon\ 0\.1\.1(?:\b|)"     3;
"~*(?:\b|)\+soapbox(?:\b|)"                     3;

There is ofc more than just Mastodon and Pleroma, but if you were to do this with Misskey too, it gets kind of complicated considering that certain old pre v12.x versions have been forked, aka before Misskey overhauled it's UI. Mastodon and Pleroma are pretty simple ones though. Though this would also block say a Pleroma 3.5.0 instance if someone forked and changed the useragent to say something like "Pleroma 1.0+3.5.0", as like there's Hometown fork of Mastodon which does like "Mastodon/1.1.1+4.0.2" to mean it's version 1.1.1 of Hometown on version 4.0.2 of Mastodon. Personally it should be the other way around so there's no risk of say Hometown some day reaching v2.1.0 and getting blocked because of the useragent is Mastodon/2.1.0+... that matches with v2.1.0 of Mastodon from December 2017, which indeed there still exists a live to this day instance running on that version.

[jwbjnwolf]

Oh and I just realised that some instances get blocked due to false flags from the global block list, such as .ninja instances for example. So you'll want to add to the useragent allow list "~*(?:\b)ninja(?:\b)" 0; for that.

Anarchy is another, so allowlist that as well:

"~*(?:\b)Anarchy(?:\b)" 0;

Ok had another much deeper careful look and came to all these that can definitely cause false flags for fedi instances so allow listing them is recommended if running a fedi instance:

"~*(?:\b)Alligator(?:\b)"	0;
"~*(?:\b)Anarchie(?:\b)"	0;
"~*(?:\b)Anarchy(?:\b)"		0;
"~*(?:\b)Badass(?:\b)"		0;
"~*(?:\b)Bandit(?:\b)"		0;
"~*(?:\b)Blow(?:\b)"	        0;
"~*(?:\b)Bolt(?:\b)"		0;
"~*(?:\b)Buck(?:\b)"		0;
"~*(?:\b)Buddy(?:\b)"		0;
"~*(?:\b)Cosmos(?:\b)"		0;
"~*(?:\b)Curious(?:\b)"		0;
"~*(?:\b)Demon(?:\b)"		0;
"~*(?:\b)Devil(?:\b)"		0;
"~*(?:\b)Disco(?:\b)"		0;
"~*(?:\b)Dragonfly(?:\b)"	0;
"~*(?:\b)Drip(?:\b)"		0;
"~*(?:\b)Evil(?:\b)"            0;
"~*(?:\b)Fuzz(?:\b)"		0;
"~*(?:\b)Leap(?:\b)"		0;
"~*(?:\b)Harvest(?:\b)"         0;
"~*(?:\b)Magnet(?:\b)"		0;
"~*(?:\b)Nibbler(?:\b)"		0;
"~*(?:\b)Ninja(?:\b)"		0;
"~*(?:\b)Not(?:\b)"		0;
"~*(?:\b)Octopus(?:\b)"         0;
"~*(?:\b)Pump(?:\b)"		0;
"~*(?:\b)Reaper(?:\b)"		0;
"~*(?:\b)Ripper(?:\b)"		0;
"~*(?:\b)Screaming(?:\b)"	0;
"~*(?:\b)Snake(?:\b)"		0;
"~*(?:\b)Snoopy(?:\b)"		0;
"~*(?:\b)Sucker(?:\b)"		0;
"~*(?:\b)TheNomad(?:\b)"	0;
"~*(?:\b)Teleport(?:\b)" 	0;
"~*(?:\b)Titan(?:\b)"		0;
"~*(?:\b)Twice(?:\b)"		0;
"~*(?:\b)Whack(?:\b)"		0;

Definitely tbf wouldn't recommend as a fedi instance to have the auto updating for this. I planned to have it run the updater whenever I update packages, but as a fedi instance definitely want to check over the keywords whenever you update to ensure there's not something new added that happens to false flag a fedi instance.

[jwbjnwolf]

Did a fork including the changes https://github.com/jwbjnwolf/nginx-bad-bot-blocker, removing the hotlinking, adding the .well-known exclusion, and commenting out the user-agent keyword blocking that can/may cause instances to get falsely blocked.

Far easier to manage than allowlisting them, with the rate each update to the global blocklist file adds & removes keywords, that I can see a changes/conflicts for each update.

GitHub

GitHub - jwbjnwolf/nginx-bad-bot-blocker: Nginx Bad Bot Blocker, customised for fedi instances

Nginx Bad Bot Blocker, customised for fedi instances - GitHub - jwbjnwolf/nginx-bad-bot-blocker: Nginx Bad Bot Blocker, customised for fedi instances