How to prevent abusive scans?

[garethjax]

Hello,
nice initiative.

I'm wondering if it can be used with malevolent intents, since somebody has tried to find a vulnerability on my blog (of course it didn't, because i'm using Pelican to publish it).
This has been detected by cloudflare:

{
"action": "challenge",
"clientASNDescription": "HOSTWAY-AS",
"clientAsn": "207566",
"clientCountryName": "RU",
"clientIP": "91.241.19.175",
"clientRequestHTTPHost": "(REDACTED)",
"clientRequestHTTPMethodName": "GET",
"clientRequestHTTPProtocol": "HTTP/1.1",
"clientRequestPath": "/.env.dev.local",
"clientRequestQuery": "",
"datetime": "2021-02-11T04:47:11Z",
"rayName": "61fb6e32cb927b5b",
"ruleId": "country",
"source": "country",
"userAgent": "Nuclei - Open-source project (github.com/projectdiscovery/nuclei)",
"matchIndex": 0,
"metadata": [],
"sampleInterval": 1
}

[ehsandeep]

Hi @garethjax,

Please check the existing discussion on the same topic here and here.

[garethjax]

i've read the topics :( so the answer is unfortunately yes.

And i've checked the ASN data, it's a russian provider... which has a .hk domain... which is expired.

I'm able to defend against those intrusions (thanks mainly to cloudflare), i'm sorry for those who can't.

[garethjax]

I might as well do a mini guide to mitigate the issue using cloudflare for other people:

HOW TO BLOCK ABUSE BY USING CLOUDFLARE

STEP 1: activate Cloudflare.

Cloudflare is a Content Delivery Network which offers quite a few features for a cheap price: the basic tier start a zero USD/EUR/GBP per month.

Activating cloudflare is pretty easy:

• register for an account using an email
• verify the account click on the link which arrives in email
• add your domain(s) to cloudflare
• the domain DNS records are migrated (usually) automatically, but double check just in case.
• change the DNS servers from the original ones, to the 2 servers indicated by cloudflare.
• If there are no issues, the domain should be migrated "into" cloudflare

I'm sure there are plenty of guides if you google "how to activate cloudflare". Please don't make me write that.

STEP 2: Understanding how Cloudflare works.

In layman terms, it's like a chastity belt wrapped around your domain. In order to interact with the site, the visitors has to bypass the chastity belt.

This normally happens without the user realizing it, but there are exceptions, because Cloudflare has been designed to mitigate Distribute Denial of Service attacks.

In order to repels intruders, we don't need a full protection, we just need to customize a feature called FIREWALL.

Just to be clear, we are only wrapping the domain into the chastity belt: the whole webserver/database/files/images remains on your normal hosting server.

STEP 3: You kids get off my lawn!

There is a specific section dedicated to keeping people/bots off the lawn, the Firewall section.

The icon is a Medieval Shield, becase it's all it does: it shields your site. Keep in mind, that in the free version you can only have 5 "rules" but in most cases they are enough.

We will proceed with increasingly aggressive stances:

STANCE 1: Strategic defense

Cloudflare has already implemented a bot prevention mode called "bot fight mode". It detects bots and provides them a "challenge" in order to decide to bypass them or kick them in the balls.

The challenge may be a javascript test (usually bots don't have javascript enabled) or a captcha test (they are not smart enough to solve them)

This mode is totally automatic and you will see the results under the first sub-menu "overview", while the options is under Firewall/tools/bot fight mode.

STANCE 2: American Sniper

The first stance, could not be enough, you totally want to esclude nuclei for some reason, because the basic "bot fight mode" it's not working well enough for you.

Ok, the next tool is able to block visits depending on their user agent, a string which identify the "device" used to access the site. It's called "user agent blocking" but in the vernacular is called "MASTER GATEKEEPER OF NAMED SOULS".
Or maybe i'm the only one calling it that way, i don't know.

You have 10 rules to use in the free tier, so choose wisely. The inner working is quite simple:

• choose a name for the rule
• choose an action (block, challenge, challenge with javascript)
• write the COMPLETE user agent.

so for nuclei you should write exactly: Nuclei - Open-source project (github.com/projectdiscovery/nuclei)

Pay attention, there's a catch: This is like a sniper. It's lethal if you aim well. If you don't get the complete string for the user agent, it doesn't work.

so... there is another stance...

STANCE 3: Dr. Strangelove (How I Learned to Stop Worrying and Love the Bomb)

It's the nuclear option, you have 5 chances to block the world. The whole frigging world.
Or more if you are paying customer.
Let's stare into the abyss, i'm talking about the...

Firewall Rules

• We start by clicking on "create a firewall rule".
• decide on a name, i suggest you to be very detailed in the description, something like "duck all the wp-amin pen test" or "duck you Wish hackers". You get the gist
• Select one (or more) incoming rule:

This is fun, there are a lot of options and we can mix them with and/or conditions.

For now, let's keep it simple, we want to block any user agent which contains "nuclei".

• And finally an action, which could be "block" (back off!), challenge js (back off bot), challenge with captcha ("sorry, could you please remove the shoes before entering?"), Allow ("I assure you, we are open") and Bypass ("but wait, there's more..."). Let's select "BLOCK".

AND THAT'S ALL FOLKS...

...well, that's all for Nuclei, because we have blocked any possible future visit on cloudflare, before it arrives on our server. By the script point of view, it gets a "500 error" or some other kinds of nasty errors.

BUT WAIT THERE'S MORE

• Let's suppose that we only want visitors from the USA and UK.
• We don't want anybody from the rest of the world. Well, we MAY want to give the the benefits of the doubts, but otherwise ... we challenge with captcha. all. the. continents.
• and we exclude USA and UK.

Check this out:

Mind Blown Gif (i'm too lazy to add it, look for it on Google).

One Final note

the nuclear option could really become a double edged sword: don't block unless you're really sure about the rules, because you could stop your site to get visitors or also the search engine bots (google, bing, etc) which need to read your contents. Start by creating challanges in order to avoid issues for your users and check the overview list to identify which kind of visit has been challenged.

if you block visits from the GOOGLE servers, you will be removed from the serch results in a few weeks!

I declare this guide completely CC-0, change and adapt as you prefer.

Hasta la vista, baby!

[geeknik]

This deserves a place in the wiki. 👍🏻