Customize api nginx to rate-limit bots

[auspicacious]

We should both:

• Set up nginx rate-limiting to automatically slow down crawlers, bots, etc.
• Create a pattern to allow us to explicitly block IPs or IP address ranges

This should be accomplished through .ebextensions.

[sasharevzin]

I would suggest using middleware https://github.com/kickstarter/rack-attack
PS. I understand that it's better to do it via Nginx, so the request won't even ping the app, but rack-attack is way more convenient with all its options, and it's a middleware, so that rails won't be loaded. Currently using for https://streeteasy.com/ and it works great

[auspicacious]

It's interesting code, but I feel like nginx is a more general solution that we can re-use in other places in the future easily. Plus it's pretty easy to do with Elastic Beanstalk.

[auspicacious]

I thought I would restate our priorities a little:

These two will require the nginx rate limiting module:

• Automatically rate limit IP addresses that are making too many requests too quickly
• Allow certain IP addresses to be whitelisted to make more requests without limiting (e.g. https://realtime.safecast.org/)

These two can be accomplished with more basic nginx configuration:

• Make it easy to completely block specific IP addresses
• Make it easy to completely block specific user-agents (this has been useful in the past)