Added ability to redirect incoming requests via ipinfo

[puzzlepeaches]

Description

This pull request adds the ability for the dynamically generated Flask app used by a campaign to redirect incoming requests based on source IP. This will allow an operator to ensure that security solutions and cloud platforms are unable to access phishing landing pages and file downloads. Specifically, the following was added to the code base:

• The ability to redirect incoming requests to the worker based on source IP. This happens in the file app/functions.py
• Adds two new configuration options to specify an ipinfo.io API key (free) and a list of organizations associated with an IP you would like to redirect to an alternate location.
• Added two lists of bad orgs to the templates directory within the repository

Type of change

• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

How Has This Been Tested?

I stood up a new worker with the code base and was able to successfully start campaigns with and without the new configuration options specified.

Notes:

• Existing redlure gitbook will need updates to reflect this change. Can do this myself if you would like after we have thoroughly tested this. We should probably include some notes on getting an ipinfo.io API key.
• The redirect function currently sends bad incoming requests back to the originating IP. This can in certain instances tarpit the crawler which I thought was kind of funny. We could potentially add the option to also send the bad incoming request to the URL specified in the redlure-client. Let me know your thoughts.
• The list of bad orgs included in templates is not all-inclusive and can be changed. The shortlist is going to be more reliable and cause less false positives in the field. The large list can trigger more false positives, but attempts to include a much larger list of cloud providers. Let me know if you know of an alternative list we can source from for this.
• I ran the black code formatter on this sort of accidentally so some formatting changes also occured

[Tw1sm]

Looking to test this sometime this week. Should be a great addition!

[Tw1sm]

Finally got around to playing with this - super cool to see this implemented. Couple of thoughts ranging from small changes to project ideals:

1. ipinfo is added to the Pipfile, but I don't think any ipinfo functions get called by the worker API. The flask app stood up by the worker API calls it (but runs outside of the worker's virtual environment). In short: thinking it would work to remove the ipinfo line from the Pipfile and add pip3 install ipinfo to the install.sh script.
2. I think it'd be beneficial to cache all the IPs that get looked with ipinfo so lookups don't have to done for repeat IPs. Kind of sloppy, but we could easily do this with 2 globals like below (campaign's routes.py). Probably cleaner solutions out there, but the basis of this worked when I quickly tested it.

   url_1 = '/owa'
   url_2 = url_1 + '/2'
   bad = [line.strip() for line in open('/opt/ed/redlure-worker/templates/BAD_ORGS.txt')]
   ipinfo_apikey = '<key>'
   bad_ips = []
   good_ips = []
   
   # snip
   
   @app.route(url_1, methods=['GET', 'POST'])
   def url_1():
       global good_ips
       global bad_ips
       ip = request.environ['REMOTE_ADDR']
       id = request.args.get('id')
   
   
       ipinfo_handler = ipinfo.getHandler(ipinfo_apikey)
       if ip in good_ips:
           pass
       elif ip in bad_ips:
           # return redirect
       else:
           ip_details = ipinfo_handler.getDetails(ip)
           for org in bad:
              if org in ip_details.org.lower():
                  bad_ips.append(ip)
                  return redirect(f'https://{ip}', code=302)
           else:
              good_ips.append(ip)
       if id is not None:
           report_action(id, 'Clicked', request.remote_addr, request.headers.get('User-Agent'))
   
       return render_template('1.html', next_url = base_url + url_for('url_2', id=id))

3. I think if the campaign has a safety_url configured I'd prefer, to see the redirect hit the safety_url. If left unconfigured, I like the current behavior. I think this can get implemented with an extra if in functions.py
4. Now bigger picture - one of the goals with the redlure-workers was to minimize any need to ssh out to them for configuration updates and small changes. With that in mind, do you think the BAD_ORGS config is a "set-it-and-forget-it" config, or do you think there would be any reason to turn it on/off between campaigns or update the list between campaigns? If it is the latter, I'd want to make some updates to the console and client to accommodate, but I'm still kind of 50/50. Curious what your thoughts are on this.

Once we merge this in, I'll care of updating the GitBooks documentation!

[puzzlepeaches]

To address your questions:

• I will definitely strip that out of the Pipfile, not problem.
• Never thought to try; I'll give it a go.
• Will add the if statement there. Was thinking of doing that anyway.
• We can by default use that BAD_ORGS shortlist and allow the user to configure it after the fact as needed. I would ideally like the BAD_ORGS and banned IP lists integrated into the client, but messing with the Flask app is still a little beyond my skillset for the time being. In a perfect world, it may be nice to update that list between campaigns especially if the operator is seeing a large number of requests come in from an unblocked security control. Would be even better if worker gunicorn logs were visible in the client to identify potentially bad IP's. Reach out in Slack and we can work on architecting this a bit more in the future.

Hoping to have the recommended changes implemented by end of week so we can get this merged.

[Tw1sm]

• We can by default use that BAD_ORGS shortlist and allow the user to configure it after the fact as needed. I would ideally like the BAD_ORGS and banned IP lists integrated into the client, but messing with the Flask app is still a little beyond my skillset for the time being. In a perfect world, it may be nice to update that list between campaigns especially if the operator is seeing a large number of requests come in from an unblocked security control. Would be even better if worker gunicorn logs were visible in the client to identify potentially bad IP's. Reach out in Slack and we can work on architecting this a bit more in the future.

First, I love the idea of getting the gunicorn logs to the client, definitely something I can add. As far as how the BAD_ORGS list gets updated, I also agree - I think what I want to do then is merge this into a branch on the worker for the meantime, and work on integrating customizability into the client/console (I can take those parts, I've already got some of this laid out from the "IP Blocklist" stuff I showed you). Once everything is ready, I'll merge each branch into their respective master branches.