Skip to content
This repository has been archived by the owner on Aug 29, 2023. It is now read-only.

csirtgadgets/cif-v5

Repository files navigation

Getting Started

CIF is a model, .. it's a toy. While it's used in a number of large scale environments, it's meant to be a teaching tool. It enables you as an operator to learn from years of operational, institutional knowledge. The default, out of the box configuration is geared 100% towards getting you up and running as quickly as possible (5min or less) so you can LEARN from our experience.

The default, CIF/Docker configuration is NOT meant to be deployed in large scale operations. That's your job. Taking what you've learned and the components we've given you, then creating your own master piece with them. Better yet- because of them.

python 3.6+ is required (eg: Ubuntu 18 LTS or higher)

Docker

Up and Running

$ export CSIRTG_TOKEN=1234  # sign up at csirtg.io
$ export MAXMIND_USERID=1234  # sign up at maxmind.com to leverage geo location data
$ export MAXMIND_LIC=1234

$ git clone https://github.com/csirtgadgets/cif-v5.git
$ mkdir data  # shared data directory for containers
$ cp cif-v5/docker-compose.yml ./
$ mkdir -p data/rules  # copy any local custom rules here
$ # cp my_custom_rule.yml data/rules/
$ chmod 755 data/rules
$ docker-compose pull
$ docker-compose up -d

Status

$ docker-compose logs -f
Attaching to cif-router, cif-httpd, cif-enrichers, csirtg-fm
cif-httpd       | [2020-03-14 15:05:29 +0000] [7] [INFO] Starting gunicorn 19.10.0
cif-httpd       | [2020-03-14 15:05:29 +0000] [7] [INFO] Listening at: http://0.0.0.0:5000 (7)
cif-httpd       | [2020-03-14 15:05:29 +0000] [7] [INFO] Using worker: gevent
cif-httpd       | [2020-03-14 15:05:29 +0000] [10] [INFO] Booting worker with pid: 10
cif-httpd       | [2020-03-14 15:05:29 +0000] [11] [INFO] Booting worker with pid: 11
cif-httpd       | [2020-03-14 15:05:29 +0000] [12] [INFO] Booting worker with pid: 12
cif-httpd       | [2020-03-14 15:05:29 +0000] [13] [INFO] Booting worker with pid: 13
csirtg-fm       | 2020-03-14 15:05:30,107 - INFO - csirtg_fm.cli[316] - random delay is 1.0
csirtg-fm       | 2020-03-14 15:05:30,107 - INFO - csirtg_fm.cli[317] - running every 60 after that  # <--- data will start coming in after this 60 delay
csirtg-fm       | 2020-03-14 15:06:30,184 - INFO - csirtg_fm.cli[85] - starting run...
csirtg-fm       | 2020-03-14 15:06:30,261 - INFO - csirtg_fm.cli[157] - processing: openphish.yml - urls
csirtg-fm       | 2020-03-14 15:06:38,681 - INFO - csirtg_fm.cli[157] - processing: abuse_ch.yml - urlhaus
csirtg-fm       | 2020-03-14 15:07:01,407 - INFO - csirtg_fm[125] - sending: 6
csirtg-fm       | 2020-03-14 15:07:01,598 - INFO - csirtg_fm.cli[157] - processing: abuse_ch.yml - feodo_malware
csirtg-fm       | 2020-03-14 15:07:20,339 - INFO - csirtg_fm.cli[157] - processing: abuse_ch.yml - feodo_botnet
csirtg-fm       | 2020-03-14 15:07:32,947 - INFO - csirtg_fm[125] - sending: 31
csirtg-fm       | 2020-03-14 15:07:33,745 - INFO - csirtg_fm.cli[157] - processing: alexa.yml - top-1000
csirtg-fm       | 2020-03-14 15:07:36,896 - INFO - csirtg_fm[125] - sending: 500
csirtg-fm       | 2020-03-14 15:07:45,937 - INFO - csirtg_fm[125] - sending: 500

Testing

# this requires python3.6 or higher (eg: ubuntu 18 LTS..)

$ pip3 install geoip2 'cifsdk>=5.0b1,<6.0'

$ export CIF_REMOTE='http://localhost:5000'
$ cif -nq example.com
$ cif --itype url --tags malware  # may have to wait 5-10 after starting as data flows in

Where Next?

What's Changed?

  • Less. Stuff.
  • Abstracted a lot of the technical pieces into separate libraries (eg: csirtg-geo, csirtg-peers, etc)
  • Codebase has been significantly simplified
  • NO TOKENS, YOU ARE RESPONSIBLE FOR PROTECTING YOUR NODES!!!
  • Pipelines and better Plugin support for "external enrichment"
  • Docker Compose is the first class citizen, feel free to customize from there
  • Docker containers are now split up, not running as supervisord anymore
  • cif-httpd is significantly simpler to interact with, improved REST doc (openapi)
  • New threat intel rules
  • Python 3.7+ support

Building Locally

$ git clone https://github.com/csirtgadgets/cif-v5.git
$ cd cif-v5/
$ make docker
$ make docker-tag
$ docker-compose up -d
$ docker-compose logs -f

Architecture

                                                              cif-enricher
                                                               ^        +
                                                               |        |
                                                               +        v
csirtg-fm +--> cifsdk  +--------->  cif-httpd +------------> cif-router +-----> cif-store +-----> sqlite
                                                               +
                                           ^                   |        ^
                                           |                   |        |
                                           |                   v        +
                                           |                   cif-hunter
                                           +

                                        cifsdk

Getting Involved

There are many ways to get involved with the project. If you have a new and exciting feature, or even a simple bugfix, simply fork the repo, create some simple test cases, generate a pull-request and give yourself credit!

If you've never worked on a GitHub project, this is a good piece for getting started.

paypal

COPYRIGHT AND LICENSE

Copyright (C) 2020 the CSIRT Gadgets

Free use of this software is granted under the terms of the Mozilla Public License (MPLv2).