CyAnnuaire

Get noticed latest cybersecurity news and threats from Twitter. Easily visualize and query historical data.
Explore the docs »

View Demo · Report Bug · Request Feature

Table of Contents

1. About The Project
   • Built With
2. Roadmap
3. Contributing
4. License
5. Contact

About The Project

CyAnnuaire is a platform to get insights from Twitter. It has a built-in crawler that fetches cybersecurity related tweets. Crawled tweets are classified as Cyber or Non-Cyber with LSTM Neural Networks. A Named Entity Recognition algorithm applied to tweets which are classified as Cyber and some important features such as Affected Company,Affected Product, Affected Version Information, Threat Type captured. All the information that generated by CyAnnuaire are indexed into Elasticsearch, therefore arbitrary elastic queries could be applied into past data and you can create dashboards via Kibana.

{
  "tweet": "A critical vulnerability found at Linux kernel version 5.4.35 that may cause buffer overflow while reading configurations. More details at CVE-2020-1234.",
  "label": "CYBER",
  "fields": {
    "ID": "CVE-2020-1234",
    "PRO": "Linux kernel",
    "VER": "5.4.35",
    "VUL": "buffer overflow"
  }
}

Architecture

Handling tweet data requires a scalable architecture with cutting edge big data tools due to the concerns of big data, it's volume, velocity, variety needs a scalable architecture to live seamlessly.

Here's why we use:

• Elasticsearch: Indexing a huge and unstructured tweet data
• Kafka: Asynchronous message passing among different types of microservices
• Python: Dozens of useful third party libs for data science/engineering.

Built With

• Apache Kafka
• Elastic Stack
• spaCy
• tunga

Screenshots

Roadmap

See the open issues for a list of proposed features (and known issues).

Contributing

Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated.

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

License

Distributed under the MIT License. See LICENSE for more information.

Contact

Burak Tahtacı - @btahtaci - tahtaciburak@gmail.com

Project Link: https://github.com/tahtaciburak/CyAnnuaire