chg: [doc] add AIL v5.0 + objects + Importers + sync

CIRCL · Jun 5, 2023 · 683d52d · 683d52d
1 parent f3c3cb5
commit 683d52d
Show file tree

Hide file tree

Showing 5 changed files with 1,538 additions and 1,351 deletions.
diff --git a/HOWTO.md b/HOWTO.md
@@ -1,73 +1,18 @@
 
 # Feeding, adding new features and contributing
 
-## How to feed the AIL framework
-
-Currently, there are three different ways to feed data into AIL:
-
-1. Be a collaborator of CIRCL and ask to access our feed. It will be sent to the static IP you are using for AIL.
+## [Documentation AIL Importers](./doc/README.md#ail-importers)
 
-2. You can setup [pystemon](https://github.com/cvandeplas/pystemon) and use the custom feeder provided by AIL (see below).
+[Documentation AIL Importers](./doc/README.md#ail-importers)
 
-3. You can feed your own data using the [./tool/file_dir_importer.py](./tool/file_dir_importer.py) script.
-
-### Feeding AIL with pystemon
+## How to feed the AIL framework
 
 AIL is an analysis tool, not a collector!
 However, if you want to collect some pastes and feed them to AIL, the procedure is described below. Nevertheless, moderate your queries!
 
-Feed data to AIL:
+1. [AIL Importers](./doc/README.md#ail-importers)
 
-1. Clone the [pystemon's git repository](https://github.com/cvandeplas/pystemon):
-	```
-	git clone https://github.com/cvandeplas/pystemon.git
- 	```
-
-2. Edit configuration file for pystemon ```pystemon/pystemon.yaml```: 
-	- Configure the storage section according to your needs:
-		```
-		storage:  
-			archive:  
-				storage-classname:  FileStorage  
-				save: yes  
-				save-all: yes  
-				dir: "alerts"  
-				dir-all: "archive"  
-				compress: yes
-			
-			redis:  
-				storage-classname:  RedisStorage  
-				save: yes  
-				save-all: yes  
-				server: "localhost"  
-				port: 6379  
-				database: 10  
-				lookup: no
-		```
-	- Adjust the configuration for paste-sites based on your requirements (remember to throttle download and update times).
-
-3. Install python dependencies inside the virtual environment:
-	```shell
-	cd ail-framework/
-	. ./AILENV/bin/activate
-	cd pystemon/
-	pip install -U -r requirements.txt
-	``` 
-4. Edit the configuration file ```ail-framework/configs/core.cfg```:
-	- Modify the "pystemonpath" path accordingly.
-
-5. Launch ail-framework, pystemon and PystemonImporter.py (all within the virtual environment):
-	 - Option 1 (recommended): 
-		``` 
-		 ./ail-framework/bin/LAUNCH.py -l #starts ail-framework
-		 ./ail-framework/bin/LAUNCH.py -f #starts pystemon and the PystemonImporter.py
-		```
-     - Option 2 (may require two terminal windows): 
-        ``` 
-        ./ail-framework/bin/LAUNCH.py -l #starts ail-framework
-        ./pystemon/pystemon.py
-        ./ail-framework/bin/importer/PystemonImporter.py
-        ```
+2. ZMQ: Be a collaborator of CIRCL and ask to access our feed. It will be sent to the static IP you are using for AIL.
 
 ## How to create a new module
 

diff --git a/README.md b/README.md
@@ -34,6 +34,25 @@ AIL is a modular framework to analyse potential information leaks from unstructu
 
 ![Finding webshells with AIL](./doc/screenshots/webshells.gif?raw=true "Finding webshells with AIL")
 
+## AIL V5.0 Version:
+
+AIL v5.0 introduces significant improvements and new features:
+
+- **Codebase Rewrite**: The codebase has undergone a substantial rewrite, 
+resulting in enhanced performance and speed improvements.
+- **Database Upgrade**: The database has been migrated from ARDB to Kvrocks.
+- **New Correlation Engine**: AIL v5.0 introduces a new powerful correlation engine with two new correlation types: CVE and Title.
+- **Enhanced Logging**: The logging system has been improved to provide better troubleshooting capabilities.
+- **Tagging Support**: [AIL objects](./doc/README.md#ail-objects) now support tagging, 
+allowing users to categorize and label extracted information for easier analysis and organization.
+- **Trackers**: Improved objects filtering, PGP and decoded tracking added.
+- **UI Content Visualization**: The user interface has been upgraded to visualize extracted and tracked information.
+- **New Crawler Lacus**: improve crawling capabilities.
+- **Modular Importers and Exporters**: New importers (ZMQ, AIL Feeders) and exporters (MISP, Mail, TheHive) modular design.
+Allow easy creation and customization by extending an abstract class.
+- **Module Queues**: improved the queuing mechanism between detection modules.
+- **New Object CVE and Title**: Extract an correlate CVE IDs and web page titles.
+
 ## Features
 
 - Modular architecture to handle streams of unstructured or structured information