AIL Project version 5.6 released with many improvements in the OCR and correlation functions and many updates.

Release Notes for AIL Project - Version 5.6 (2024-05-10)

What's New in v5.6

Changes

• TRON Cryptocurrency Detection & Correlation: Added detection and correlation features for TRON cryptocurrency. Thanks to @pventuzelo (from Fuzzinglabs for this contribution.
• Tag Search Enhancements: Improved the search functionality for OCRs and images by tags and fixed issues with OCR and filtering invalid images.
• Correlation Graph Updates: Added a separator for date first seen and last seen in the correlation graph. Also, fixed the display of OCR object tags.

Fixes

• Language Detector Stability: Fixed an issue where an exception occurred if the LibreTranslate URL was not specified.
• OCR Error Handling: Implemented catching of cv2 errors in OCR processes.
• PGPDump Installer Update: Updated the installer to handle new pgpdump version requirements, including launching autoreconf.
• OCR File Handling: Enhanced error handling with a catch for OSError on MP4 files.
• OCR Language Support: Fixed issues with supported languages in OCR and updated the filter for OCR supported languages.
• Language Detector Improvement: Updated the language detector to return an empty list when no language is detected, enhancing the robustness of language detection.
• OCR Tracker Fixes: Fixed the type of object accepted by the OCR tracker for better accuracy and performance.

Documentation

• New Introduction Presentation: A new introductory presentation is now available. It includes overview diagrams and can be accessed here.

AIL Framework 5.5 Released: New OCR Module for Images, Report Generator for Tracker Module, and Numerous Improvements.

AIL Framework 5.5 Released: New OCR Module for Images, Report Generator for Tracker Module, and Numerous Improvements.

Changes

• Documentation: Improved including overview diagrams.
• OCR Features:
  • Introduced the OCR object and added functionality to retrieve OCRs by daterange with fixes.
  • Added OCR Extractor module and implemented a filter for GIF images in the reprocess tool.
  • Enhanced OCR capabilities with cache addition, correlation between OCR and chat messages, and enabled the OCR extractor by default.
  • Added language detection and translation, a new view for OCR results, and a blueprint for additional languages.
  • Developed a model to get languages, group extracted content by line, process OCR objects, and retrieve all associated images.
• Performance Enhancements: Reduced memory usage.
• Domain-Specific Updates:
  • Fixed the last check in card template domain.
• Tracking and Reporting:
  • Introduced an experimental report generator in the tracker module.
  • Implemented heatmap visualization for the number of user messages.
  • Enhanced message module to show tracker and module matches.
  • Updated trackers UI to allow removal of objects.
• User Account Management:
  • Added display features for chats and subchannels.

Fixes

• OCR and UI Improvements:
  • Fixed an issue where 'None' was copied incorrectly in date fields.
  • Resolved UI correlation issues in the language block.
• Correlation Fixes:
  • Multiple fixes applied to the objects selector to enhance functionality.
• Export and Logging:
  • Fixed export functionality in TheHive component.
• UI Enhancements:
  • Resolved issues with empty matches and added handling for overlapping matches in the UI matches extractor.
• Language Processing:
  • Fixed the minimum probability setting for item languages to improve accuracy.

Overview of AIL features

The new AIL OCR module in action

A sample correlation between chat users

JTAN

Development of the AIL framework is co-funded by the European Union CEF program and CIRCL.

The Action will establish a Joint Threat Analysis Network, an open collaboration group of European computer security incident response teams (CSIRTs) with the focus on collecting, sharing and analysis of technical, operational and strategic threat intelligence. The purpose of this collaboration is to combine unique advantages of different teams to obtain comprehensive situational awareness and actionable information to effectively defend constituencies in each Member State, from critical infrastructure operators targeted by state-sponsored actors to individual citizens affected by cybercrime. The main part of the Action addresses gaps in the Cyber Threat Intelligence (CTI) tooling that is currently used by the national level CSIRTs in Europe. By strengthening individual tools and interconnecting them, the beneficiaries will achieve a new level of common situational awareness and they will benefit from shared knowledge and tooling.

AIL framework 5.4 released with many semantic improvements in chat channels, new Tor vanity domain explorer, Favicon Correlations and various improvements.

The AIL Project version 5.4, released on March 25, 2024, includes a series of changes, fixes, and other updates.

Changes:

• Language Enhancements:

  • Add thread languages stats.
  • Add nb languages stats by chat/subchannel objects.
  • Improve language detection + UI for manual translation.

• New Features and Improvements:

  • Add lexilang and demoji to requirements.
  • Add default and basic cards for user accounts, chat-subchannels, image objects, etc.
  • Performance improvements in the global module.
  • Reprocessing tools for objects by type and improved error outputs in modules.
  • RetroHunt on messages, Direct Correlations UI changes, and heatmap for messages per hour in chats.
  • API enhancements for message retrieval and object investigations.
  • New crawler stats, pie charts, and stacked bar charts.
  • Basic API tests and updates in gitignore.
  • Refactor API blueprint.
  • Favicon Correlations.
  • Update vanity domains, including a vanity domain explorer.
  • Sort onion vanity names and new APIs for titles.
  • Enhanced tracker listings and retro hunt features.

Fixes:

• Chat and Subchannel Issues:

  • Fix subchannel-message correlation and empty chat card issues.
  • Address language detection in empty messages.

• Dashboard and Tracker Corrections:

  • Fix object links in the dashboard and Yara content errors in the tracker.
  • Correct typos.

• Module and Domain Adjustments:

  • Fix SQL Injection Detection object ID and domain-related issues.
  • Fix favicon crawler and export issues.

• Testing and Documentation:

  • Adjust tests and update documentation for chat JSON fields and chat translation.
  • Various fixes in monthly crawled domain stats, favicon export, vanity titles, and chat image importer.

For detailed overview of the all the changes, the changelog provides an extensive overview of all the changes.

JTAN

Development of the AIL framework is co-funded by the European Union CEF program and CIRCL.

The Action will establish a Joint Threat Analysis Network, an open collaboration group of European computer security incident response teams (CSIRTs) with the focus on collecting, sharing and analysis of technical, operational and strategic threat intelligence. The purpose of this collaboration is to combine unique advantages of different teams to obtain comprehensive situational awareness and actionable information to effectively defend constituencies in each Member State, from critical infrastructure operators targeted by state-sponsored actors to individual citizens affected by cybercrime. The main part of the Action addresses gaps in the Cyber Threat Intelligence (CTI) tooling that is currently used by the national level CSIRTs in Europe. By strengthening individual tools and interconnecting them, the beneficiaries will achieve a new level of common situational awareness and they will benefit from shared knowledge and tooling.

AIL framework 5.3 released with chat explorer, Discord and Telegram monitoring, automatic translation, new features and various bugs fixed

Chat Explorer

The Chat Explorer in AIL v5.3 streamlines chat data analysis. It allows users to easily navigate through messages, threads, and subchannels, offering a straightforward approach to monitoring chat activities and extracting insightful information.

Discord and Telegram chats can now be imported using new importers/feeders:

• Discord Feeder: Monitors and imports Discord chat data. Learn more
• Telegram Feeder: Monitors and imports Telegram chat data. Learn more

Future releases will aim to further simplify the import process for other chat platforms. Basic documentation on the JSON format for importing messages is available here.

New Chat Features

AIL v5.3 introduces various new chat features, including:

• Chat Metadata: Name, username, icon, description, participants, etc.
• Chat Subchannels
• Threads in Chats/Subchannels/Messages
• Emoticons
• Images in Messages
• Message Replies
• User Metadata: Account ID, name, icon, additional info, username, etc.

To address language barriers in chat analysis, AIL v5.3 integrates LibreTranslate, an open-source, self-hosted machine translation tool. This feature allows for the translation of chat messages, facilitating easier analysis and comprehension of conversations in different languages without using external services. The information collected via the different chat are processed in the AIL framework and benefit from all the different analysis modules.

AIL v5.3 enhances the correlation of User Accounts, Chats, Images, and Messages, providing a more integrated view of the data and enabling deeper analytical insights.

Improvements

• Significant improvements in the crawling capabilities and integration with Lacus.
• Support for HHHash correlation added to correlate server per HTTP header profile.
• DomainClassifier improved especially on the performance aspect and caching.
• Kvrocks back-end updated to the latest version.

For detailed overview of the all the changes, the changelog provides an extensive overview of all the changes.

AIL images in LXD are available as download and can be used directly in production environment.

JTAN

Development of the AIL framework is co-funded by the European Union CEF program and CIRCL.

The Action will establish a Joint Threat Analysis Network, an open collaboration group of European computer security incident response teams (CSIRTs) with the focus on collecting, sharing and analysis of technical, operational and strategic threat intelligence. The purpose of this collaboration is to combine unique advantages of different teams to obtain comprehensive situational awareness and actionable information to effectively defend constituencies in each Member State, from critical infrastructure operators targeted by state-sponsored actors to individual citizens affected by cybercrime. The main part of the Action addresses gaps in the Cyber Threat Intelligence (CTI) tooling that is currently used by the national level CSIRTs in Europe. By strengthening individual tools and interconnecting them, the beneficiaries will achieve a new level of common situational awareness and they will benefit from shared knowledge and tooling.

AIL framework 5.2 released with new features and various bugs fixed

The latest release, version 5.2 of the AIL project framework, introduces several changes, fixes, and improvements. Some notable changes include the removal of old updates prior to version 5.0, a refactoring of the background updater along with the addition of the v5.2 update, and the introduction of a new etag object. The correlation graph now offers an option to hide objects/nodes and reset functionality using the "H" key. Additionally, an object comment feature has been added to investigations.

Several fixes have been implemented in this release. The environment issues in the updater and background update modules have been resolved. The crawler now includes timeouts for Unknown captures and exception handling for ping_lacus. It also performs an existence check for screenshot sets. The decoding process for downloaded files has been fixed, and the tag functionality for correlation objects has been improved. Fixes have also been made to the updater module, including the removal of old ARDB environment references. Lastly, issues with MISP event JSON export and retro hunt date search and description in the hunter module have been addressed.

In terms of contributors, Thirion Aurélien and fukusuket have made significant contributions to this release, addressing specific issues and providing fixes.

Detailed Change Log

v5.2 (2023-07-12)

Changes

• [update] remove old updates < 5.0. [Terrtia]

• [updater] refactor background updater + add v5.2 update. [Terrtia]

• [crawler har] compress HAR. [Terrtia]

• [correlation] correlation graph, add an option to hide an object/node by pressing H + reset correlation graph. [Terrtia]

• [etag] add new etag object. [Terrtia]

• [investigation] add object comment. [Terrtia]

Fix

• [updater] fix env. [Terrtia]

• [background update] fix logger. [Terrtia]

• [crawler] add timeout to Unknown captures. [Terrtia]

• [crawler] add exception handing for ping_lacus. [fukusuket]

• [crawler] added existence check for screenshot set. [fukusuket]

• [decoded] fix download file. [Terrtia]

• [updater] fix db checker. [Terrtia]

• [correlation tags] fix tag all objects. [Terrtia]

• [correlation card decoded meta] mimetype + size. [Terrtia]

• [correlation card decoded meta] mimetype + size. [Terrtia]

• [updater] remove old ARDB env. [Terrtia]

• [hunter + misp export] fix misp event json export + retro hunt date search and description. [Terrtia]

Other

• Merge pull request #174 from fukusuket/fix-500-erro-when-invalid-lacus-url. [Thirion Aurélien]

  fix: [crawler] add exception handing for ping_lacus

• Merge pull request #176 from fukusuket/fix-500-error-when-crawler-screenshot-setting-off. [Thirion Aurélien]

  fix: [crawler] added existence check for screenshot set

AIL framework 5.1 released with new features and many bugs fixed

Version 5.1 (2023-06-26) includes several changes, fixes, and updates. The changes include fixing gzipped pastes in the pystemon importer, showing a message when the maximum number of nodes is reached in the correlation graph, and adding the ability to auto tag crawled domains. Additionally, new features were added such as pagination for title searches, the ability to search title IDs and contents, and the inclusion of a favicon object.

Several fixes were implemented, including resolving issues with base64 encoding in the pystemon importer, maintaining the same capture UUID for already crawled domains in the crawler, and handling empty queues in the IPAdress module. Other fixes addressed issues with title searches returning empty results, incomplete responses in the crawler, and errors related to user tokens and deletion.

Various improvements were made to different modules and objects, such as the addition of a new cookie-name object along with its correlation, enhancements to importers, improvements to the HOWTO guide, and updates to correlation graphs and statistics. The Phone module was also updated to filter invalid phone numbers and display extracted information in the user interface.

In addition to the changes and fixes, there were updates to the MISP export, domains explorer, daterange object, tracker module, and various other components. The README.md file and CI badge were corrected, the installer was fixed for YARA and pycld3 installations, and tests were updated and replaced.

Lastly, there were some general updates, including merging changes from the old CIRCL/AIL-framework repository (the official repository is ail-project/ail-framework and incorporating a pull request related to email categorization.

Overall, version 5.1 introduced new features, addressed several issues, and included various updates and improvements to different parts of the system.

Detailed Change Log

v5.1 (2023-06-26)

Changes

• [pystemon importer] fix gzipped pastes. [Terrtia]

• [correlation graph] show message if max_nodes reached + fix cookie-name sparkline. [Terrtia]

• [crawler] auto tag crawled domains. [Terrtia]

• [correlation] add an option to remove max number of nodes if max_node == 0. [Terrtia]

• [object cookie-name] add new cookie-name object + correlation. [Terrtia]

• [title search] add pagination. [Terrtia]

• [titles] add title IDs and contents search. [Terrtia]

• [favicon object] add favicon object. [Terrtia]

• [sow item] show item investigations. [Terrtia]

• [kvrocks migration] mv update/v.50. [Terrtia]

• [redis] update minimal version. [Terrtia]

• [doc] add AIL v5.0 + objects + Importers + sync. [Terrtia]

• [correlation] filter blank screenshots. [Terrtia]

• [importers] improve abstract class and logs. [Terrtia]

• [domains explorer] unsafe tag default image. [Terrtia]

• [README.md] update. [Terrtia]

• [HOWTO] improve HOWTO. [Terrtia]

• [correlation graph] update node legend. [Terrtia]

• [correlation graph] select correlation depth. [Terrtia]

• [correlation] correlation graph: filter title objects. [Terrtia]

• [correlation] add direct correlation stats. [Terrtia]

• [new title object] add new title object + correlation on page title. [Terrtia]

• [Phone module] Filter Invalid Phone numbers + UI Show extracted. [Terrtia]

• [importers] add Dir/File Importer. [Terrtia]

Fix

• [pystemon importer] fix base64 encoding. [Terrtia]

• [crawler] same capture uuid if a domain is already crawled. [Terrtia]

• [IPAdress module] empty queue if no IP ranges provided. [Terrtia]

• [retro hunt] fix object tag queue + decoded content. [Terrtia]

• [daterange object] fix objects by date. [Terrtia]

• [title] fix title search empty result. [Terrtia]

• [crawler] fix incomplete response. [Terrtia]

• [user] fix get user token #163. [Terrtia]

• [user] fix user delete #163. [Terrtia]

• [MISP export] fix ail object first/last seen + obj logger. [Terrtia]

• [MISP export] fix empty event. [Terrtia]

• [d4] change enable d4. [Terrtia]

• [kvrocks migration] [Terrtia]

• [objects] fix investigation + ail2ail + screenshot MISP export. [Terrtia]

• [domains explorer] None screeenshot. [Terrtia]

• [show domains] fix down domains. [Terrtia]

• [domains explorer] domain screeenshot. [Terrtia]

• [domains explorer] fix empty screenshots. [Terrtia]

• [correlation] fix tagging nb nodes. [Terrtia]

• [README.md] fix CI badge. [Terrtia]

• [README.md] fix logo. [Terrtia]

• [module.cfg] fix templateModule example. [Terrtia]

• [module extractor] fix tracker extractor. [Terrtia]

• [tracker] fix tracker delete. [Terrtia]

• [tracker] fix webhook. [Terrtia]

• [crawler] fix undefined capture status. [Terrtia]

• [correlation btc info] catch btc txs error. [Terrtia]

• [Phone module] Filter Invalid Phone numbers. [Terrtia]

• [phone] fix phone module. [Terrtia]

• [domain search] fix template domain types filter. [Terrtia]

• [domain search] fix template domain types filter. [Terrtia]

• [MISP auto export] fix module input message. [Terrtia]

• [tests] replace unmaintened nose by nose2. [Terrtia]

• [tests] fix tests. [Terrtia]

• [instaler] fix yara and pycld3 install. [Terrtia]

• [tests] github workflow. [Terrtia]

• [tests] github workflow. [Terrtia]

• [flask] remove old import. [Terrtia]

Other

• Merge github.com:CIRCL/AIL-framework. [Terrtia]

• Merge pull request #592 from shadow2033/patch-2. [Thirion Aurélien]

  Update Categ Mail

• Update Mail. [shadow2033]

  ///English
  added (inbox; zoho)

  ///Russian
  добавлен (inbox; zoho)

AIL framework version 5.0: Major Rewrite, Kvrocks Database, and Lacus Crawler Migration.

AIL v5.0 introduces significant improvements and new features:

• Codebase Rewrite: The codebase has undergone a substantial rewrite resulting in enhanced performance and speed improvements.
• Database Upgrade: The database has been migrated from ARDB to Kvrocks.
• New Correlation Engine: AIL v5.0 introduces a new powerful correlation engine with two new correlation types: CVE and Title.
• Enhanced Logging: The logging system has been improved to provide better troubleshooting capabilities.
• Tagging Support: AIL objects now support tagging, allowing users to categorize and label extracted information for easier analysis and organization.
• Trackers: Improved objects filtering, PGP and decoded tracking added.
• UI Leak Visualization: The user interface has been upgraded to visualize extracted and tracked information.
• New Crawler Lacus: improve crawling capabilities.
• Modular Importers and Exporters: New importers (ZMQ, AIL Feeders) and exporters (MISP, Mail, TheHive) modular design.
  Allow easy creation and customization by extending an abstract class.
• Module Queues: improved the queuing mechanism between detection modules.
• New Object CVE and Title: Extract an correlate CVE IDs and web page titles.

Correlation:

UI Extracted/Tracked content:

AIL Framework version 4.2 released including typo squatting tracker, improved AIL2AIL sync, zerobinz fetcher and many bugs fixes

AIL Framework version 4.2 released including typo squatting tracker, improved AIL2AIL sync, zerobinz fetcher and many bugs fixes v4.2 (2022-06-24)

AIL Framework version 4.2 has been released including:

• A new tracker for tracking potential typo squatted domains. This feature relies on the new ail-typo-squatting library which can be also used outside of AIL framework. This contribution is from @DavidCruciani
• Many improvement and bugs fixed for the AIL2AIL sync. A huge thanks to @aaronkaplan from EU Directorate-General for Informatics (DIGIT) for support and tests during the long debugging sessions.
• A new module for zerobinz to create an immediate crawler request if a zerobinz link appears in an item. The module can be used for other services with ephemeral content. Thanks to @gallypette for the contribution and the improvement ideas.
• A new hosts detection module has been introduced.
• Multiple bugs were fixed.

Detailed Changes

• [Tracker} Tracker_Typo_Squatting. [David Cruciani]

• [v4.2] add v4.2 update. [Terrtia]

• [investigation] fix investigation by user + delete an obj from all investigation. [Terrtia]

• [install vitualenv] remove travis env. [Terrtia]

• [Retro Hunt] add logs. [Terrtia]

• [Retro Hunt] add logs. [Terrtia]

• [Retro Hunt] add logs. [Terrtia]

• [AIL2AIL Sync] update exchange format. [Terrtia]

• [AIL2AIL Sync] update exchange format. [Terrtia]

• [add Hosts module] [Terrtia]

• [sync module] debug. [Terrtia]

• [sync client] debug. [Terrtia]

• [websockets client] bind client ip. [Terrtia]

• [websocket server] add host and port config. [Terrtia]

• [telegram importer] add username correlation. [Terrtia]

• [UI subtype objs] get obj by subtype + name. [Terrtia]

• [misp export] add username. [Terrtia]

Fix

• [typosquatting] remove unused import. [Thirion Aurélien]

• [tracker] clean import. [Thirion Aurélien]

• [tracker term] fix typosquatting key. [Thirion Aurélien]

• [Typo] tracker typo. [David Cruciani]

• [tracker] UI for other than typosquat. [David Cruciani]

• [typo] UI. [David Cruciani]

• [Language] fix cld3 import. [Terrtia]

• [launcher] kill AIL_2_AIL screen. [Terrtia]

• [cld3] enable cld3. [Terrtia]

• [cld3 python3.10] temp disable cld3. [Terrtia]

• [launcher] remove Travis test. [Terrtia]

• [Retro Hunt] item directory. [Terrtia]

• [Retro Hunt] item directory. [Terrtia]

• [Retro Hunt] fix item directory. [Terrtia]

• [AIL exchange mime-type] [Terrtia]

• [Hosts module] module + launcher. [Terrtia]

• [abstract module] exception traceback #145. [Terrtia]

• [ui tag selector] force custom tags. [Terrtia]

• [installer] remove old tor install. [Terrtia]

• [sync module] fix redis tag queue. [Terrtia]

• [sync module] fix tags filter. [Terrtia]

• [sync client] debug. [Terrtia]

• [sync client] debug. [Terrtia]

• [sync module] debug. [Terrtia]

• [websockets client] fix client bind. [Terrtia]

• [websockets] remove size limit. [Terrtia]

• [UI subtype objs] fix form. [Terrtia]

• [misp config] https. [Thirion Aurélien]

Other

• Merge pull request #147 from ail-project/typo. [Thirion Aurélien]

  Integration of the typo-squatting tracker

• Fix; [set tracker] missing function. [Thirion Aurélien]

• Merge branch 'master' into typo. [David Cruciani]

• Add: [tracker] typo-squatting. [David Cruciani]

• Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia]

• Merge pull request #146 from gallypette/master. [Thirion Aurélien]

  add: [modules] zerobinz

• Add: [modules] zerobinz. [huynenjl@gmail.com]

• Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia]

AIL Framework version 4.1 released with new investigation/case handling, improved MISP export and many improvements.

AIL Framework version 4.1 released with new investigation/case handling, improved MISP export and many improvements.

Investigation in AIL

The major new functionality is the investigation handling in AIL. An analyst can now easily create investigation where
any objects from AIL can be added. This helps an analyst to build collection or cases to work on. The integration allows
to export it as a standard MISP event.

Support for Jabber/XMPP

AIL has been extended to support Jabber/XMPP addresses. The source feeder just need to submit the keys such as jabber:to, jabber:from, jabber:ts, jabber:id.
An example feeder is available. The new feature can be used to inject existing leak
or stream from XMPP/Jabber server. As an example, the Conti leak can be easily injected into AIL and show automatically all correlations between users.

Many bugs were fixed.

The complete changelog can be seen below.

v4.1 (2022-03-11)

Changes

• [flask] updated. [Alexandre Dulaunoy]
• [flask] requirements for higher version of flask. [Alexandre Dulaunoy]
• [v4.1] add Investigation with MISP Export + v4.1 update. [Terrtia]
• [Telegram module] refactor module + fix str format. [Terrtia]

Fix

• [Investigation] edit misp event + add misp instance url. [Terrtia]

• [Investigation] fix MISP Export + UI sidebar. [Terrtia]

• [UI inestigations] add items link. [Terrtia]

• [UI inestigations] add objects link. [Terrtia]

• [telegram launcher] [Terrtia]

• [items] abstract class. [Terrtia]

• [Investigation] UI sidebar. [Terrtia]

• [v4.1] fix ardb # tracking DB. [Terrtia]

• [username] user icon. [Thirion Aurélien]

• [Term tracker] fix item date. [Terrtia]

• [Telegram module] fix launcher. [Terrtia]

• [pybgpranking] package install. [Terrtia]

• [popper install] rename popper repository. [Terrtia]
  floating-ui/floating-ui#1425

• [UI] remove update note. [Terrtia]

• [trackers] fix get_all_items_sources. [Terrtia]

• [crawler] fix is_splash_manager_connected #133. [Terrtia]

Other

• Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia]
• Merge pull request #139 from gallypette/jabber-feeder. [Thirion Aurélien]
  add: [username] jabber support
• Add: [username] jabber support. [Jean-Louis Huynen]
• Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia]
• Create SECURITY.md. [Alexandre Dulaunoy]

AIL Framework version 4.0 released with new synchronisation feature, tracker webhook improvement and many bugs fixed.

AIL Framework version 4.0 has been released including a major new feature to allow synchronisation to other AIL instance(s).

The new synchronisation mechanism allow the sync from one AIL instance to another AIL using a standard WebSocket using AIL JSON protocol. The synchronisation allows to filter and sync specific collected items including crawled items or specific tagged items matching defined rules. This feature can be very useful to limit the scope of analysis in specific fields or resource intensive activity. This sync can be also used to share filtered streams with other partners.

A new functionality has been added to trigger a webhook when a tracker is matched in AIL. This is in addition to email notification. The webhook can be used to trigger additional pipelines in AIL.

Additional API endpoints were added such get_item_sources get_check_item_source and get_default_yara_rule_content.

Thanks to the numerous external contributors such as Olivier Sagit and Tony Jabbour. A special thank for the webhook and API developed by Tony Jabbour from CSIRT POST Cyberforce in Luxembourg.

The first version of the synchronisation protocol has been developed in the scope of the JTAN (Joint Threat Analysis Network), a CEF co-funded project (2020-EU-IA-0260).

Many bugs were fixed in this release and many small improvements were added.

Detailed changelog is available on https://www.ail-project.org/ChangeLog.

Changelog

v4.0 (2021-12-01)

Changes

• [sync UI] disable pull. [Terrtia]

• [sync UI] dashboard, show nb of imported items + launch/kill ail servers when a queue is subscribed/unsubscribed. [Terrtia]

• [ail sync UI] restarr/launch/kill sync connections + show sync mode api/pull/push. [Terrtia]

• [ail sync server] add server controller + list connected clients ail_uuid->sync_modes. [Terrtia]

• [ail sync ui] copy to clipboard ail_uuid, ail server key. [Terrtia]

• [ail sync] edit ail_servers/sync_queues + fix logs. [Terrtia]

• [api] rename endpoints. [Terrtia]

• [ail sync] add sync api (ping, version) + UI/client error handler. [Terrtia]

• [doc] GI Badge. [Steve Clement]

• [v4.0 AIL SYNC / AIL 2 AIL] SYNC Manager + fixs + views. [Terrtia]

• [crawler] add auto crawler functions. [Terrtia]

Fix

• [sync client] don't launch client if ail server not linked with a sync queue. [Terrtia]

• [sync server] remove hardcoded host. [Terrtia]

• [sync server] host. [Terrtia]

• [sync client] fix websockets client connect for python >= 3.8. [Terrtia]

• [ail sync] fix refresh_ail_instance_connection. [Terrtia]

• [ail sync] fix refresh_ail_instance_connection. [Terrtia]

• [ail sync] server + client: resend object in queue on ConnectionClosedError. [Terrtia]

• [crawler] add comment. [Terrtia]

• [UI ail sync] fix nav. [Terrtia]

• [UI ail sync] add missing ail icon. [Terrtia]

• [doc] Remove Travis. [Steve Clement]

• [py] Minor python dependency change. [Steve Clement]

• Inherit AbstractModule to prevent stuck queues. [osagit]

  regex compiled only at start, not in the loop
  no duplicate warning string
  comments

• Error message contains http protocol twice. [osagit]

  Error Can't connect to AIL Splash Manager, http://https://localhost:7001/

Other

• Merge pull request #130 from TonyJabbour/master. [Thirion Aurélien]

  New restAPIs

• Merge branch 'dev' into master. [Thirion Aurélien]

• Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia]

• Merge branch 'master' of github.com:ail-project/ail-framework. [Alexandre Dulaunoy]

• Merge pull request #569 from SteveClement/master. [Steve Clement]

• Chg_ [AIL 2 AIL] add backend. [Terrtia]

• Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia]

• Add tracker fixed api function replaced it with internal function. [TonyJabbour]

• Added get_tracker_metadata_api Removed unnecessarily parentheses. [TonyJabbour]

• New API Endpoint: Fixed get_item_content_encoded_text Added get_item_sources Added get_check_item_source Added get_default_yara_rule_content. [TonyJabbour]

• Removed unnecessarily parentheses. [TonyJabbour]

• New API Endpoint: Return Item Content in base64 in non JSON format. [TonyJabbour]

• Merge remote-tracking branch 'origin/master' [TonyJabbour]

• Merge branch 'ail-project:master' into master. [Tony]

• Merge pull request #129 from TonyJabbour/master. [Thirion Aurélien]

  Webhook implementation

• Unnecessarily parenthesis removed. [TonyJabbour]

• Base64 Problem. [TonyJabbour]

• Base64 Problem. [TonyJabbour]

• Type fixed. [TonyJabbour]

• Type fixed. [TonyJabbour]

• Error Fixed. [TonyJabbour]

• Add new API endpoint that return only content encoded in base64. [TonyJabbour]

• Webhook unnecessarily line removed Removed unnecessarily parentheses. [TonyJabbour]

• Type fixed. [TonyJabbour]

• Added a try/catch to handle exceptions Replaced the raise to send message to redis_logger. [TonyJabbour]

• Fixed tracker_metadata. [TonyJabbour]

• Typo Fixed. [TonyJabbour]

• Fixed UI Datatable in showTracker Page Removed "Advanced Search" text from menu misleading button. [TonyJabbour]

• Fixed Webhook integration with Trackers. [TonyJabbour]

• -Fixed "get_tracker_metedata" typo -Typo Fixed. [TonyJabbour]

• -Fixed "description" arg -Typo Fixed. [TonyJabbour]

• -Fixed the 500 error issue when installing new instance of ail when adding new trackers -Fixed missing arguments -Typo Fixed. [TonyJabbour]

• Fix webhook. [TonyJabbour]

• Remove dict from Trackers. [TonyJabbour]

• Add webhook post support in yara and regex trackers. [TonyJabbour]

• Fix get_term_webhook. [TonyJabbour]

• Add some changes for webhook. [TonyJabbour]

• Add initial support for Webhook in Term Trackers. [TonyJabbour]

• Fix spelling issue in Webhook. [TonyJabbour]

• Add more support for Webhook URL. [TonyJabbour]

• Add initial UI support for Webhook in tracker. [TonyJabbour]

• Merge pull request #127 from osagit/patch-3. [Thirion Aurélien]

  fix: inherit AbstractModule to prevent stuck queues

• Merge pull request #126 from osagit/patch-2. [Thirion Aurélien]

  fix: error message contains http protocol twice