Skip to content

AIL Framework version 3.1 with new crawling with cookies, Telegram username correlation, new external feeders (e.g. Twitter) and many improvements

Latest
Latest
Compare
Choose a tag to compare
@Terrtia Terrtia released this 12 May 15:12
· 945 commits to master since this release
v3.1
3c49135
This commit was signed with the committer’s verified signature.
Terrtia Thirion Aurélien
GPG key ID: 1E1B1F50D84613D0
Learn about vigilant mode.

AIL framework v3.1 released

telegram

manual crawler

cookiejar

AIL framework v3.1 has been released including many new features such as:

  • Crawling website with a set of cookies collected from a browser (allowing to crawl website password protected or alike)
  • An extraction module to find Telegram users
  • New correlation engine for username starting with Telegram
  • Improve timeout on various modules
  • New JSON importer to be used with new external feeder. A first Twitter feeder and url-feeder is available to feed specific tweets and discovered urls in AIL.

The AIL project moved into a dedicated project to allow new contributions and project within the AIL project scope. If you want to join us and contribute new modules or specific feeders, don't hesitate to contact us.

Complete changelog

  • [update] add update v3.1 + install crawler python requirements by
    default. [Terrtia]
  • [UI correlation] add username correlation card. [Terrtia]
  • [UI correlation] add username correlation graph. [Terrtia]
  • [UI correlation] add username correlation: daterange pages. [Terrtia]
  • [telegram module] add new tag: telegram invite code. [Terrtia]
  • [telegram backend] add username correlation + save invite hash.
    [Terrtia]
  • [importer] add map twitter id - item id, add parents link between
    twitter and url extracted. [Terrtia]
  • [travis install] shallow clone. fetch tags + commit id. [Terrtia]
  • [travis install] shallow clone. fetch tags. [Terrtia]
  • [install] debug travis. [Terrtia]
  • [README] update gitter link. [Terrtia]
  • [import er url_extract] add item twitter parent. [Terrtia]
  • [importer url_extract] fix item id. [Terrtia]
  • [importer] add url_extract importer. [Terrtia]
  • [API json import] take list as imput. [Terrtia]
  • [core import] add AIL JSON import format, API endpoint added (AIL
    feeders) [Terrtia]
  • [update doc] update doc install + logo + fix updater. [Terrtia]
  • [Updater] change default branch origin. [Terrtia]
  • [Updater] change default branch origin. [Terrtia]
  • [update Readme logo + links] [Terrtia]
  • [add new logo] [Terrtia]
  • [AIL logo UI] [Terrtia]
  • [doc] add cookiejar screenshot. [Terrtia]
  • [Crawler] default docker memory usage. [Terrtia]
  • [MISP export] export domain as domain-crawled object. [Terrtia]
  • [domain explorer UI] add shortcut button: misp export + correlation.
    [Terrtia]
  • [crawler] edit cookie and cookiejar + add cookie to cookiejar + fix
    screenshot duplicate. [Terrtia]
  • [Crawler core + UI] crawler lua: handle retry + fix cookie loader and
    selector. [Terrtia]
  • [cookiejar UI] add cookiejar + show all. [Terrtia]
  • [crawler cookies] use cookiejar. [Terrtia]
  • [crawler - cookies] add/show/select cookies. [Terrtia]
  • [crawler] add cookies list by user/global, save cookies from file +
    dict(name, value), TODO: API + handle errors. [Terrtia]
  • [crawler] bypass login: use cookie provided by user and accept cookie
    from server + refractor. [Terrtia]
  • [Splash Crawler] use cookies to bypass login. [Terrtia]
  • [UI] bump jquery to 3.4.1. [Terrtia]
  • [import_dir] fix is_gzip test, use magic number. [Terrtia]
  • [MISP Importer] files: handle missing sha1/sha256 attributes + fix
    Items and Screenshots dir. [Terrtia]
  • [Updater] relauch updater on change (git pull) [Terrtia]

Fixes

  • [Mail module] replace signal by multiprocessing
    (https://docs.python.org/3.4/library/signal.html#execution-of-python-
    signal-handlers) [Terrtia]
  • [Mail] debug signal. [Terrtia]
  • [Mail] debug. [Terrtia]
  • [Mail module] remove test time.wait. [Terrtia]
  • [Mail module] debug signal timeout. [Terrtia]
  • [Mail module] debug signal timeout. [Terrtia]
  • [telegram module] typo. [Terrtia]
  • [telegram module] fix tagging. [Terrtia]
  • [telegram module] check username length. [Terrtia]
  • [telegram module] remove debug. [Terrtia]
  • [Mails] regex timeout. [Terrtia]
  • [Mails] change module output. [Terrtia]
  • [Mails] remove print + test. [Terrtia]
  • [Mails] refactor Mail module. [Terrtia]
  • [urlextract importer] fix parent map + replace set JSON queue by list.
    [Terrtia]
  • [import urlextract parent] fix typo. [Terrtia]
  • [travis install] shallow clone. fetch missing commit id. [Terrtia]
  • [Mails regex timeout] reduce default timeout. [Terrtia]
  • [Mails dns resolver] update timeout exception. [Terrtia]
  • [Mails] typo. [Terrtia]
  • [Mails] import. [Terrtia]
  • [Mails] import. [Terrtia]
  • [Mails] add regex timeout. [Terrtia]
  • [Credential] add regex timeout. [Terrtia]
  • [Credential] add regex timeout. [Terrtia]
  • [urlexport importer] fix item name + redis config. [Terrtia]
  • [urlexport importer] item_id, force str type. [Terrtia]
  • [urlexport importer] add missing import. [Terrtia]
  • [urlextract importer] class name. [Terrtia]
  • [importer] fix typo. [Terrtia]
  • [Global] extend ungzip error catching, catch invalid compressed file.
    [Terrtia]
  • [json import API] remove list input. [Terrtia]
  • [install create default user] default passwd file: add missing new
    line. [Terrtia]
  • [installer] get last git version. [Terrtia]
  • [Readme travis badge] [Terrtia]
  • [Readme travis badge] [Terrtia]
  • [Readme logo size] [Terrtia]
  • [show item UI] return 404. [Terrtia]
  • [Cookiejar UI] fix typo. [Terrtia]
  • [Flask session cookie name] add uuid to cookie name. [Terrtia]
  • [crawler] typo. [Terrtia]
  • [Crawler splash ResponseNeverReceived] add retry. [Terrtia]
  • [crawler] error catcher. [Terrtia]
  • [MISP export UI] fix input: name + value overwrite. [Terrtia]
  • [update thirdparty] update taxonomies. [Terrtia]
  • [crawler] cleanup. [Terrtia]
  • [PgpDump parser] remove header comment (rfc4880) + remove empty lines.
    [Terrtia]
  • [Pgp Dump] remove tool version. [Terrtia]
  • [ZMQ Feeder] performance: replace zmq recv NOBLOCK by Poller.
    [Terrtia]
  • [pgpdump] fix subtype save. [Terrtia]
  • [Updater] force updater update. [Terrtia]
  • [Updater] fix current_tag parser. [Terrtia]
  • [import_dir] remove special characters. [Terrtia]
  • [import_dir] remove dir whitespaces #475. [Terrtia]

Other

  • Merge pull request #492 from sunil3590/master. [Thirion Aurélien]

    crawler_time -> crawler_delta

  • Crawler_time -> crawler_delta. [Sunil D S]

  • Chg [telegram + correlation] new module: telegram (username + login
    code + join_chat) + add simple_correlation backend. [Terrtia]

  • Chg [telegram + correlation] new module: telegram (username + login
    code + join_chat) + add simple_correlation backend. [Terrtia]

  • Merge pull request #487 from CIRCL/crawler_v2. [Thirion Aurélien]

    fix: [crawler] error catcher

  • Merge pull request #486 from CIRCL/crawler_v2. [Thirion Aurélien]

    Crawler v2 - Add cookiejar - use cookie to bypass login form

  • Merge branch 'master' into crawler_v2. [Terrtia]

Assets 2