Changing IntelMQ to Namespace Package

[gethvi]

After some discussion and testing, the following steps are necessary to turn IntelMQ package into a namespace package. This is supported since Python 3.3.

This is for allowing other third party (namespace) packages (such as the discussed intelmq-shadowserver) to tap into intelmq namespace and provide bots.

1. Change the line of IntelMQ setup.py packages=find_packages() to packages=find_namespace_packages()
2. Remove following __init__.py files:
   • intelmq/__init__.py
   • intelmq/bots/__init__.py
   • intelmq/bots/*/__init__.py
   • intelmq/bots/*/*/__init__.py

Basically any subpackage that is possibly sourced from multiple dirs needs to have it's __init__.py removed.

This is how the experimental namespace package intelmq dir tree looks (note the removed __init__.py files, other files are omitted):

intelmq
├── intelmq
│   └── bots
│       └── parsers
│           └── shadowserver
│               ├── _config.py
│               ├── parser_json.py
│               ├── parser.py
│               └── README.md
└── setup.py

This is how the dummy namespace package intelmq-shadowserver dir tree looks like (no __init__.py files either):

intelmq-shadowserver
├── intelmq
│   └── bots
│       └── parsers
│           └── shadowserver
│               └── external_parser.py
└── setup.py

Having both installed it is possible to do:

from intelmq.bots.parsers.shadwoserver import parser, external_parser

This feature is still under consideration, however I wanted to note down my initial experiment and what would need to be done.

[sebix]

As I said in another communication channel: Thank you for investigating this topic, it was on the wishlist for many years. Previously the blocker were deficiencies in Python packaging tools (the namespace packages themselves weren't sufficient, I don't remember what exactly was the culprit).

Do we want to allow one bot to be composed of multiple packages, or do we require that every bot must be at most in one package? Latter would guarantee that no other package puts data in a bot's own directory? It could reduce the complexity. For developers it could reduce the complexity of maintenance, and for the users/admins the complexity of usage and understanding the system.