pip install: discussion

[bt2901]

This project is still heavily WIP, but I thought you guys should be aware of what is happening. Also, I have some questions/issues I want to discuss.

So, the idea is to use cibuildwheel to make wheels with pre-built dependencies using Travis (then one can upload them to PyPI automatically or manually). This means many things:

• For now, tests are disabled (more precisely, I have written a new travis pipeline from scratch that does not include tests). We need additional configuring to take the best from both worlds
• I think it would be very cool to build/test C++ core and Python API separately. Travis provides a $TRAVIS_COMMIT_MESSAGE environment variable. We can add the following logic: if $TRAVIS_COMMIT_MESSAGE contains substring like [core-changes], then rebuild and run every test; otherwise just grabbing a pre-built libartm and running Python tests is enough.
• Also, it is possible to automate uploading to PyPI (for example, whenever a new version is tagged)

Second, I heavily modified setup.py to allow it to control the execution of cmake and other stuff. Some changes are very Travis-specific (main concern is about paths. Overall it is not scary, just annoying). I'm not sure if it is still possible to install locally (I see two cases here: (1) cloning repository and building the library from source, and (2) making and testing changes in Python API without re-building the library).

Finally, do we want to also distribute bigartm-cli via pip? Most notably, it is needed to build the co-occurrences dictionary.

[bt2901]

Reply originally by @ofrei (lost after my rebasing):

@bt2901 This is a good plan!
Agree, very useful to build&test C++ and Python separately. My overall question is how suitable is CMake for anything that is python-related? Maybe this works, I just don't know it well enough. I would say it's simpler to keep CMake for native code only (perhaps with exception for protobuf dependency where building C++ code depend should also build python protobuf messages). But if you see CMake can simplify making python wheels I'm all for it - no need to change your current strategy.

For bigartm-cli, I think it can be distributed as a separate binary - i.e. doesn't need to be part of pip. Users can continue using an old bigartm-cli to build the co-occurrences dictionary, or you could later include this functionality in python api . Either way, I wouldn't bother about bigartm-cli in this change - it's way more important to make pip install work.

P.S. let me know if you want to discuss over Skype/Zoom - let's try to involve @MelLain and @MichaelSolotky as well.

[bt2901]

Updates:

• I returned tests in .travis.yml. There surely much better ways to organize this, but this will do for now.
• travis now checks commit messages: [TEST] means running tests and [WHL] means building wheel files
• codestyle-checks.sh is disabled (see cpplint.py is outdated #1035)

Installing the developer version and/or building from the source is possible.

If you want to compile the C++ core:

1. run setup.py install from the root of the project
2. probably you want to run export ARTM_SHARED_LIBRARY=<path to compiled lib>

Otherwise, you could install BigARTM with pip and ignore these steps. The pre-built binary is distributed with the PyPi wheel.

If you work with Python interface only, then you probably don't need to re-build the library. To make/test such changes:

1. go to python/ folder
2. run python setup.py install. It is possible that some additional dependencies would be installed.
3. to test your changes: run py.test from this directory
   (currently, most of these tests are failing when ran that way. The likely cause is the inability to find test data)