Skip to content

Python

Jump to bottom Edit New page
Jack Gerrits edited this page Jul 19, 2023 · 75 revisions

Support

9.9.0

vowpal-wabbit-next is recommended for new users.

MacOS arm and 3.11 across all platforms are supported by the vowpal-wabbit-next package with binary wheels.

Platform 2.x 3.6 3.7 3.8 3.9 3.10 3.11
Windows unsupported Binary wheel Binary wheel Binary wheel Binary wheel Binary wheel unsupported (use vowpal-wabbit-next)
MacOS unsupported Binary wheel Binary wheel Binary wheel Binary wheel Binary wheel unsupported (use vowpal-wabbit-next)
Linux (manylinux-2010) unsupported Binary wheel Binary wheel (+arm64) Binary wheel (+arm64) Binary wheel (+arm64) Binary wheel (+arm64) unsupported (use vowpal-wabbit-next)

9.2.0 / 9.3.0 / 9.4.0 / 9.5.0 / 9.6.0 / 9.7.0 / 9.8.0

Platform 2.x 3.6 3.7 3.8 3.9 3.10
Windows unsupported Binary wheel Binary wheel Binary wheel Binary wheel Binary wheel
MacOS unsupported Binary wheel Binary wheel Binary wheel (+arm64) Binary wheel (+arm64) Binary wheel (+arm64)
Linux (manylinux-2010) unsupported Binary wheel Binary wheel (+arm64) Binary wheel (+arm64) Binary wheel (+arm64) Binary wheel (+arm64)

Note: All packages are x64 only unless specified

9.1.0

Platform 2.x 3.6 3.7 3.8 3.9 3.10
Windows unsupported Binary wheel Binary wheel Binary wheel Binary wheel Binary wheel
MacOS unsupported Binary wheel Binary wheel Binary wheel (+arm64) Binary wheel (+arm64) Binary wheel*
Linux (manylinux-2010) unsupported Binary wheel (+arm64) Binary wheel (+arm64) Binary wheel (+arm64) Binary wheel (+arm64) Binary wheel (+arm64)

Note: All packages are x64 only unless specified

* Available from conda

9.0.0

Platform 2.x 3.6 3.7 3.8 3.9 3.10
Windows unsupported Binary wheel Binary wheel Binary wheel Binary wheel Binary wheel
MacOS unsupported Binary wheel Binary wheel Binary wheel (+arm64) Binary wheel (+arm64) Unsupported*
Linux (manylinux-2010) unsupported Binary wheel (+arm64) Binary wheel (+arm64) Binary wheel (+arm64) Binary wheel (+arm64) Binary wheel (+arm64)

Note: All packages are x64 only unless specified

* Required dependencies are not yet supported

8.11.0

8.11.0 is the last release where 3.5 remains at this level of support. Additionally, the "may still work" status of 2.7 will be changed to explicitly unsupported in 9.0.0 to allow for supporting Python 3 only features such as type hints.

Platform 2.7 3.5 3.6 3.7 3.8 3.9
Windows Source, unsupported* Source Binary wheel Binary wheel Binary wheel Binary wheel
MacOS Source, unsupported* Source, unsupported^ Binary wheel Binary wheel Binary wheel (+arm64) Binary wheel (+arm64)
Linux (manylinux-2010) Source, unsupported* Binary wheel Binary wheel Binary wheel Binary wheel Binary wheel

Note: All packages are x64 only unless specified

* May still work, but is not supported and will be broken in 9.0.0

^ Due to complications from libc++ vs libstdc++ incompatibility on MacOS, 3.5 is not supported

Installing

Latest released version

Install PyPi package

pip install vowpalwabbit

Bleeding edge (latest commit on master)

  1. Go to workflow run for respective OS
  2. Select a commit and download the artifact for the Python version you require
  3. Extract the zip
  4. pip install vowpalwabbit-*.whl

From source

Windows

  1. Install vcpkg
  2. Install dependencies with vcpkg 
    vcpkg --triplet x64-windows install zlib boost-system boost-program-options boost-test boost-align boost-foreach boost-python boost-math python3
  3. cd <repo_root>
  4. python setup.py --vcpkg-root C:\path\to\vcpkg install - Where C:\path\to\vcpkg is the root directory of where you cloned vcpkg

NOTE: Attempting to install boost-python in vcpkg while multiple python versions are installed in vcpkg will cause errors. Ensure only the relevant python version is installed in the environment.

OSX

  1. Clone repo and Install dependencies
  2. python setup.py install

Linux

The commands listed below will be using apt, replace this for the package manager used by your preferred flavor.

  1. sudo apt update
  2. sudo apt install libboost-dev libboost-program-options-dev libboost-system-dev libboost-math-dev libboost-thread-dev libboost-test-dev libboost-python-dev zlib1g-dev cmake python3 python3-pip
  3. pip3 install vowpalwabbit

Troubleshooting

Some common causes of failure for installation are due to missing or mismatched dependencies when Vowpal Wabbit builds. Make sure you have boost and boost-python installed on your system. Ensure you have all dependencies installed.

Windows

  1. If pip install vowpalwabbit fails, ensure you are using the 64-bit version of Python 3.6 or 3.7 (The default download bitness for Windows is 32-bit)
  2. If you need to build VowpalWabbit from source on Windows, use these following instructions

OSX

There are several known issues regarding the from source VowpalWabbit installation for OSX.

  1. Building and installing VowpalWabbit with Anaconda will crash on OSX 10.14 or later. See this issue for details
    • Uninstall Anaconda before attempting to install VowpalWabbit
  2. CMake can't find Boost
    • Ensure the required Boost libraries are installed
    • If your Boost version is 1.70 or earlier, install using the command python setup.py --enable-boost-cmake install

Linux

For python3 on Ubuntu 16.04 LTS: Ubuntu 16.04 defaults to an old, custom-built version of boost. As such, the boost_python library names do not follow the standard naming convention adopted by official boost releases for the boost_python libraries. You may need to manually create the relevant symlinks in this case.

Example commands for Python 3.5 are as follows. You must change py35 to the right Python version you're using.

$ cd /usr/lib/x86_64-linux-gnu/
$ sudo ln -s libboost_python-py35.so libboost_python3.so
$ sudo ln -s libboost_python-py35.a libboost_python3.a

Debugging Python/C++

The Python bindings can be debugged in a mixed mode fashion by connecting two debuggers. This lets you set breakpoints in both languages.

Requirements

Steps

  1. Edit launch.json to add the required targets
  2. Ensure VW bindings are installed as debug
    1. Run setup.py --debug install -f under the repo root to install the bindings
  3. Open the Python file to debug and run "Launch current Python file" debugger target
  4. While the Python debugger is attached and broken at some breakpoint run "Attach GDB to Python" debugger target and select the Python process
    • A Handy way to get the PID of the Python process is adding this to the script somewhere: 
        import os
  print(os.getpid())
    • If you get an error message like (I think this only happens in WSL): 
      Error getting authority: Error initializing authority: Could not connect: No such file or directory
[1] + Done(127)                  /usr/bin/pkexec "/usr/bin/gdb" --interpreter=mi --tty=${DbgTerm} 0<"/tmp/Microsoft-MIEngine-In-zxx2mqu5.eh4" 1>"/tmp/Microsoft-MIEngine-Out-3ynea04r.784"
    • Then run the following: echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope

Note: you cannot step over the language boundary. To move between you need to set breakpoints on either side and use "continue".

Documentation

Python documentation is automatically built and deployed on each commit to master using this GitHub Action.

Setup

The Python documentation is built using Sphinx and uses Google docstring style.

Install dependencies in build-requirements.txt

Build

To run the Sphinx build:

cd python/docs/
make html

This will output the documentation into a build directory. The entry point will be at python/docs/build/html/index.html

Note: Documentation is built off the currently installed vowpalwabbit package. Therefore you must install your locally edited package to see edits in the documentation.

Resources

launch.json

{
  "name": "Launch current Python file",
  "type": "python",
  "request": "launch",
  "pythonPath": "/usr/bin/python",
  "program": "${file}",
  "cwd": "${workspaceFolder}",
  "console": "integratedTerminal",
  "justMyCode": false
},
{
  "name": "Attach GDB to Python",
  "type": "cppdbg",
  "request": "attach",
  "program": "/usr/bin/python",
  "processId": "${command:pickProcess}",
  "MIMode": "gdb",
  "setupCommands": [
       {
           "description": "Enable pretty-printing for gdb",
           "text": "-enable-pretty-printing",
           "ignoreFailures": true
       }
  ]
}

Links

Release Process

Locations that need to be updated for a release:

  1. PyPI (see instructions below)
  2. Conda (see instructions below)
  3. The Support table on this wiki page. Just edit this page.

PyPI Deployment Process

We upload binary wheels and a source package to PyPI. Binary wheels are automatically produced for every commit to master and uploaded as artifacts in the CI job. The exception is MacOS ARM wheels, which need to be built locally.

  1. View the results of the CI run for the commit being released. Ensure the manylinux.aarch64 builds and tests are passing as these only run after a PR is merged.
  2. Download all artifacts using this tool
    • Generate a token here, it needs workflows scope
    • python run.py --username <gh_username> --token <gh_token> --commit <release_commit>
  3. Build the MacOS arm wheels locally on the dedicated Apple M1 Mac in the office. Use this script.
  4. Run twine check vowpalwabbit-<version>-<binary_tags>.whl on each wheel (can use a wildcard)
  5. Upload all wheels to test PyPI twine upload --repository testpypi vowpalwabbit-<version>-<binary_tags>.whl
    • Test uploaded packages:
    cd /tmp
python -m venv test_vw_package
source test_vw_package/bin/activate
pip install --index-url https://testpypi.python.org/simple/ vowpalwabbit
python -c 'from vowpalwabbit import Workspace'
deactivate
rm -rf test_vw_package
  6. --STOP-- If all looks good up until now then you can proceed
  7. Upload all wheels twine upload vowpalwabbit-<version>-<binary_tags>.whl
  8. Upload source package twine upload vowpalwabbit-<version>.tar.gz

Conda Deployment Process

See conda-forge documentation for more details on maintaining packages. The steps below should be sufficient to update the release.

  1. Create a fork of vowpalwabbit-feedstock
  2. Update the recipe/meta.yaml file:
    1. Change {% set version = "x.xx.x" %} line to desired version number
    2. If changing the version reset build number to 0, otherwise increment the build number
    3. Download the file referred by the URL in the file, run sha256sum on it, and update the hash in meta.yml.
  3. Make a PR back to original vowpalwabbit-feedstock repo
  4. Wait for CI tests to pass and merge
Add a custom footer
Clone this wiki locally