-
Notifications
You must be signed in to change notification settings - Fork 379
HOWTO:UploadAReleaseForPythonInstalls
These are instructions for uploading a new WiredTiger release version to pypi. That site is used by Python developers as a trusted source to download/install the latest versions of packages.
It is possible to build a single downloadable release that works with both Python2 and Python3. To do this, you must use Python3 to build the release. When you configure WiredTiger below, it will run the command python to find out what version it is building for, so this needs to evaluate appropriately. Merely setting PATH to include /opt/mongodbtoolchain/v3/bin/ is not enough, as the python in that directory is version 2.7. A good solution is to use a virtual environment. Specifically, with the MongoDB toolchain, running a command like:
/opt/mongodbtoolchain/v3/bin/virtualenv -p /opt/mongodbtoolchain/v3/bin/python3 ~/venv-py3
creates a ~/venv-py3 directory with all the python tools. Using this virtual environment is done with . ~/venv-py3/bin/activate, which resets the PATH.
As a sanity check, after you've activated your virtual environment, you should be able to do python --version and see the expected version of Python.
Next, you need to be sure that all the normal build tools, plus git and a browser are available.
You need the twine tool for uploading Python packages.
$ which twine
If twine is not installed, pip install twine should do it (it will be installed in your virtual environment if you have one enabled). See below about running twine on OS/X.
To test out the Python package, you need snappy and zlib installed. If they are in non-standard places, you will need to set CMAKE_LIBRARY_PATH on OS/X. You can create a package without these, you just can't test it.
You'll need to get the pypi password for wiredtiger. You should also register your own username/account on testpypi. Testpypi is useful to do a sanity test to make sure you're doing all the steps right until you're comfortable with the process. Just know that the account database on testpypi is wiped periodically, so you may have to reregister at some point if you repeat this procedure.
Edit your ~/.pypirc file with your credentials (see https://docs.python.org/2/distutils/packageindex.html#the-pypirc-file for formatting):
[distutils]
index-servers=
pypi
testpypi
[testpypi]
repository = https://test.pypi.org/legacy/
username = yourname
password = XXXXXX
[pypi]
repository = https://pypi.python.org/pypi
username = wiredtiger
password = XXXXXX
In theory, the entry for [pypi] is not needed (see https://packaging.python.org/guides/migrating-to-pypi-org/#uploading).
You need a git tree (if you have one, you don't need to clone again):
$ git clone https://github.com/wiredtiger/wiredtiger.git wt-top
$ cd wt-top
$ git tag -l
Find the version you're going to build from the output of git tag. In this example, we're using 3.2.0.
$ export WTVER=3.2.0
$ git checkout tags/${WTVER} -b ${WTVER}
$ git log | head
Check the git log output to verify it shows the commit message for the release.
$ cd lang/python
$ python setup_pip.py sdist # creates lang/python/dist/wiredtiger-${WTVER}.tar.gz
Create a clean working environment to install the package, similar to what a new user will experience.
$ sh # fresh shell
$ mkdir ~/pytest # this can be any directory that does not exist
$ cd ~/pytest
$ unset PYTHON_PATH # Don't reference other versions of WiredTiger
Create a virtual environment to install the package. Depending on what version of python your virtualenv was installed with, you'll get an environment that uses that python. See Python Version Support for info on testing with different python versions.
$ virtualenv venv-${WTVER}
$ . venv-${WTVER}/bin/activate
(venv-3.2.0) $ python --version
Python 3.7.0 # make sure you have the version of Python you are expecting
(venv-3.2.0) $ pip install wheel
(venv-3.2.0) $ pip install <release_tree>/dist/wiredtiger-${WTVER}.tar.gz
If the pip install fails, it may be because you don't have snappy or zlib installed. See the Prerequisites step above. That may require you to set LD_LIBRARY_PATH after all, just make sure you don't have _wiredtiger.so or some other WiredTiger library in that path.
For a quick test (still within the virtual environment), run two programs from the source distribution of WiredTiger. They are not installed with the package.
(venv-3.2.0) $ python <wiredtiger_source>/test/py_install/testbase.py
Expect 200 = 200
['WiredTiger 3.2.0: (May 9, 2019)', 3, 2, 0]
testbase success.
(venv-3.2.0) $ python <wiredtiger_source>/test/py_install/testpack.py
testpack success.
(venv-3.2.0) $
Make sure the version of WiredTiger printed by testbase.py is what you expect.
If you should get an error like TypeError: in method 'Cursor__set_key', argument 2 of type 'void *', this probably means that you have a WiredTiger library that was configured and built using a python that was in fact Python2, and you used Python3 when installing and running it. See Python Version Support.
Clean up:
$ deactivate # deactivate the virtual environment
$ cd ~
$ rm -rf pytest # remove the virtual environment directory and test residue
$ exit # exit the 'fresh shell' previously created.
At this point (for as long as we care about supporting Python2 users), it is a good idea to repeat the procedure above in Testing the Release, but instead creating a virtual environment where Python2 is the default.
This is an optional step, allowing you to upload to a test system that is used for download by pip install. It can be useful if you've uncertain about your version of twine, or any other part of the upload procedure.
It's important to get the upload right, because once you've uploaded a version X of wiredtiger in pypi, you may not (ever) upload a different version X. You can delete version X, but you can't upload version X ever after that. You can create a version Y, or X.1, or X-fixed. This restriction applies independently to the testpypi and pypi. Everyone knows that testpypi is a playground, so it's good to make all your mistakes there.
This step uses the testpypi system to test out the upload procedure. Log into the https://testpypi.python.org site with the credentials for your private account as given above when you created the ~/.pypirc file.
Upload the package via:
$ twine upload dist/wiredtiger-${WTVER}.tar.gz -r testpypi
Now test the download. See Testing the Release above except at the pip install step, use:
(venv-3.2.0) $ pip install -i https://test.pypi.org/simple/ wiredtiger
Optionally, retest using Python2.
Now, we're going to upload using pypi instead of testpypi.
Log into the https://pypi.python.org site with the credentials for wiredtiger.
Upload the package via:
$ twine upload dist/wiredtiger-${WTVER}.tar.gz -r pypi
Repeat Testing the Release above except at the pip install step, use:
$ pip install wiredtiger
If a simple twine upload command fails with a complaint about SSL being out of date, you need to install more recent versions of SSL. I (dda) was unable to make this work after doing pip install pyOpenSSL ndg-httpsclient pyasn1. I've generally had better results on OS/X running homebrew Python, which I have installed as python2.7. twine is hardcoded to use the system Python, but that's easy to overcome, substitute python2.7 $(which twine) for twine wherever it appears in the instructions.