Uploading to PyPI (and conda)

[baileythegreen]

I'd like to ensure I am following proper procedure for uploading to PyPI (specifically for this package, in case there is a specific workflow that is desired), and also have that discussion documented here, with it potentially being moved to some part of the actual documentation once a set procedure is identified.

Steps as I understand them

1. Ensure local branch is up-to-date with the changes to be included.

2. Then run:

python setup.py sdist

which generates a new .tar.gz.¹

3. Publish using:

python -m twine upload --repository-url https://pypi.org/project/pyani/ dist/*

or, perhaps, a slight variation on that.

Questions

Do I need to do anything else prior to uploading, or should I be running

python setup.py sdist bdist_wheel

to generate the files for upload instead? It looks like there is currently a wheel on PyPI: pyani-0.2.11-py3-none-any.whl.

Footnotes

1. Related to the version numbering. PyPI currently shows pyani 0.2.11, and the file I have just generated is called pyani-0.2.11.tar.gz. It seems that it should be pyani-0.2.12.tar.gz, and I believe this discrepancy is perhaps because this is an untracked file(?) that @widdowquinn would have generated for 0.2.11. I believe that running this command just automatically increments the file name based on existing .tar.gz files, but it may also look at checksums or something, in order to determine if any changes have been made. If the former is correct, then I can just run the command again to get the file to upload. However, if not, I'm unsure how to force increment it so that there is a new version number attached to the changes. ↩

[widdowquinn]

There are two stages to this. The first is to make sure that the version is ready for release. A possibly incomplete checklist for this would be:

• Are the changes intended for this version complete?
• Are the changes logged in CHANGES.md
• Are new contributors logged in CONTRIBUTORS.md
• Is the documentation up to date? (e.g. README.md and ReadTheDocs)
• Do all the tests pass?
• Is the version number updated in `pyani/init.py?
• Have you checked that you are ready to release a new version? (i.e. communicate the intent to release to @widdowquinn for comment)

When ready to release:

• Make the release and tag

Create git tag using this command:

$ git tag -a vX.X.X -m "Release version X.X.X"

The mechanisms for (Bio)conda and pypi release differ quite a bit.

pypi

You will need to have in your repository (see link above):

• a suitably-configured setup.py file
• a README.md file (this will be your pypi documentation page)
• a LICENSE file
• If these are not present, add them now

1: Update local packages for distribution

python -m pip install --user --upgrade setuptools wheel
python -m pip install --user --upgrade twine

2: Create distribution packages on your local machine, and check the dist/ directory for the new version files

python setup.py sdist bdist_wheel
ls dist

3: Upload the distribution files to pypi's test server

python -m twine upload --repository-url https://test.pypi.org/legacy/ dist/*

• Check the upload on the test.pypi server
  • https://test.pypi.org/project/PACKAGE/VERSION/

4: Test the upload with a local installation

Create a new test environment

conda create --name test_pypi python=3.8

• Install your package from the test server

python -m pip install --index-url https://test.pypi.org/simple/ --no-deps <PACKAGE>

- Start Python, import the package, and test the version

5: Upload the distribution files to pypi

python -m twine upload dist/*

• Check the upload at pypi
  • https://pypi.org/project///

(Bio)conda

Read this first: bioconda-bot will usually update your package automatically

In most cases, you will not need to act. The bioconda-bot webpage will detect your new version release, and package it up accordingly.

You will receive an email from notifications@github.com in the name of Bioconda Bot, which will provide details of the pull request it has already created on your behalf.

The kind people who maintain bioconda will review the changes and merge when the required standards are met.

Manually updating your package

• See also:
  • Initial local setup of bioconda

1: Fork bioconda-recipes to your GitHub account

Create or update an existing fork of the bioconda-recipes repository on GitHub in your account

• Click here to create the fork

2: Get a current local fork of bioconda-recipes

Clone, or update your local copy of your fork of bioconda-recipes repository.

git clone git@github.com:<YOUR_GITHUB_USERNAME>/bioconda-recipes.git

3: Set the upstream remote

Go into the repository and set the upstream remote to be the official bioconda repository:

cd bioconda-recipes
git remote add upstream https://github.com/bioconda/bioconda-recipes.git

4: Sync local and remote repositories

Make sure the local fork is up to date with the remote repositories

git checkout master          # change to master branch
git pull upstream master     # get changes from official repository
git push origin master       # push changes to your fork

5: Create and checkout a new branch for this package update

Name the branch update_<package> where <package> is the package name.

git checkout -b update_<package>

6: Edit your package recipe to point to the new release

Change to the package recipe directory and edit with your preferred editor.

cd recipes/<package>

• Edit the meta.yaml file to update elements for the new version of the package.
  • Update the version variable:
    • {% raw %}{% set version = "<NEW_VERSION>" %}{% endraw %}
    • Updating the version variable should mean that the source: url: is updated automatically (so long as the versions correspond to our standard version naming scheme)
  • Update the hash:
    • sha256: <HASH>
    • To get the sha256 hash on macOS, use shasum -a 256 <PATH_TO_FILE> on the .tar.gz file downloaded from the GitHub release.
  • If this is not the first meta.yaml file you've edited for this release, increment the build: number: value by one.
  • If necessary, update requirements: host: to add or remove the necessary packages (one per line)

7: Push your modified files to your own fork

Change to the recipes directory and add your modified files

cd ..
git add <package>

• Push to your fork

git push

8: Create a pull request from your fork

Go to your GitHub repo at https://github.com/<YOUR_GITHUB_USERNAME>/bioconda-recipes

• Change to the branch you just created
• Click on the corresponding Pull Request button, and follow the online instructions
• Go to the official repository's Pull Requests page and locate your PR.
  • When this goes green (gets a green tick) the tests have been passed. If there is a red cross, more attention is needed to the update.
  • If the tests have been passed, open your pull request and select a reviewer. If they approve your changes, the PR will be merged, and the new package version will be available.

9: Only after merging: delete your branch

Checkout the master branch, then delete the update_<package> branch locally.

git checkout master
git branch -d update_<package>

- Delete the `update_<package>` branch on the remote repository
- Sync your fork to the remote master, and update your fork on `GitHub`:

git checkout master
git pull upstream master
git push origin master

10: Congratulations, you're done!

[widdowquinn]

If there are any problems with the above, I would be grateful if you would please include the corrections below as well as in the ultimate documentation.

[baileythegreen]

I'll be sure to note down any differences I find.

[baileythegreen]

So far I have come up with these differences/potentially-preferred alternatives:

1.

• Is the version number updated?

Changed to:

• Is the version number updated in `pyani/init.py?

---

2.

Create git tag using this command:

$ git tag -a vX.X.X -m "Release version X.X.X"

---

3.

It's possible that:

$ pip install build
$ python -m build --sdist --wheel --outdir dist .

is preferable to:

python setup.py sdist

as it ensures the local environment does not affect the built package.

[baileythegreen]

From what I can tell, the test server is a different server to the standard PyPI one, so someone needs a separate account that is added to the project there in order to perform an upload.

[widdowquinn]

• Is the version number updated in `pyani/init.py?

If we change how we handle the version number location/versioning approach we would need to update this text. It would be better to leave the precise location open here, and note how we manage versioning elsewhere.

it ensures the local environment does not affect the built package.

I don't understand what you mean. Please be precise about what the current effect is, what the effect of the proposed approach is, and how they differ.

From what I can tell, the test server is a different server to the standard PyPI one

Please see the documentation if you are unsure about this.