From 2cf782ec1a95bc2934138e20ab5193982a46d620 Mon Sep 17 00:00:00 2001 From: Matthias Bussonnier Date: Mon, 21 Nov 2016 10:05:41 -0800 Subject: [PATCH 1/9] Start working on the practicalities section. --- .gitignore | 3 ++- _config.yml | 1 + _practicalities/intro.md | 42 +++++++++++++++++++++++++++++++++++++++ practicalities/index.html | 33 ++++++++++++++++++++++++++++++ 4 files changed, 78 insertions(+), 1 deletion(-) create mode 100644 _practicalities/intro.md create mode 100644 practicalities/index.html diff --git a/.gitignore b/.gitignore index f11e635..a406e93 100755 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1,3 @@ _site/ -.DS_Store \ No newline at end of file +.DS_Store +*.swo diff --git a/_config.yml b/_config.yml index fdb01cf..a1e0563 100755 --- a/_config.yml +++ b/_config.yml @@ -35,4 +35,5 @@ colors: collections: - sections +- practicalities diff --git a/_practicalities/intro.md b/_practicalities/intro.md new file mode 100644 index 0000000..239eaa3 --- /dev/null +++ b/_practicalities/intro.md @@ -0,0 +1,42 @@ +--- +--- +# As a user + +If you are already a Python 3 user, you should not encounter a lot of +disruption. Please check that the libraries you use follow best practises not +to break for other users. + +Make sure you have Pip >= 9.0 + +If you are using a custom local package index, for example if you are working +at a company, make sure it implement correctly pep-512 and let pip knows about +the `python_requires` field. + + + + +# Preparing your library + +Things to speak about: + +- Be on recent enough setuptools +- Add a warning at _runtime_ early on master (before switching to Python 3 + only) +- Add an error early at import at runtime with a clear error message, leave the + early import compatible Python 2 for user to not be welcomed with SyntaxError + + + +# Mitigations + +- Leave `setup.py` python 2 compatible and fail early. If you detect Python 2 + raise a clear error message and ask user to make sure they have pip >9.0 (or + migrate to Python 3). You can (try to) conditionally import pip and check for + its version but this might not be the same pip. + +- If you control dependant packages, Make sure to include conditional + dependencies depending on the version of Python. + + + + diff --git a/practicalities/index.html b/practicalities/index.html new file mode 100644 index 0000000..e04b5d1 --- /dev/null +++ b/practicalities/index.html @@ -0,0 +1,33 @@ +--- +--- + + + + + + {{ site.title }} + + + + + + + + {% if site.favicon %}{% endif %} + {% if site.touch_icon %}{% endif %} + + +
+ + + {% for page in site.practicalities %} +
+ +
+ {{ page.content }} +
+
+ {% endfor %} +
+ + From b5e232402e82ea15186902400c66fe9895599016 Mon Sep 17 00:00:00 2001 From: Matthias Bussonnier Date: Mon, 21 Nov 2016 10:47:20 -0800 Subject: [PATCH 2/9] Expand informations --- _practicalities/intro.md | 85 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 81 insertions(+), 4 deletions(-) diff --git a/_practicalities/intro.md b/_practicalities/intro.md index 239eaa3..767a4b5 100644 --- a/_practicalities/intro.md +++ b/_practicalities/intro.md @@ -1,10 +1,19 @@ --- --- + + +Add note here to explain that this does _not_ prevent _nor_ discourage library +author to release 2 version of their software one Python 3 only and the other +python 2. + +This actually made the above easier and less-likely to break. + + # As a user If you are already a Python 3 user, you should not encounter a lot of -disruption. Please check that the libraries you use follow best practises not -to break for other users. +disruption. Please check that the libraries you use follow best practices not +to break for Python 3 users. Make sure you have Pip >= 9.0 @@ -19,11 +28,79 @@ the `python_requires` field. Things to speak about: -- Be on recent enough setuptools +- Be on recent enough setuptools, since [This + PR](https://github.com/pypa/setuptools/pull/631), 24.2.0 (or above, July 20, + 2016, Xavier Fernandez PR.) + + Add the followign to your setup.py + +``` + setup( + ... + python_requires='>=3.3' + ... + ) +``` + +change >= + + + - Add a warning at _runtime_ early on master (before switching to Python 3 only) + +``` +import warnings +import sys +if sys.version_info < (3,): + warnings.warn('You are using master of `Frobulator` with Python 2. Frobulator will soon be Python 3 only. See this issue to know more.', UserWarning) +else: + +``` + - Add an error early at import at runtime with a clear error message, leave the - early import compatible Python 2 for user to not be welcomed with SyntaxError + early import compatible Python 2 for users to not be welcomed with a useless `SyntaxError`. + You are _allowed_ to use multiline strings in error messages. + + +``` +import sys + +if sys.version_info < (3,): + Raise ValueError( + """You are running Frobulator 6.0 on Python 2 + +Unfortunately Frobulator 6.0 and above re not compatible with Python 2 anymore, +and you still ended up with this version installed on your system. That's a +bummer sorry about that it should not have happen. Make sure you have pip >= +9.0 to avoid this kind of issues: + + $ pip install pip --upgrade + +Use the followign to check pip version + +You have various choices: + +- You can still install an older version of Frobulator: + $ pip install frobulator<6.0 + +- Upgrade your system to use Python 3. + +It would be great if you can figure out how this version ended up being +installed, and try to check how to prevent that for future users. + +This this page for more information : url to here for example. +""") + + + +- Make sure your version number match pep440 or you will get surprises during + beta in particular as the `sdist` and `wheel` will appear as being different + versions, in particular sdist (during beta/rc/post) might appear with a + greater version number than wheels... and pip will try to install the sdist + instead of the wheel... The regular expression is trickier than expected: + + `^([1-9]\\d*!)?(0|[1-9]\\d*)(\\.(0|[1-9]\\d*))*((a|b|rc)(0|[1-9]\\d*))?(\\.post(0|[1-9]\\d*))?(\\.dev(0|[1-9]\\d*))?` From c344f3148c70f9ebd17ec4a54ee0dc09d6fb8cbb Mon Sep 17 00:00:00 2001 From: Matthias Bussonnier Date: Mon, 21 Nov 2016 13:38:18 -0800 Subject: [PATCH 3/9] Expand more sections --- _includes/css/base.css | 11 ++++- _practicalities/intro.md | 95 +++++++++++++++++++++++++++++++++------- 2 files changed, 88 insertions(+), 18 deletions(-) diff --git a/_includes/css/base.css b/_includes/css/base.css index bdc9578..a7b244b 100644 --- a/_includes/css/base.css +++ b/_includes/css/base.css @@ -274,7 +274,16 @@ width:90%; } -.highlight pre, .highlight code { display:block; margin:0; padding:0; background: none; overflow:auto; word-wrap: normal; } +.highlight pre, .highlight code { + display:block; + margin:0; + padding:0; + background: + none; + overflow:auto; + word-wrap: normal; + white-space: pre; +} .highlight, .linenodiv { background-image: url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVQIHWPQ1dU1BgABzQC7XXMTYQAAAABJRU5ErkJggg==); diff --git a/_practicalities/intro.md b/_practicalities/intro.md index 767a4b5..5020b38 100644 --- a/_practicalities/intro.md +++ b/_practicalities/intro.md @@ -11,39 +11,75 @@ This actually made the above easier and less-likely to break. # As a user +## Install Pip 9.0 + If you are already a Python 3 user, you should not encounter a lot of -disruption. Please check that the libraries you use follow best practices not -to break for Python 3 users. +disruption. Please still check that the libraries you use follow best practices +not to break for Python 2 users. Python is a community regardless of which +python version you have to a decide to run, making sure that things works make +the community strong. + +Make sure you have Pip >= 9.0, this is especially important if you have Python +2 installations. Having pip 9.0+ will not insure that you install will not +break, but they are more likely not to. Having a version off pip <9.0 can lead +your system to try to upgrade to non-compatible versions of Python packages +even if these are marked as non-compatible. + +Make as many other _users_ as possible to install pip >=9.0, for the +transition, it is the slowest part of the ecosystem to update, and is the only +piece that concern all all installations. + +## Setuptools + +If you are on a system that will not install python wheel and use `setuptools`, +make sure you have setuptools >=24.2.0, or building Python 3 only libraries +might fail, even on Python 2. -Make sure you have Pip >= 9.0 +## Local package index If you are using a custom local package index, for example if you are working -at a company, make sure it implement correctly pep-512 and let pip knows about -the `python_requires` field. +at a company with private packages, make sure it implement correctly pep-503 +and let pip knows about the `python_requires` field. +## The state of PyPI +Note that at the time of this writing the patches to `pypi.python.org` are not +deployed yet but should hopefully be deployed soon. # Preparing your library -Things to speak about: -- Be on recent enough setuptools, since [This - PR](https://github.com/pypa/setuptools/pull/631), 24.2.0 (or above, July 20, - 2016, Xavier Fernandez PR.) - Add the followign to your setup.py +As a library author one of the most important factor in a smooth transition is +planning and communication, letting your user base know in advance that the +transition is happening and what step to take is critical for a transition. + +For your library code here the steps you need to take to ensure that +installation will fail in the least number of case: + +You need to release your new packages version with [setuptools] version 24.2.0 +or above, or use one of the alternate package manager that can set the +[`python_require`] metadata field. Without this, pip 9.0 **will try** to +install non-compatible version of your software on Python 2. This version of +setuptools is recent (July 20, 2016) and this possible thank to the [work of +Xavier Fernandez](https://github.com/pypa/setuptools/pull/631) + +Add the following to your `setup.py` ``` - setup( - ... - python_requires='>=3.3' - ... - ) +setup( + ... + python_requires='>=3.3' + ... +) ``` -change >= +Changes `>=3.3` accordingly depending on what version your library decides to +support. +This will make [PyPI aware](linkto mike's PR on warehouse) that your package is +Python 3 only, and [allow pip](link to pip's PR) to be [made aware of this](link to PyPI PR). - Add a warning at _runtime_ early on master (before switching to Python 3 @@ -102,9 +138,17 @@ This this page for more information : url to here for example. `^([1-9]\\d*!)?(0|[1-9]\\d*)(\\.(0|[1-9]\\d*))*((a|b|rc)(0|[1-9]\\d*))?(\\.post(0|[1-9]\\d*))?(\\.dev(0|[1-9]\\d*))?` +- depend on setuptools greater than 24.3 + -# Mitigations +# Recommende Mitigations + +Of course regardless of all the care you will take for your library to no break +and to install only on python 2, you will likely have cases where it still end +up being installed on incompatible versions of Python. Simply because users +upgrades rarely and only an old version of pip or setuptools is enough to make +the all update process broken. - Leave `setup.py` python 2 compatible and fail early. If you detect Python 2 raise a clear error message and ask user to make sure they have pip >9.0 (or @@ -115,5 +159,22 @@ This this page for more information : url to here for example. dependencies depending on the version of Python. +# Non recommended mitigation + +This is a collection of "mitigation" or "solutions" you will find on the web +and that you will hear about. This is an attempt to acknowlege them, and +explain why they can't work and what are their drawbacks before you attempt to +implement them. + +### Use a meta-package. + + + + +# Why all that ? + +You might wonder why all thi, it's 2016 already, so how come all these issues ? +Python 3 has been out for 8+ years now ! +Well there are many reasons to this, From b30b9664f5e92b13af3bf5ed3c74db5a90adec09 Mon Sep 17 00:00:00 2001 From: Matthias Bussonnier Date: Sat, 26 Nov 2016 13:09:15 -0800 Subject: [PATCH 4/9] Fix some comments --- _practicalities/intro.md | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/_practicalities/intro.md b/_practicalities/intro.md index 5020b38..8a2c3be 100644 --- a/_practicalities/intro.md +++ b/_practicalities/intro.md @@ -25,9 +25,11 @@ break, but they are more likely not to. Having a version off pip <9.0 can lead your system to try to upgrade to non-compatible versions of Python packages even if these are marked as non-compatible. -Make as many other _users_ as possible to install pip >=9.0, for the +Help as many other _users_ as possible to install pip >=9.0, for the transition, it is the slowest part of the ecosystem to update, and is the only -piece that concern all all installations. +piece that concerns all installations: + + pip install --upgrade setuptools pip ## Setuptools @@ -38,8 +40,9 @@ might fail, even on Python 2. ## Local package index If you are using a custom local package index, for example if you are working -at a company with private packages, make sure it implement correctly pep-503 -and let pip knows about the `python_requires` field. +at a company with private packages, make sure it implement correctly +[pep-503](https://www.python.org/dev/peps/pep-0503/) and let pip knows about +the `python_requires` field. ## The state of PyPI @@ -96,7 +99,7 @@ else: - Add an error early at import at runtime with a clear error message, leave the early import compatible Python 2 for users to not be welcomed with a useless `SyntaxError`. - You are _allowed_ to use multiline strings in error messages. + You are _allowed_ to use multi-line strings in error messages. ``` @@ -113,7 +116,7 @@ bummer sorry about that it should not have happen. Make sure you have pip >= $ pip install pip --upgrade -Use the followign to check pip version +Use the following to check pip version You have various choices: @@ -142,7 +145,7 @@ This this page for more information : url to here for example. -# Recommende Mitigations +# Recommended Mitigations Of course regardless of all the care you will take for your library to no break and to install only on python 2, you will likely have cases where it still end @@ -159,10 +162,10 @@ the all update process broken. dependencies depending on the version of Python. -# Non recommended mitigation +# Alternative mitigation This is a collection of "mitigation" or "solutions" you will find on the web -and that you will hear about. This is an attempt to acknowlege them, and +and that you will hear about. This is an attempt to acknowledge them, and explain why they can't work and what are their drawbacks before you attempt to implement them. @@ -170,10 +173,9 @@ implement them. - # Why all that ? -You might wonder why all thi, it's 2016 already, so how come all these issues ? +You might wonder why all this, it's 2016 already, so how come all these issues ? Python 3 has been out for 8+ years now ! Well there are many reasons to this, From 3bf015606219889f4447100ac4f6734c287b4277 Mon Sep 17 00:00:00 2001 From: Matthias Bussonnier Date: Sat, 26 Nov 2016 14:27:46 -0800 Subject: [PATCH 5/9] More infos --- _practicalities/intro.md | 138 ++++++++++++++++++++++++++++++++------- 1 file changed, 114 insertions(+), 24 deletions(-) diff --git a/_practicalities/intro.md b/_practicalities/intro.md index 8a2c3be..a30a0e1 100644 --- a/_practicalities/intro.md +++ b/_practicalities/intro.md @@ -1,7 +1,6 @@ --- --- - Add note here to explain that this does _not_ prevent _nor_ discourage library author to release 2 version of their software one Python 3 only and the other python 2. @@ -9,6 +8,14 @@ python 2. This actually made the above easier and less-likely to break. +Too long, did not read: + + - Help and encourage users to install **pip 9.0+** + - Help and encourage users to install **setuptools 24.3+** + - Use **`setup(..., python_requires='>=3.3')`** new option. + - **Fail** early at **install time** if on Python 2. + + # As a user ## Install Pip 9.0 @@ -21,27 +28,49 @@ the community strong. Make sure you have Pip >= 9.0, this is especially important if you have Python 2 installations. Having pip 9.0+ will not insure that you install will not -break, but they are more likely not to. Having a version off pip <9.0 can lead +break, but they are more likely not to. Having a version off pip < 9.0 can lead your system to try to upgrade to non-compatible versions of Python packages even if these are marked as non-compatible. Help as many other _users_ as possible to install pip >=9.0, for the transition, it is the slowest part of the ecosystem to update, and is the only -piece that concerns all installations: +piece that concerns all installations. + +The simplest way to make sure all is up to date is to run the following for +each installation of Python: pip install --upgrade setuptools pip +This will install the latest version of pip and setuptools. + +You can issue the following to see the version of pip: + + pip --version + + + ## Setuptools If you are on a system that will not install python wheel and use `setuptools`, make sure you have setuptools >=24.2.0, or building Python 3 only libraries -might fail, even on Python 2. +might fail. In particular if authors have taken time to mark their library as +Python 3 only, the `python_requires` argument to `setup()` will not be +recognize and installation will fail. + +Use the following to check setuptools version : + + python -c 'import setuptools; print(setuptools.__version__) + +Again make sure tu upgrade pip and setuptools to make sure you have an up to +date system: + + pip install --upgrade setuptools pip ## Local package index If you are using a custom local package index, for example if you are working at a company with private packages, make sure it implement correctly -[pep-503](https://www.python.org/dev/peps/pep-0503/) and let pip knows about +[pep-503](https://www.gg python.org/dev/peps/pep-0503/) and let pip knows about the `python_requires` field. ## The state of PyPI @@ -53,7 +82,6 @@ deployed yet but should hopefully be deployed soon. # Preparing your library - As a library author one of the most important factor in a smooth transition is planning and communication, letting your user base know in advance that the transition is happening and what step to take is critical for a transition. @@ -61,12 +89,14 @@ transition is happening and what step to take is critical for a transition. For your library code here the steps you need to take to ensure that installation will fail in the least number of case: -You need to release your new packages version with [setuptools] version 24.2.0 -or above, or use one of the alternate package manager that can set the -[`python_require`] metadata field. Without this, pip 9.0 **will try** to -install non-compatible version of your software on Python 2. This version of -setuptools is recent (July 20, 2016) and this possible thank to the [work of -Xavier Fernandez](https://github.com/pypa/setuptools/pull/631) +You need to release your new packages version with +[setuptools](https://pypi.python.org/pypi/setuptools) version 24.2.0 or above. +You can also use one of the alternate package manager that can set the +[Requires-Python](https://www.python.org/dev/peps/pep-0345/#requires-python) +metadata field. Without this, pip 9.0 **will try** to install non-compatible +version of your software on Python 2. This version of setuptools is recent +(July 20, 2016) and this possible thank to the [work of Xavier +Fernandez](https://github.com/pypa/setuptools/pull/631) Add the following to your `setup.py` @@ -81,8 +111,10 @@ setup( Changes `>=3.3` accordingly depending on what version your library decides to support. -This will make [PyPI aware](linkto mike's PR on warehouse) that your package is -Python 3 only, and [allow pip](link to pip's PR) to be [made aware of this](link to PyPI PR). +This will make [PyPI aware](https://github.com/pypa/warehouse/pull/1448) that +your package is Python 3 only, and [allow +pip](https://github.com/pypa/pip/pull/3877) to be [made aware of +this](https://github.com/pypa/pypi-legacy/pull/506). - Add a warning at _runtime_ early on master (before switching to Python 3 @@ -131,17 +163,26 @@ installed, and try to check how to prevent that for future users. This this page for more information : url to here for example. """) +``` + -- Make sure your version number match pep440 or you will get surprises during - beta in particular as the `sdist` and `wheel` will appear as being different - versions, in particular sdist (during beta/rc/post) might appear with a - greater version number than wheels... and pip will try to install the sdist - instead of the wheel... The regular expression is trickier than expected: +Make sure your version number match pep440 or you will get surprises during +beta in particular as the `sdist` and `wheel` will appear as being different +versions, in particular sdist (during beta/rc/post) can appear with a greater +version number than wheels. Pip thus try to install the sdist instead of the +wheel, which have more chance of failing, in particular with pre 24.2 versions +of setuptools. + +The regular expression to check for validity of pep440 can be find below: `^([1-9]\\d*!)?(0|[1-9]\\d*)(\\.(0|[1-9]\\d*))*((a|b|rc)(0|[1-9]\\d*))?(\\.post(0|[1-9]\\d*))?(\\.dev(0|[1-9]\\d*))?` -- depend on setuptools greater than 24.3 + + +You can mark your library as dependent on setuptools greater than 24.3 starting +now, this will insure that during the next upgrade (when the packages drop +python 2 support) will have the right version of setuptools. @@ -156,11 +197,17 @@ the all update process broken. - Leave `setup.py` python 2 compatible and fail early. If you detect Python 2 raise a clear error message and ask user to make sure they have pip >9.0 (or migrate to Python 3). You can (try to) conditionally import pip and check for - its version but this might not be the same pip. + its version but this might not be the same pip. Failing early is important to + make sure the Python installation does not install and incompatible version. + Otherwise user code can fail at runtime arbitrary later in the future, which + can be a difficult to debug and fix. - If you control dependant packages, Make sure to include conditional dependencies depending on the version of Python. +- Regardless of whether the installation step fails on Python 2, implement a + similar check in the top level import of your package. + # Alternative mitigation @@ -171,12 +218,55 @@ implement them. ### Use a meta-package. +It is possible to release a meta-package that has _virtually_ no code and rely +on conditional dependency to install its actual core code on the user system. +For example, Frob-6.0 could be a meta-package which depends on +Frob-real-py2 on Python <3.0, and Frob-real-py3 on Python >= 3.4. While +this approach is _doable_ this can make import confusing. + +Moreover, upgrading your package may need the user to explicitly tell pip to +upgrade dependencies as `pip install frob` will only upgrade the meta-package. + +### Multiple Sdist. + +Pip (used to) support a "feature" where a sdist ending in `-pyX.Y.tar.gz` would +only be seen as compatible on Python X.Y, thus it used to be possible to +publish multiple sdist of a package targeting various python version. + +Though it is not possible anymore to upload multiple sdist on PyPI. This +solution is thus not possible. + +### Wheel only ? + +Break downstream packages. + + # Why all that ? -You might wonder why all this, it's 2016 already, so how come all these issues ? -Python 3 has been out for 8+ years now ! +You might wonder why all this, it's 2016 already, so how come all these +issues ? Python 3 has been out for 8+ years now ! + +Well there are many reasons to this, first of all, this issue mostly affect +libraries that are currently python 2 and Python 3 compatible at the same time. +Many libraries have transitioned from Python 2-only to Python 2 + 3. And the +issue of transitioning to Python 3 only is relatively recent. Technically it +can also apply to libraries that are only stopping support for 2.6, or even are +already Python 3 only, but are starting to stop support for earlier versions of +Python. For example a library releasing a Python 3.4+ only version. + +Python 3.3 was release end of 2012, and was the first version to support +(again) `u` as a prefix for Unicode string. It was one of the first minor +version of Python 3 that saw a majority of single-source project working both +on Python 2 and Python 3. These are the Project that will likely be affected by +this issue. + +The introduction of Python 3 was chaotic, there are still strong argument both +in Python 2 and Python 3 camps. In the one suffering the most from this are +users. Starting with the fact that inevitably some libraries will stop support +for Python 2 and release Python 3 only library. And that inevitably some system +will will not be upgraded to Python 3 how can we _ensure_ that users get the +_least_ breakage as possible ? And what are the best practices to follow. -Well there are many reasons to this, From 958ec43c8375eb312c366a7d17a53aa70f6d0ad7 Mon Sep 17 00:00:00 2001 From: Matthias Bussonnier Date: Fri, 14 Apr 2017 14:07:00 -0700 Subject: [PATCH 6/9] Some refinments --- _practicalities/intro.md | 226 ++++++++++++++++++++++++++------------- 1 file changed, 151 insertions(+), 75 deletions(-) diff --git a/_practicalities/intro.md b/_practicalities/intro.md index a30a0e1..5a0b099 100644 --- a/_practicalities/intro.md +++ b/_practicalities/intro.md @@ -3,81 +3,129 @@ Add note here to explain that this does _not_ prevent _nor_ discourage library author to release 2 version of their software one Python 3 only and the other -python 2. +python 2. -This actually made the above easier and less-likely to break. +This page gather information and links to resources allowing to release a +library that stop supporting an older version of Python without causing too +much disruption for users who haven't upgraded to this new version. -Too long, did not read: +Wether you are a user, or a developer, being aware of the issue listed here, at +least the main points should ease lots of the pain. + +# Too long, did not read: - Help and encourage users to install **pip 9.0+** - Help and encourage users to install **setuptools 24.3+** - - Use **`setup(..., python_requires='>=3.3')`** new option. + - As maintainer use `setup(..., python_requires='>=3.4')` new option. + - do use `pip install [-e] .` and do **not** invoke `setup.py` directly. - **Fail** early at **install time** if on Python 2. +## The problem + +Up until December 2016 it was hard to publish a new major version of library +that changed requirements in Python version and mark it as such so that user +system will not try to upgrade said library. + +With the recent changes in Python packaging this is now possible. + +As an example let's look at the example of the `fictious` library. + +- `fictious` 1.1, 1.2, 1.3, 1.4 are compatible Python 2.7 and 3.3+ +- `fictious` 2.0 has been released and is python 3.4+ only. -# As a user +As a Python 2.7 user, if I don't pay attention, or if the library is not +correctly tagged, if I issue the following: -## Install Pip 9.0 + $ python -c 'import fictious; print(fictious.__version__)' + 1.3.2 + $ pip install fiction --upgrade + +Either my system will install 2.0, which will not work, on the worst case +scenario, or fail to install, in which case I will not get the critical 1.4 +upgrade. + +## As a user + +### Install Pip 9.0 If you are already a Python 3 user, you should not encounter a lot of disruption. Please still check that the libraries you use follow best practices not to break for Python 2 users. Python is a community regardless of which -python version you have to a decide to run, making sure that things works make -the community strong. +python version you have to (or decided to) run, making sure that everything +works make the community strong. Make sure you have Pip >= 9.0, this is especially important if you have Python -2 installations. Having pip 9.0+ will not insure that you install will not -break, but they are more likely not to. Having a version off pip < 9.0 can lead -your system to try to upgrade to non-compatible versions of Python packages -even if these are marked as non-compatible. +2 installations. Having pip 9.0+ is not a guaranty to flawless upgrade. But pip +9.0+ does have a number of safety check not available on previous versions. + +Having a version of pip < 9.0 can lead your system to try to upgrade to +non-compatible versions of Python packages even if these are marked as +non-compatible. Help as many other _users_ as possible to install pip >=9.0, for the transition, it is the slowest part of the ecosystem to update, and is the only -piece that concerns all installations. +piece that requires action of all Python users. The simplest way to make sure all is up to date is to run the following for each installation of Python: - pip install --upgrade setuptools pip + $ pip install --upgrade setuptools pip This will install the latest version of pip and setuptools. You can issue the following to see the version of pip: - pip --version + $ pip --version + 9.0.0 + +All good. ## Setuptools -If you are on a system that will not install python wheel and use `setuptools`, -make sure you have setuptools >=24.2.0, or building Python 3 only libraries -might fail. In particular if authors have taken time to mark their library as -Python 3 only, the `python_requires` argument to `setup()` will not be -recognize and installation will fail. +If you are on a system for which no wheel is available, pip will try to +install a source distribution (aka `sdist`). + +Installing an `sdist` will require setuptools make sure you have setuptools +`>=24.2.0` (mnemonic: 2-42, [why +42](https://en.wikipedia.org/wiki/The_answer_to_life_the_universe_and_everything) +?) or building Python 3 only libraries is likely to fail. In particular if +library authors have taken time to mark their library as Python 3 only, the +`python_requires` argument to `setup()` may not be recognized and installation +will fail. Use the following to check setuptools version : - python -c 'import setuptools; print(setuptools.__version__) + $ python -c 'import setuptools; print(setuptools.__version__) + 24.2.0 -Again make sure tu upgrade pip and setuptools to make sure you have an up to +Again make sure to upgrade pip and setuptools to make sure you have an up to date system: - pip install --upgrade setuptools pip + $ pip install --upgrade setuptools pip ## Local package index If you are using a custom local package index, for example if you are working at a company with private packages, make sure it implement correctly -[pep-503](https://www.gg python.org/dev/peps/pep-0503/) and let pip knows about -the `python_requires` field. +[pep-503](https://www.python.org/dev/peps/pep-0503/) and let pip knows about +the `python_requires` field. This _mostly_ mean that the html you are exposing +should get a `data-python-requires` data attribute with the (html escaped) +version specifier. ## The state of PyPI Note that at the time of this writing the patches to `pypi.python.org` are not deployed yet but should hopefully be deployed soon. +[Warehouse](https://github.com/pypi/warehouse) and [Legacy +PyPI](https://github.com/pypa/legacy-pypi) have received various patches to +insure they support this new functionality. + +You can publish a package with the `requires_python` metadata **now**, it will +be correctly exposed once pypi is deployed. # Preparing your library @@ -109,65 +157,99 @@ setup( ``` Changes `>=3.3` accordingly depending on what version your library decides to -support. +support. In particular you can use `>=2.6` or `>=3.5` ! Note that this also +support the _compable with_ syntax: `~=2.5` (meaning, `>=2.5` and `<3`. This will make [PyPI aware](https://github.com/pypa/warehouse/pull/1448) that -your package is Python 3 only, and [allow +your package is Python 3.3+ only, and [allow pip](https://github.com/pypa/pip/pull/3877) to be [made aware of this](https://github.com/pypa/pypi-legacy/pull/506). +Thus as long as your user have a recent enough version of pip, and setuptools +they will get the right version of your library. + +# Unit Testing and documentation + +It is recommended **not** to invoke `setup.py` directly either with `install` or +`develop` subcommands. These may not correctly resolve dependencies, and can +install incompatible versions of dependencies. Please recommend and use `pip +install . ` and `pip install -e .` for regular and developer install. + +Check in scripts, and documentation that the correct installation command is +used. + +# Recommended Mitigations + +These are not mandatory but should make the transition seamless by warning your +user early enough _and_ providing useful error messages. -- Add a warning at _runtime_ early on master (before switching to Python 3 +## Runtime warning on master + +Add a warning at _runtime_ early on master (before switching to Python 3 only) ``` import warnings import sys if sys.version_info < (3,): - warnings.warn('You are using master of `Frobulator` with Python 2. Frobulator will soon be Python 3 only. See this issue to know more.', UserWarning) -else: - + warnings.warn('You are using master of `Frobulator` with Python 2. ' + 'Frobulator will soon be Python 3 only. ' + 'See this issue to know more.', + UserWarning) ``` -- Add an error early at import at runtime with a clear error message, leave the - early import compatible Python 2 for users to not be welcomed with a useless `SyntaxError`. - You are _allowed_ to use multi-line strings in error messages. +Your Python 2 user have a chance to upgrade, or get off master, (for example on +the LTS branch). + +## Fail early at import time + +Add an error early at import at runtime with a clear error message, leave the +early import compatible Python 2 for users to not be welcomed with a useless +`SyntaxError`. You are _allowed_ to use multi-line strings in error messages. + +Error at import time _will_ happen on system with old version of pip and +setuptools. Keep in mind that saying the package is Python 3 only is not a lot +more helpful than a Syntax error. The most reasonable reason would be out of +data pip and setuptools: ``` import sys if sys.version_info < (3,): - Raise ValueError( + raise ImportError( """You are running Frobulator 6.0 on Python 2 -Unfortunately Frobulator 6.0 and above re not compatible with Python 2 anymore, -and you still ended up with this version installed on your system. That's a -bummer sorry about that it should not have happen. Make sure you have pip >= -9.0 to avoid this kind of issues: +Unfortunately Frobulator 6.0 and above are not compatible with Python 2 +anymore, and you still ended up with this version installed on your system. +That's a bummer. Sorry about that. It should not have happen. Make sure you +have pip >= 9.0 to avoid this kind of issues, as well as setuptools >= 24.2: - $ pip install pip --upgrade + $ pip install pip setuptools --upgrade -Use the following to check pip version +You have various other choices -You have various choices: +- install an older version of Frobulator: -- You can still install an older version of Frobulator: - $ pip install frobulator<6.0 + $ pip install 'frobulator<6.0' - Upgrade your system to use Python 3. It would be great if you can figure out how this version ended up being installed, and try to check how to prevent that for future users. -This this page for more information : url to here for example. +See the following url for more up to date informations: + +https://i.am.an/url + """) ``` - +## Watch out for beta releases -Make sure your version number match pep440 or you will get surprises during + +Make sure your version number match pep 440 or you will get surprises during beta in particular as the `sdist` and `wheel` will appear as being different versions, in particular sdist (during beta/rc/post) can appear with a greater version number than wheels. Pip thus try to install the sdist instead of the @@ -178,38 +260,35 @@ The regular expression to check for validity of pep440 can be find below: `^([1-9]\\d*!)?(0|[1-9]\\d*)(\\.(0|[1-9]\\d*))*((a|b|rc)(0|[1-9]\\d*))?(\\.post(0|[1-9]\\d*))?(\\.dev(0|[1-9]\\d*))?` - +## Depend on setuptools You can mark your library as dependent on setuptools greater than 24.3 starting now, this will insure that during the next upgrade (when the packages drop python 2 support) will have the right version of setuptools. - - -# Recommended Mitigations - Of course regardless of all the care you will take for your library to no break and to install only on python 2, you will likely have cases where it still end up being installed on incompatible versions of Python. Simply because users upgrades rarely and only an old version of pip or setuptools is enough to make the all update process broken. -- Leave `setup.py` python 2 compatible and fail early. If you detect Python 2 - raise a clear error message and ask user to make sure they have pip >9.0 (or - migrate to Python 3). You can (try to) conditionally import pip and check for - its version but this might not be the same pip. Failing early is important to - make sure the Python installation does not install and incompatible version. - Otherwise user code can fail at runtime arbitrary later in the future, which - can be a difficult to debug and fix. +## fail early in setup.py -- If you control dependant packages, Make sure to include conditional - dependencies depending on the version of Python. +Leave `setup.py` python 2 compatible and fail early. If you detect Python 2 +raise a clear error message and ask user to make sure they have pip >9.0 (or +migrate to Python 3). You can (try to) conditionally import pip and check for +its version but this might not be the same pip. Failing early is important to +make sure the Python installation does not install and incompatible version. +Otherwise user code can fail at runtime arbitrary later in the future, which can +be a difficult to debug and fix. Get inspiration from the message of failure at +runtime, and adapt for installation time. -- Regardless of whether the installation step fails on Python 2, implement a - similar check in the top level import of your package. +## Fix dependant libraries +If you control dependant packages, Make sure to include conditional dependencies +depending on the version of Python. -# Alternative mitigation +# Incorrect mitigations This is a collection of "mitigation" or "solutions" you will find on the web and that you will hear about. This is an attempt to acknowledge them, and @@ -222,26 +301,25 @@ It is possible to release a meta-package that has _virtually_ no code and rely on conditional dependency to install its actual core code on the user system. For example, Frob-6.0 could be a meta-package which depends on Frob-real-py2 on Python <3.0, and Frob-real-py3 on Python >= 3.4. While -this approach is _doable_ this can make import confusing. +this approach is _doable_ this can make imports confusing. Moreover, upgrading your package may need the user to explicitly tell pip to -upgrade dependencies as `pip install frob` will only upgrade the meta-package. +upgrade dependencies as `pip install -U frob` will only upgrade the meta-package. ### Multiple Sdist. Pip (used to) support a "feature" where a sdist ending in `-pyX.Y.tar.gz` would only be seen as compatible on Python X.Y, thus it used to be possible to -publish multiple sdist of a package targeting various python version. +publish multiple sdist of a package targeting various python version. Though it is not possible anymore to upload multiple sdist on PyPI. This solution is thus not possible. ### Wheel only ? -Break downstream packages. - - - +Releasing a package only using wheel for a given python version is doable, but +this will break downstream packages that may require the original source to +reproduce the build. # Why all that ? @@ -254,13 +332,13 @@ Many libraries have transitioned from Python 2-only to Python 2 + 3. And the issue of transitioning to Python 3 only is relatively recent. Technically it can also apply to libraries that are only stopping support for 2.6, or even are already Python 3 only, but are starting to stop support for earlier versions of -Python. For example a library releasing a Python 3.4+ only version. +Python. For example a library releasing a Python 3.4+ only version. Python 3.3 was release end of 2012, and was the first version to support (again) `u` as a prefix for Unicode string. It was one of the first minor version of Python 3 that saw a majority of single-source project working both on Python 2 and Python 3. These are the Project that will likely be affected by -this issue. +this issue. The introduction of Python 3 was chaotic, there are still strong argument both in Python 2 and Python 3 camps. In the one suffering the most from this are @@ -268,5 +346,3 @@ users. Starting with the fact that inevitably some libraries will stop support for Python 2 and release Python 3 only library. And that inevitably some system will will not be upgraded to Python 3 how can we _ensure_ that users get the _least_ breakage as possible ? And what are the best practices to follow. - - From 7cf319e1f40bf36bc475343004466a1231567ea0 Mon Sep 17 00:00:00 2001 From: Matthias Bussonnier Date: Mon, 17 Apr 2017 11:28:04 -0700 Subject: [PATCH 7/9] 'finish up practicalities' --- _includes/css/main.css | 10 ++++++++++ _practicalities/intro.md | 15 +++++++++++---- _sections/80-update-my-project.md | 7 ++++--- practicalities/index.html | 10 +++++++++- 4 files changed, 34 insertions(+), 8 deletions(-) diff --git a/_includes/css/main.css b/_includes/css/main.css index f511a0f..8424c97 100644 --- a/_includes/css/main.css +++ b/_includes/css/main.css @@ -25,6 +25,16 @@ html { box-sizing: border-box; } #{{id}} .sectiondivider { color: {{ bg }}; } {% endfor %} +{% for node in site.practicalities %} + {% capture id %}{{ node.id | remove:'/' | downcase }}{% endcapture %} + {% capture bg %}{% if site.colors[node.bg] %}{{ site.colors[node.bg] }}{% else %}{{ node.bg }}{% endif %}{% endcapture %} + {% capture fg %}{% if site.colors[node.color] %}{{ site.colors[node.color] }}{% else %}{{ node.color }}{% endif %}{% endcapture %} + nav .p-{{id}} { border-color: {{ bg }}; } + #{{id}} { background-color: {{ bg }} !important; color: {{ fg }}; } + #{{id}} a { color: {{ fg }}; } + #{{id}} .sectiondivider { color: {{ bg }}; } +{% endfor %} + /* ----- code, syntax highlighting, etc ----- */ diff --git a/_practicalities/intro.md b/_practicalities/intro.md index 5a0b099..9495a0c 100644 --- a/_practicalities/intro.md +++ b/_practicalities/intro.md @@ -1,16 +1,22 @@ --- +bg: '#4da45e' +color: white +title: practicalities +fa-icon: pencil +id: bar --- -Add note here to explain that this does _not_ prevent _nor_ discourage library -author to release 2 version of their software one Python 3 only and the other -python 2. +We do not discourage authors to release software on Python 2. While this guide +is mostly written with the assumption that software are going to stop Python 2 +support, it does perfectly apply to a package that wish to not support Python 3, +or is stopping support for any minor version. This page gather information and links to resources allowing to release a library that stop supporting an older version of Python without causing too much disruption for users who haven't upgraded to this new version. -Wether you are a user, or a developer, being aware of the issue listed here, at +Whether you are a user, or a developer, being aware of the issue listed here, at least the main points should ease lots of the pain. # Too long, did not read: @@ -20,6 +26,7 @@ least the main points should ease lots of the pain. - As maintainer use `setup(..., python_requires='>=3.4')` new option. - do use `pip install [-e] .` and do **not** invoke `setup.py` directly. - **Fail** early at **install time** if on Python 2. + - We are giving a talk at PyCon 2017 (likely recorded), add link here. ## The problem diff --git a/_sections/80-update-my-project.md b/_sections/80-update-my-project.md index b59e159..1bd4287 100644 --- a/_sections/80-update-my-project.md +++ b/_sections/80-update-my-project.md @@ -15,9 +15,10 @@ encountered both by developers and users during this process. We are thus collecting a list of things to be aware of for when transitioning that are less obvious. For example : as a developer, how to make sure that your package manager like pip does not upgrade your libraries to incompatible -versions on Python 2. Check [our -repository](https://github.com/python3statement/python3statement.github.io) for -more information. +versions on Python 2. See the [Practicality Page](/practicalities/) for +informations and Check [our +repository](https://github.com/python3statement/python3statement.github.io) to +contribute. diff --git a/practicalities/index.html b/practicalities/index.html index e04b5d1..f0e0dc1 100644 --- a/practicalities/index.html +++ b/practicalities/index.html @@ -19,9 +19,17 @@
+ + {% for page in site.practicalities %} -
+ {% capture id %}{{ page.id | remove:'/' | downcase }}{% endcapture %} +
{{ page.content }} From b542e012093e0e36e34b17731fbe7690b8ed02fa Mon Sep 17 00:00:00 2001 From: Matthias Bussonnier Date: Tue, 18 Apr 2017 11:46:53 -0700 Subject: [PATCH 8/9] Take into account most of Thomas comments. --- _practicalities/intro.md | 68 ++++++++++++++++++++-------------------- 1 file changed, 34 insertions(+), 34 deletions(-) diff --git a/_practicalities/intro.md b/_practicalities/intro.md index 9495a0c..117cdf9 100644 --- a/_practicalities/intro.md +++ b/_practicalities/intro.md @@ -36,15 +36,15 @@ system will not try to upgrade said library. With the recent changes in Python packaging this is now possible. -As an example let's look at the example of the `fictious` library. +As an example let's look at the example of the `fictitious` library. -- `fictious` 1.1, 1.2, 1.3, 1.4 are compatible Python 2.7 and 3.3+ -- `fictious` 2.0 has been released and is python 3.4+ only. +- `fictitious` 1.1, 1.2, 1.3, 1.4 are compatible Python 2.7 and 3.3+ +- `fictitious` 2.0 has been released and is python 3.4+ only. As a Python 2.7 user, if I don't pay attention, or if the library is not correctly tagged, if I issue the following: - $ python -c 'import fictious; print(fictious.__version__)' + $ python -c 'import fictitious; print(fictitious.__version__)' 1.3.2 $ pip install fiction --upgrade @@ -96,10 +96,8 @@ If you are on a system for which no wheel is available, pip will try to install a source distribution (aka `sdist`). Installing an `sdist` will require setuptools make sure you have setuptools -`>=24.2.0` (mnemonic: 2-42, [why -42](https://en.wikipedia.org/wiki/The_answer_to_life_the_universe_and_everything) -?) or building Python 3 only libraries is likely to fail. In particular if -library authors have taken time to mark their library as Python 3 only, the +`>=24.2.0` or building Python 3 only libraries is likely to fail. In particular +if library authors have taken time to mark their library as Python 3 only, the `python_requires` argument to `setup()` may not be recognized and installation will fail. @@ -131,9 +129,6 @@ deployed yet but should hopefully be deployed soon. PyPI](https://github.com/pypa/legacy-pypi) have received various patches to insure they support this new functionality. -You can publish a package with the `requires_python` metadata **now**, it will -be correctly exposed once pypi is deployed. - # Preparing your library @@ -163,16 +158,16 @@ setup( ) ``` -Changes `>=3.3` accordingly depending on what version your library decides to +Change `>=3.3` accordingly depending on what version your library decides to support. In particular you can use `>=2.6` or `>=3.5` ! Note that this also -support the _compable with_ syntax: `~=2.5` (meaning, `>=2.5` and `<3`. +support the _compable with_ syntax: `~=2.5` (meaning, `>=2.5` and `<3`). This will make [PyPI aware](https://github.com/pypa/warehouse/pull/1448) that your package is Python 3.3+ only, and [allow pip](https://github.com/pypa/pip/pull/3877) to be [made aware of this](https://github.com/pypa/pypi-legacy/pull/506). -Thus as long as your user have a recent enough version of pip, and setuptools +Thus as long as your user have recent enough versions of pip and setuptools they will get the right version of your library. # Unit Testing and documentation @@ -212,7 +207,7 @@ the LTS branch). Add an error early at import at runtime with a clear error message, leave the early import compatible Python 2 for users to not be welcomed with a useless -`SyntaxError`. You are _allowed_ to use multi-line strings in error messages. +`SyntaxError`. Don't hesitate to use multi-line strings in error messages. Error at import time _will_ happen on system with old version of pip and setuptools. Keep in mind that saying the package is Python 3 only is not a lot @@ -229,7 +224,7 @@ if sys.version_info < (3,): Unfortunately Frobulator 6.0 and above are not compatible with Python 2 anymore, and you still ended up with this version installed on your system. -That's a bummer. Sorry about that. It should not have happen. Make sure you +That's a bummer. Sorry about that. It should not have happened. Make sure you have pip >= 9.0 to avoid this kind of issues, as well as setuptools >= 24.2: $ pip install pip setuptools --upgrade @@ -267,17 +262,6 @@ The regular expression to check for validity of pep440 can be find below: `^([1-9]\\d*!)?(0|[1-9]\\d*)(\\.(0|[1-9]\\d*))*((a|b|rc)(0|[1-9]\\d*))?(\\.post(0|[1-9]\\d*))?(\\.dev(0|[1-9]\\d*))?` -## Depend on setuptools - -You can mark your library as dependent on setuptools greater than 24.3 starting -now, this will insure that during the next upgrade (when the packages drop -python 2 support) will have the right version of setuptools. - -Of course regardless of all the care you will take for your library to no break -and to install only on python 2, you will likely have cases where it still end -up being installed on incompatible versions of Python. Simply because users -upgrades rarely and only an old version of pip or setuptools is enough to make -the all update process broken. ## fail early in setup.py @@ -285,7 +269,7 @@ Leave `setup.py` python 2 compatible and fail early. If you detect Python 2 raise a clear error message and ask user to make sure they have pip >9.0 (or migrate to Python 3). You can (try to) conditionally import pip and check for its version but this might not be the same pip. Failing early is important to -make sure the Python installation does not install and incompatible version. +make sure the Python installation does not install an incompatible version. Otherwise user code can fail at runtime arbitrary later in the future, which can be a difficult to debug and fix. Get inspiration from the message of failure at runtime, and adapt for installation time. @@ -295,7 +279,7 @@ runtime, and adapt for installation time. If you control dependant packages, Make sure to include conditional dependencies depending on the version of Python. -# Incorrect mitigations +# Non recommended mitigations This is a collection of "mitigation" or "solutions" you will find on the web and that you will hear about. This is an attempt to acknowledge them, and @@ -313,6 +297,22 @@ this approach is _doable_ this can make imports confusing. Moreover, upgrading your package may need the user to explicitly tell pip to upgrade dependencies as `pip install -U frob` will only upgrade the meta-package. +## Depend on setuptools + +You can mark your library as dependent on setuptools greater than 24.3 this +will insure that during the next upgrade (when the packages drop python 2 +support) will have the right version of setuptools. + +Of course regardless of all the care you will take for your library to no break +and to install only on python 2, you will likely have cases where it still end +up being installed on incompatible versions of Python. Simply because users +upgrades rarely and only an old version of pip or setuptools is enough to make +the all update process broken. + +Plus setuptools is rarely an actual dependency of your project but a +requirement to build wheels. + + ### Multiple Sdist. Pip (used to) support a "feature" where a sdist ending in `-pyX.Y.tar.gz` would @@ -341,11 +341,11 @@ can also apply to libraries that are only stopping support for 2.6, or even are already Python 3 only, but are starting to stop support for earlier versions of Python. For example a library releasing a Python 3.4+ only version. -Python 3.3 was release end of 2012, and was the first version to support -(again) `u` as a prefix for Unicode string. It was one of the first minor -version of Python 3 that saw a majority of single-source project working both -on Python 2 and Python 3. These are the Project that will likely be affected by -this issue. +Python 3.3 was release at the end of end of 2012, and was the first version to +support (again) `u` as a prefix for Unicode string. It was one of the first +minor version of Python 3 that saw a majority of single-source project working +both on Python 2 and Python 3. These are the Project that will likely be +affected by this issue. The introduction of Python 3 was chaotic, there are still strong argument both in Python 2 and Python 3 camps. In the one suffering the most from this are From f9ed6e81a5b67652dd82ab2267e22da70df01f08 Mon Sep 17 00:00:00 2001 From: Matthias Bussonnier Date: Tue, 18 Apr 2017 13:41:39 -0700 Subject: [PATCH 9/9] Last changes from Thomas --- _practicalities/intro.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/_practicalities/intro.md b/_practicalities/intro.md index 117cdf9..d8a932d 100644 --- a/_practicalities/intro.md +++ b/_practicalities/intro.md @@ -294,9 +294,6 @@ For example, Frob-6.0 could be a meta-package which depends on Frob-real-py2 on Python <3.0, and Frob-real-py3 on Python >= 3.4. While this approach is _doable_ this can make imports confusing. -Moreover, upgrading your package may need the user to explicitly tell pip to -upgrade dependencies as `pip install -U frob` will only upgrade the meta-package. - ## Depend on setuptools You can mark your library as dependent on setuptools greater than 24.3 this @@ -341,7 +338,7 @@ can also apply to libraries that are only stopping support for 2.6, or even are already Python 3 only, but are starting to stop support for earlier versions of Python. For example a library releasing a Python 3.4+ only version. -Python 3.3 was release at the end of end of 2012, and was the first version to +Python 3.3 was release at the end of 2012, and was the first version to support (again) `u` as a prefix for Unicode string. It was one of the first minor version of Python 3 that saw a majority of single-source project working both on Python 2 and Python 3. These are the Project that will likely be