April (then May) 2020 Release Tracking Issue

[techalchemy]

This is an internal tracking issue which will tie in the related issues which will be tackled/still need to be updated for the purposes of cutting the release. It's been a long time coming (see #4058 (comment) and #3742 (comment) for some past comments on that) and there's a tentative goal of getting a pre-release out in March 2020.

(Edited by @brainwane to say: fixing some brokenness in the continuous integration setup is delaying this release 2020.04.1a1 till -- new estimate -- 21 April 2020.)

(Edited by @brainwane to say: the prerelease 2020.4.1b1 is now out, as of 29 April, and Dan aims to get the next release out in about a week.)

(Edited by @brainwane in conversation with Dan on 5 March 2020 and then throughout March & April)

• Other Items in the March 2020 release milestone:
  • Pipenv installs wrong packages (ignoring package lock) #3613 Pipenv installs wrong packages (ignoring package lock)
  • Change docs links to pipenv.pypa.io #4137 Update README and Read the Docs project to point to docs at pipenv.pypa.io
• Fix pip-shims test failures (merge the bugfix/56 branch)
• requirementslib fixes:
  • AST parsing of setup.py fails on non-addition type binOps sarugaku/requirementslib#204 requirementslib issue AST parsing of setup.py fails on non-addition type binOps
  • AST parsing fails on non-utf8 setup.py files sarugaku/requirementslib#205 AST parsing fails on non-utf8 setup.py files
  • We shouldn't install upgraded versions of test dependencies in CI sarugaku/requirementslib#214 We shouldn't install upgraded versions of test dependencies in CI
  • Add requirement calculator for artifacts in pypi sarugaku/requirementslib#219 calculator Add requirement calculator for artifacts in PyPI
  • Review requirements-lib issues for other pipenv release blockers
• Make new releases of related libraries:
  • requirementslib
  • pip-shims
  • pythonfinder
• Finish, approve and merge Update vendored dependencies, relevant API usage, and CI config #4169 to address missing dependencies/imports
  • Fix news entry/entries
  • Fix broken CI configuration
  • Get tests to pass
• Fix Your API Key is invalid #4188 by either getting pyup API key working again or choosing and implementing an alternative
• Get GitHub tests to pass on master (skipping the test mentioned in Debug test_install_markers for python 3.6 in github CI #4201)
• Improve release automation to allow generation of prereleases
  • Generate changelog in draft mode
  • Don't autotag & push tag to repo
• Update and run through "Test locally" and "Uploading the release" for packaging/release checklist
• Package and upload pipenv prerelease version 2020.04.1a1 to PyPI

That's what Dan aims to do by 21 April 2020. Then:

• Make another new release of requirementslib that incorporates a fix to parsing setup.cfg via setuptools is currently broken sarugaku/requirementslib#216
  • Relock requirementslib dependencies
• Publicize pre-release and ask for a week of testing
  • Especially testing on Windows against PEP 517 backends and involving virtualenvs
• After ~1 week (if no showstopper bugs), publish new release

How others can help:

• replying to new users' questions in pipenv's GitHub issues
• help develop a "Pipenv workflows to test" list, along these lines

[JBKahn]

This seems like a good place to ask about when the next release might be, what the blockers are and if there is anything someone can do to help?

[adawalli]

My team is anxious for the next update, specifically to address #3298. Are there major blockers still left?

[AlJohri]

linking #3742

[brainwane]

I noticed pypa/packaging.python.org#701 today and donated about 90 minutes of time helping @techalchemy get some more clarity on what's blocking him from making the new release (IRC conversation)). #3369 (comment) now has a release checklist. @techalchemy could use help with those release blockers, in case @JBKahn or anyone else wants to help out.

[Froskekongen]

@techalchemy: When looking at the issues linked here, a lot of them are fixed. I think it would be nice to check the boxes for the issues that have been fixed, so that people can see that there is progress by just looking at the first post in this thread (:

[amhrasmussen]

@brainwane, @techalchemy, excuse my interference and potential ignorance, but may I suggest taking #2227 and/or #3520 out of this release? My impression is that getting more or less anything released is essential and beneficial. #2227 looks like a new feature while #3520 has a workaround described (with no objection from reporter).

Similarly, is it strictly needed to "make new releases of related libraries" before getting next pipenv out? While looking into #3613, I found the current master branch to just work...

I'm not suggesting these issues are not important, just that they should not block the improvements already in master from getting to the people.

[shamoons]

March is almost over

[fridex]

It looks like #3520 is fixed in the current master, #2227 does not look like a critical fix for an update.

Is there anything blocking where the community can be helpful? I'm happy to give you my hand. Otherwise, I see just release updates and docs updates.

[brainwane]

@Froskekongen @amhrasmussen @fridex thanks for the pointers! I donated some more time yesterday and @techalchemy and I worked through a few more of the relevant items on the list.

When looking at the issues linked here, a lot of them are fixed. I think it would be nice to check the boxes for the issues that have been fixed, so that people can see that there is progress by just looking at the first post in this thread (:

Thanks! Several more are checked now. :-)

It looks like #3520 is fixed in the current master

Could you please say that in a comment on #3520? Thanks.

#2227 does not look like a critical fix for an update.

@brainwane, @techalchemy, excuse my interference and potential ignorance, but may I suggest taking #2227 and/or #3520 out of this release? My impression is that getting more or less anything released is essential and beneficial. #2227 looks like a new feature while #3520 has a workaround described (with no objection from reporter).

Thanks. Dan agreed with you and we removed #2227 as a blocker on this release. If you could help with #3520 by confirming that the workaround works, and commenting there, that would be great.

Similarly, is it strictly needed to "make new releases of related libraries" before getting next pipenv out? While looking into #3613, I found the current master branch to just work...

I'm not suggesting these issues are not important, just that they should not block the improvements already in master from getting to the people.

As I understand it, those libraries are important for pipenv functionality, as well as properly testing pipenv to ensure the release works on the supported OS/environment combinations. I could be wrong but Dan said "tbh most of the work happens in the ancillary libraries these days".

Is there anything blocking where the community can be helpful? I'm happy to give you my hand. Otherwise, I see just release updates and docs updates.

Please help by replying to new users' questions in pipenv's GitHub issues; that way Dan doesn't have to worry about replying to those, and can concentrate on this release.

Getting closer to a release!

[brainwane]

New update email from @techalchemy on distutils-sig (mirrored on the pypa-dev list). Includes a few ways you can help.

[Systemcluster]

@brainwane I'd think #4218 would be a blocker as well? It's a regression and completely breaks both existing and new projects with certain dependencies.

[techalchemy]

To provide an additional update here, I will try to have this release out tomorrow (I realize this is a day later than planned, but a few of the issues were relatively tricky to track down -- huge thanks to those of you who were able to test, provide feedback, and help provide insight into some of the subtle issues going on.

[techalchemy]

I've posted a more thorough postmortem on the release blocking issue here -- this is now waiting on builds to finish but as it's now 1am on Saturday morning I will most likely refrain from releasing until Monday to avoid breaking anything while nobody is around to respond / react.

Thanks again to everyone who has provided valuable debugging information to help nail down some of these issues, it has made the process much smoother.

[Immortalin]

@techalchemy ?

[liammonahan]

@Immortalin Even if Dan said to potentially expect something on Monday, your comment is not constructive. Please check yourself.

[liammonahan]

Dan, we all appreciate your hard work. I think most of us realize that we need to find more ways to be helpful to you.

[gentooboontoo]

#4251 could be a showstopper (pipenv install --outdated failure).

[washeck]

Do you plan to release another beta version? I am using version 2020.4.1b1 and I encountered an error in locking a project with psycopg2. I see there were fixes such as #4231 so I'd rather check it using the latest version of pipenv with all bugfixes rather than reporting something already fixed.

[techalchemy]

Here's a release update. Tl;dr: We ran into unexpected issues updating dependent libraries that pipenv vendors, and that caused a delay. There'll be a new prerelease sometime in the next few days.

Details:
Late last week, I was merging what should have been a simple fix in vistir to address #4195. I was making this change as part of what should have been some quick revendoring in preparation to release; vistir is one of the libraries that provides some of the cross-platform/Python 2/3 compatibility support for Pipenv. However, CI failed across the board for all kinds of strange reasons, and this has been a bit time-consuming to figure out and address properly. I couldn't just revert the fix and release anyway, because without this fix, there was a risk of breaking cross-platform and compatibility in Pipenv.

Plan:

• Release the libraries (https://github.com/sarugaku/vistir / https://github.com/sarugaku/requirementslib)
• Re-vendor updated libraries in Pipenv
• Merge remaining low-risk, ready-to-merge PRs in Pipenv
• Review Pipenv issues for release blockers
• Pre?Release: probably make a fresh prerelease of Pipenv prior to a canonical release

Additional notes

• Failures were not related to the change, but may be related to changes in how Azure (CI environment) creates Python instances
• In one case this may have exposed a significant potential bug in how output is encoded and decoded on Windows, while the other case relates to representing paths to the latest version of MacOS
• The functionality is tested thoroughly by property-based tests, so we can be confident that the failures are real issues and the tests would capture ongoing issues
• Thanks to the other library and Pipenv maintainers, and testers and other users, for helping me track down stuff

[r-richmond]

First off let me say thank you so much for the work you are putting into this release and keeping this project going. Pipenv has saved me multiple times over.

All that said and at the risk of going slightly off topic. Given the latest hiccup in the release process

Late last week, I was merging what should have been a simple fix in vistir to address #4195. I was making this change as part of what should have been some quick revendoring in preparation to release; vistir is one of the libraries that provides some of the cross-platform/Python 2/3 compatibility support for Pipenv. However, CI failed across the board for all kinds of strange reasons, and this has been a bit time-consuming to figure out and address properly. I couldn't just revert the fix and release anyway, because without this fix, there was a risk of breaking cross-platform and compatibility in Pipenv.

Have you considered dropping python 2 support (it is EOL) and removing its requirements (i.e. vistir and maybe others)?

I assume long term python2 support will be dropped but if dropping it now simplifies the release process, reduces the complexity of the project, and makes it easier to cut this release perhaps it should be done now rather than later?

p.s. thanks again for your hard work

edit: as flimm suggested I opened another issue #4261 for discussion on dropping python 2 to try and keep the conversation here on 2020's first release

[Flimm]

Let's keep conversation about dropping support for Python 2 in a separate GitHub issue, as I have a feeling it could get noisy.

[GPHemsley]

@r-richmond vistir is used in the Python 3 branch of logic, too, so I don't think removing support for Python 2 would make this release easier.

[techalchemy]

So I have gone ahead and cut another pre-release of pipenv (2020.4.1b2 -- the release will be tagged with the date it comes out so don't worry to much about the naming).

I think this release captures a majority, if not all of the changes I'm hoping to include. I did merge one significant change since the last pre-release which should avoid re-launching processes to attempt to pip install already-satisfied dependencies, so please report any issues as I plan to release for real on Wednesday of next week.

Thanks again to everyone who helped test, provided feedback, and helped get fixes merged!

[brainwane]

I just spoke with @techalchemy . He is checking on #4263 and #3592 to ensure they are not blockers. He also said

i only saw one issue about vendored import paths ...
revendoring finished up and didn't resolve the import path issue, so i'll likely just accept the PR on the issue & generate a patch for it for now

I presume this is #4267 but I'm not sure.

Once those are taken care of, I believe he aims to release today.

[techalchemy]

^ Release is up, thank you to everyone who helped with testing!

[haizaar]

Wow! Absolutely amazing! Thank you so much for bringing pipenv back to life!

…

On Thu, 28 May 2020, 15:17 Dan Ryan, ***@***.***> wrote: ^ Release is up, thank you to everyone who helped with testing! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3369 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAOGWNSCAEOBRJ5S4WZHADRTXXWHANCNFSM4GJK7LZA> .

[sim0nx]

^ Release is up, thank you to everyone who helped with testing!

Very much appreciated! 👍 🥳

[brainwane]

Per https://pypi.org/project/pipenv/#history , Pipenv 2020.5.28 is now out, so I'm closing this issue and thus the release milestone.

Thanks to @techalchemy - and thanks to Canonical for letting him work on this during some of his day job time.

In my opinion: If you use Python for your job, and want better and more frequent releases of this and other Python packaging/distribution/installation tools, ask your employer to chip in with a sponsorship. If it has a bigger budget, the Packaging Working Group can pay contractors to consistently work on these projects and maintain them well.

And if you'd like to help out by volunteering, read this announcement and follow issue #4130 on improving Pipenv's roadmap and contributor-maintainer processes, which is probably where more of that discussion will happen.

Thanks to everyone who contributed to this release -- users, testers, signal-boosters, reviewers, patch authors, and the people who said nice things in comments here or on mailing lists and social media!

[brainwane]

(I wrote a blog post about what it took to break the bottleneck and get this release out, and about what you can do to replicate this for other projects -- either yourself or through my firm, Changeset Consulting.)

[haizaar]

Sumana, thank you for sharing that story. And even more for putting your time and energy into pipenv revival. As you mentioned, the unlocked value is enormous.

…

On Fri, 11 Sep 2020 at 21:05, Sumana Harihareswara ***@***.***> wrote: (I wrote a blog post <https://www.harihareswara.net/sumana/2020/09/08/0> about what it took to break the bottleneck and get this release out, and about what you can do to replicate this for other projects -- either yourself or through my firm, Changeset Consulting <https://changeset.nyc/> .) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3369 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAOGWNIKWJKNTMUH5PNUZ3SFIAAZANCNFSM4GJK7LZA> .

-- Zaar

[g1augusto]

Hi everyone,

does this address the WSL issues seen on #3488 ?

[uranusjr]

The issue you mentioned cannot be addressed in pipenv. You need to configure your environment as described in the issue yourself.