pipenv seems to no longer be maintained: don't recommend it anymore

[vstinner]

The Python Packaging User Guide recommends to use pipenv:

• https://packaging.python.org/guides/tool-recommendations/
• https://packaging.python.org/tutorials/managing-dependencies/

But pipenv seems to no longer be maintained:

• Multiple users requested new pipenv release:

  • When is the next release ? pipenv#4127 created in January 2020
  • If this project is dead, just tell us pipenv#4058 created in December 2019
  • New release? pipenv#3742 created in May 2019
  • etc.

• Latest release in November 2018: https://pypi.org/project/pipenv/#history

• on the other side, https://github.com/pypa/pipenv/commits/master shows that new commits are still added

Note: this issue is not the right place to request a new pipenv release, https://github.com/pypa/pipenv/issues is the right place and there are already many open issues for that.

There is https://pypi.org/project/thoth-pipenv/ which is a more recent pipenv version but under a different name: https://pypi.org/project/thoth-pipenv/#history

See also https://pypi.org/project/micropipenv/ which says:

My Pipenv installation is broken and Pipenv upstream did not issue any new Pipenv release - see also Thoth’s Pipenv release for possible fix.

---

See also closed issue #589 "Remove references to pipenv".

[brainwane]

@techalchemy or @uranusjr could you comment on this?

[webknjaz]

Yeah, a lot of people are in favor of not recommending it but the other side feels strongly about using it in the guides.

@ncoghlan ^

[brainwane]

@techalchemy is working towards a release but facing some difficulty and would like help: see today's IRC logs.

[di]

I think at the very least we should de-prioritize or give a caveat about our recommendations of pipenv until a new release is made and the project is back on track. Following our recommendation and finding the project to be in a broken state is extremely painful for users, and is currently damaging to the reputation and trust users have in this guide.

[fconil]

I agree that pipenv should not be recommended anymore (at least deprioritized with warnings).

I started using pipenv, convinced it was the tool officially recommanded by the Python community.

I discovered pipenv was stalled for a long time, then I searched and found some information on the tool's drawbacks :

1. RIP Pipenv
2. Pipenv: promises a lot, delivers very little

Poetry seemed to be a good replacement but I am conviced by the possible vendor lock-in argument in the (2) article, especially after the pipenv experiment.

I have used setup.cfg that I prefer to setup.py but unfortunately the documentation is incomplete.

I have been told that we should now use pyproject.toml (PEP 517 and 518), it sounds good.

I would now like to use several tools, that use the Python standards (so pyproject.toml) and comply to the UNIX philosophy of “do one thing, and do it well”. So :

• pip for package installation
• virtualenv or venv pour virtual environment
• flit for putting package on PyPI

I considered pip-tools for the dependencies management, but it does not use pyproject.toml yet and does not extract "dev" dependencies specified in a setup.cfg file. So I will have to duplicate the data in requirements.in / dev-requirements.in

So I will be interested if you have some recommendations on tools that work / will work with the adopted standard.

[JBKahn]

Is it possible to update the managing dependencies page with examples from pipenv, poetry, pip-tools and pip?

As someone who made the move to poetry after just too many issues with pipenv, I was in favour of removing it. That being said, if the pypa took a non-biased approach of providing tutorial info on all the recommended options, that would also be a good middle ground.

[nealmcb]

What @fconil says makes lots of sense to me. I appreciate the fact that Poetry supports packaging of libraries, which last I heard was not a goal of Pipenv, and I gravitated to Poetry for other reasons as well.
But I'm surprised to see that Poetry is putting variables like name and description in a private [tool.poetry] section of pyproject.toml, presumably because those haven't been defined in PEP 517 and PEP 518. E.g.

[tool.poetry]
name = "myprojectname"
version = "0.1.0"
description = "my description"
authors = ["Your Name <you@example.com>"]
license = "MIT"

I note some other custom sections also.

[tool.poetry.dependencies]
python = "^3.7"

I don't see any PEPs proposing to standardize these yet - has it been discussed somewhere?
For now, perhaps @JBKahn's proposal is best, to at least summarize the most basic elements of the most popular approaches.

[uranusjr]

@nealmcb There is some effort in standardising the metadata. @brettcannon and I had been comparing formats in this repo, including competing ones in Python and other ecosystems we can draw inspiration from. Unfortunately I couldn’t find the discussion that led to this work (somewhere on discussion.python.org; should have put it in the readme when I started). This will probably get some discussion at this year’s PyCon US during the packaging mini-summit (I hope).

The process would be very long and drawn out, partly because the completing standards are very, um, conflicting. But please do help out if you are interested in this area; this topic really needs more attention and resource than it’s receiving now!

[nealmcb]

@uranusjr Thanks for the links and encouragement. I found these potentially-related dicussions:

• Structured, Exchangeable lock file format (requirements.txt 2.0?)
• WIP Package manager using __pypackages__ and pyproject.toml

[brettcannon]

So I think there are a couple of questions in this thread:

1. Do we want to continue to recommend pipenv?
2. Do we want the tutorial to provide instructions for other popular tools?

For me, I think the general answer here is we should either switch which tool is recommended at this point or explain a couple of them. The problem is as a tutorial do we want to give e.g. 4 sets of instructions for pipenv, poetry, flit, and setuptools? To me a tutorial is almost inherently opinionated and thus it might be hard to structure it such that it isn't overwhelming to a newcomer. But relying on a single tool is tough when it doesn't have the size and backing of something like pip where it's possible for a project to languish.

And we have to find someone willing to write all of this. 😉 Would it perhaps be better to come up with guidelines of what we would want from a tutorial, ask each tool to provide such a tutorial, and then we have an explainer page that says from a very high level what you need to do to be successful (e.g. virtual environments, etc.) and then link to these tutorials with an explanation as to why you may choose one tool over another? That way the tutorials are up-to-date as they would be managed with the tool, we recommend a couple so we are not in the business of making a judgment call of which tool is best or end up in this situation again, and beginners won't necessarily get overwhelmed if the explainer page keeps it simple and to the point.

@uranusjr I submitted to the summit the topic of standardizing metadata in pyproject.toml. We will see if it gets accepted. 😄

[webknjaz]

@fconil

• flit for putting package on PyPI

If you want to stick to using the default ecosystem tools, this should be twine.
Also, flit isn't "for putting package on PyPI", it's a build wrapper that also has a feature to do uploads. It's doing more than one thing (and has its own restrictions if you ask me).

Personally, I use https://pypi.org/project/pep517/ to build dists. I've heard that some of pip maintainers were thinking about adding a command for this into pip itself but atm you can only build binary dists using pip wheel, not sdists.

I considered pip-tools for the dependencies management, but it does not use pyproject.toml yet and does not extract "dev" dependencies specified in a setup.cfg file. So I will have to duplicate the data in requirements.in / dev-requirements.in

Technically, "dev" isn't a standardized location but an arbitrary helper implemented as "extra". There's no such thing as dev deps in the dist metadata. Because installable artifacts don't really need them and developers would use Git repos anyway. I think you should mostly stick to specifying dev-requirements.in b/c adding such extra doesn't seem right.

I think it's quite reasonable to use pip-tools, even though it's not a PyPA project. Consider this article https://hynek.me/articles/python-app-deps-2018/, it's still relevant.

Is it possible to update the managing dependencies page with examples from pipenv, poetry, pip-tools and pip?

It may be a good idea to implement it via smth like https://sphinxcontrib-contentui.readthedocs.io/en/latest/tabs.html / https://djungelorm.github.io/sphinx-tabs/.

[pradyunsg]

One of the things with tutorials is to not overwhelm the user (who may be a beginner) with multiple choices up front and instead, provide them with a good default and present alternatives. So, no, I don't think having tabs or providing instructions for multiple tools in the same tutorial is a good idea. That said, there's a fairly sound argument that pipenv is not a "good default" as of today.

I'm OK to have multiple tutorials, but we can/should have a discussion on this separately.

For now, I'm perfectly OK with adding a note to the top of the page that "hey, pipenv hasn't had an excellent maintainance record in 2018-2019, and you can find more details here" and adding a link to whichever issue got the latest update from the maintainers today (with better phrasing please). If anyone wants to file a PR for this, they're welcome to do so. This is much "lower effort and effect" change that should address most of the concerns raised.

Other than that, I think it'd be a good idea to have a dedicated discussion + documenting what the approach / strategy for the various parts of the documentation is.

[xcdr]

For ordinary developers like me, this situation in Python ecosystem is very problematic.

For Ruby standard is RVM and RubyGems and together works flawless, nobody think about version/venv management because "it just works".

There is virtualenv, venv, pyenv, pipenv, poetry (something else?) and nothing works as should at 100% in term of usability and trouble-freeness (even vscode integration).

I like Python, vscode extension (great work!) but something goes wrong in comparison with Ruby world.

In my opinion as person who isn't regular python developer. There should be one or two described solutions that's verified in battle and works without problems for python version/virtual env/vscode workflow.

[philgyford]

It does feel that the situation is complicated by the wide spectrum of users - from people trying to get their first (and maybe only ever) python script running through to people who spend most of their time making modules distributed via PyPI (or some other extreme of complexity and professionalism that's beyond me).

As a mainly-web-developer, who very rarely releases any packages, I was very pleased when I found pipenv as the "recommended" tool as it did all I needed - virtual envs, managing requirements, making a lock file, handling different python versions, and loading environment variables from a dot file. Brilliant! Although it's a bit slow it's been entirely fine for my needs and I've been oblivious to any of its apparent other drawbacks. Now, however, I'm finding problems that have been fixed for a while but not released, and here we all are...

I echo @xcdr as an "ordinary developer": Much as I've been happy with pipenv I don't mind if the collective drops its and anoints something else as the simple solution (even the old collection of pip, virtualenv, virtualenvwrapper, pyenv, etc). Maybe more advanced, or different kinds of, development need a different tool. Or at least different instructions that suggest alternative tools.

But at the moment I'd have no idea what to recommend to new python developers which, aside from my own frustrations, is a massive drawback to adopting the language.

[xcdr]

I can't imagine recommended solution without Poetry (or Pipenv?) because I have several Python project where I need to quickly start deterministic environment. I think that should be described simple setup with venv + pip (it already works but is too simple ) and equivalent to RVM/RubyGems such as Poetry + Pyenv which is is compliant with PEP-517 and actively maintained but not work well with vscode extension (but as I know should work after extension refactoring).

Pipenv is dead at this moment and have to many stupid bugs, especially with WSL (when Windows and Linux python coexists).

[uranusjr]

May I recommend the discussion be moved to another thread, potentially another venue? The topic of recent messages has drifted to become much wider than the title indicates, and it’s terrible for discovery, causing the dialog somewhat into a feedback loop, since only those who had commented know its existence and respond.

[ncoghlan]

While the long delay between pipenv releases is genuinely problematic, there isn't any other comparable tool that both acknowledges that app development and library development are distinct activities, and also prioritises app development over library development (other than the lower level pip-tools project).

The status quo definitely isn't OK though, so I'll be talking to @techalchemy and @brainwane regarding ways we can get the pipenv release process back on track.

[pfmoore]

I don't think this issue should have been closed. @ncoghlan please consider re-opening it. The fact that pipenv is the only tool that fits this use case doesn't mean that it should necessarily be recommended. I'm sympathetic to the process problems pipenv has had, but at the moment, I think that recommending a tool that is widely viewed as unmaintained and problematic isn't the right stance to take.

I think the current situation is awkward as the packaging guide is losing credibility in this area due to the knock-on effects of the pipenv project's problems¹, and I think we should acknowledge that, and address it.

Maybe the simplest thing would be to just remove the "Application dependency management" section until the pipenv project is back on track? Then, when things are more stable, we can discuss whether pipenv is the correct thing to recommend, how poetry fits into the picture and why it addresses a different use case, and whether it too deserves a section, etc.

¹ I actually think pipenv's credibility is still suffering due to the initial over-enthusiastic marketing of the project. Taking a more cautious approach here has the added benefit of demonstrating that we've learned the lesson from that experience.

[pradyunsg]

I second everything that @pfmoore says and, quoting myself from earlier in this issue discussion:

I'm perfectly OK with adding a note to the top of the page that "hey, pipenv hasn't had an excellent maintainance record in 2018-2019, and you can find more details here" and adding a link to whichever issue got the latest update from the maintainers today (with better phrasing please). [snip] This is much "lower effort and effect" change that should address most of the concerns raised.

[layday]

Poetry can lock dependencies, can install dependencies from a lock file and can package "applications" into "libraries" while maintaing the same dependency specification format. I would suggest some people here are experiencing NIH-itis by continuing to expend time and resources into a moribund project in the presence of a more capable alternative. What I would like to see is for the lock file format to be standardised so that pip is able to consume lock files produced by Poetry or [insert tool]. This would alleviate the lock-in concern expressed above.

[pfmoore]

@layday See this discussion about standardising a lock file format.

Whether poetry is a suitable alternative to pipenv, I'm not 100% sure (it didn't fit my use case when I looked). But I think that's a separate discussion. I'm happy to have that discussion (in an appropriate forum) but I don't want to take this discussion off-topic.

[layday]

Fair enough, and thanks for the link.

[brainwane]

Thanks to @ncoghlan for nudging and helping pipenv get back on track; Dan just posted an email to distutils-sig about work towards a new release this month. Currently it sounds like there's a pipenv maintenance gap and @techalchemy wants to work to grow more contributors and co-maintainers to close that gap.

I agree with @pfmoore and @pradyunsg that we need to re-evaluate the current way pipenv is recommended in the PyPUG, and so, Nick, I'd like for you to consider responding to these comments and possibly reopening this issue.

[ncoghlan]

@brainwane @pfmoore @pradyunsg Fair points. I've posted https://github.com/pypa/packaging.python.org/pull/715/files as a proposal to express the caveats on the status quo, pending the pipenv release this month, and at least one timely maintenance release after that.

That PR also adds a reference to poetry at the top of the dependency management guide (in addition to the existing one at the end), as poetry is a genuinely good tool for anyone willing to accept the "All Python applications must be installable as Python components" constraint. I (and other PyPUG editors) just consider that constraint to limit the potential audience of the tool too much for a general tutorial.

[pfmoore]

#715 looks like a reasonable interim measure. But it also exposes the key issue to me quite nicely - the focus of the document is framed in such a way that pipenv isn't so much "recommended" as "the only option". I'd argue that a better framing would be

pipenv - handles all types of project equally well
poetry - expects projects to be structured like a Python package
hatch - broader project management features (also requires package structure?)

IMO, it would be better to have a discussion document explaining this, and clarifying the factors that people should consider when making a choice. One of those decisions would clearly be "do you want to structure your project like a Python package?" If you don't, then pipenv or rolling your own process are the only two options, really. And in that context, saying so is fine.

Having a pipenv tutorial as a separate document is then more reasonable. Although should we not leave that to the pipenv docs, or invite poetry and hatch to contribute their own tutorials to the packaging guide as well?

[ncoghlan]

Thea and my intent with the original framing is that the tutorials should be structured for folks that don't really know what they want yet, they just want to get their application up and running (and the expectation is that that first application will be a web app, or something quite similar, such as a CLI or GUI app that only ever gets run locally).

However, for folks with more experience, especially folks publishing components for other Python developers to use (rather than standalone applications), there are more options out there, so we had the section at the end highlighting some of those options.

So the idea of the dependency management tutorial is "get folks up and running with a default option that lets them use other Python components quickly, but be clear that it's OK for them to revisit that choice later if the default recommendation isn't working out for their specific situation".

The specific choice of pipenv as the default was a matter of not wanting to have any part of the "Prepare a Python project for publication" process as a pre-requisite for consuming projects from PyPI (which means ruling out anything that relies on having a pyproject.toml file already).

Edit: in addition to the practical consideration of ensuring the publication tutorial could be after the dependency management one, there was also the philosophical one of wanting to keep "application development" clearly separated from "library development", even though there are some applications that blur the distinction (either because they're developer tools, or because they're written as very thin wrappers around a library module)

[pradyunsg]

With #715 merged, I do think this issue can be closed now.

FWIW, I spent a few hours yesterday and today; to help unblock pypa/pipenv#4169 which, as per my understanding, was the last blocker for the next pipenv release.

[JBKahn]

Even if the this release happens, I still think the guide should be more impartial about which solution to use. Both tools have their drawbacks and it would be best to provide both and not pick favorites. Although I suppose you can take that with a grain of salt from someone who prefers using poetry both for package and application development. Pipenv already benefits from being pypa/pipenv is a way the competition cannot.

[pfmoore]

I think that #715 addresses this issue, and as such this can be closed.

However, see my comment above:

Then, when things are more stable, we can discuss whether pipenv is the correct thing to recommend, how poetry fits into the picture and why it addresses a different use case, and whether it too deserves a section, etc.

Very little of the discussion on this issue was actually about the maintenance status of pipenv, more about issues such as how alternatives should be presented, whether recommending pipenv is the right choice in the first place, etc. Those issues remain, but I think someone should open a new issue, with a concrete alternative proposal to start the discussion, rather than just continue debating on this issue.

[sdispater]

Author of Poetry here :-)

poetry - expects projects to be structured like a Python package

Poetry does not expect anything regarding the structure of your Python project. Some commands (like new, build and publish) are directed to people who want to build packages and publish them, but the rest of the commands do not assume anything in regards to the project structure I( use it myself for Python applications that do not follow a standard package structure).

That's a misconception I have seen a lot coming from some people in the PyPA so I thought I would chime in to correct that misconception.

[pfmoore]

That's a misconception I have seen a lot coming from some people in the PyPA so I thought I would chime in to correct that misconception.

Thank you. It does seem to be common, and if it's not true then it adds weight to the argument that poetry should be considered alongside pipenv. (It may or may not be true that pipenv still ends up "recommended", but I'd rather we do that on a proper assessment rather than on a misconception).

This really isn't the right place to have this conversation, but in the absence of anywhere better, could you explain (or direct me to documentation) how one would go about starting a non-library project in Poetry? Suppose I wanted to write a script that did some data analysis. So I start with an empty directory, write the first part of my script and want to add pandas as a dependency. In pipenv, this would be (roughly, it's a while since I did this :-)):

• Create directory
• Create script.py
• pipenv install pandas
• pipenv run python script.py

That's basically the first few steps in the tutorial. What would be the poetry equivalent?

[JBKahn]

I think this would be the equivalent set of steps

• Create directory
• Create script.py
• poetry init -n -q
• poetry add pandas
• poetry run python script.py

where poetry init -n -q is the key step here, it creates a pyproject.toml like so:

[tool.poetry]
name = "<directory_name>"
version = "0.1.0"
description = ""
authors = ["<my git name/email>"]

[tool.poetry.dependencies]
python = "^<my python version>"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

[pradyunsg]

Closing since pipenv's maintenance situation has improved, and #729 removed the caveat that was added in #715.

[pradyunsg]

I agree with @pfmoore about what most of the discussion here has been:

more about issues such as how alternatives should be presented, whether recommending pipenv is the right choice in the first place, etc. Those issues remain, but I think someone should open a new issue, with a concrete alternative proposal to start the discussion, rather than just continue debating on this issue.

If someone wants us to move forward on this front, I suggest opening a new issue for that discussion, and not discussing here.

[astrojuanlu]

Done: #912