Simpler UX for non-package (data science) projects

[gjoseph92]

• I have searched the issues of this repo and believe that this is not a duplicate. Kind of a duplicate of Support for application-first, library-first, and non-package modes that change Poetry defaults #1132 and Clarification on using poetry with non-package non-publish code #1148, but would like to more discuss the UI than the technical limitations.
• I have searched the documentation and believe that my question is not covered.

Feature Request

Since pipenv is dead (pypa/pipenv#4058), I've been using Poetry. It's a great tool, and better than pipenv—but the UX is oriented more specifically towards creating new Python packages, not managing multiple Python projects in general. When working on data science projects, which are not and never will become Python packages, I'd prefer a less-opinionated interface.

In Pipenv, I like that creating a new environment is minimalist and simple:

$ mkdir new-project
$ pipenv install jupyterlab ipython numpy

With poetry, I have to read the docs more carefully to figure out poetry init vs poetry new, then run poetry init and hit Enter a bunch of times to bypass the pyproject.toml metadata that I don't particularly care about (or is irrelevant, such as the version), type out my packages one-by-one, forget that jupyterlab isn't jupyter-lab, screw it up, ctrl-C, break my shell (#2067), open a new Terminal because I don't know about reset, and do it all again.

poetry init -n is an okay workaround right now, but I usually forget the name of the flag and have to look it up, which is annoying.

If poetry is becoming the primary tool for Python environment and dependency management, it's worth thinking about the UX for data science, since that's such a common reason to use Python (and need to manage a bunch of per-project environments) these days.

A few ideas:

As discussed in #1132, pyproject.toml not requiring a version or name is probably a prerequisite to all of these.

• poetry add creates a pyproject.toml in the current directory, if not already within a poetry project (similar to pipenv semantics)
• poetry init is non-interactive by default
• poetry new only creates a directory and pyproject.toml by default, poetry new --package would add the boilerplate scaffolding of README, tests, package directory, etc.

This doesn't address making the simple case simple and easy to document to beginners, though.

Perhaps a larger change could be moving the current semantics under a poetry package command group (poetry package new, poetry package init), and having poetry init/new be for creating non-package "data science" projects.

My (unsubstantiated) guess is that people create new Python projects way more frequently than they create new Python packages. How many times a week do you make a new package? (For me, very rarely.) How many times a week to you make an isolated data science project to try things out? (For me, nearly every day.) To me, this justifies making non-package projects the primary interface, and package-based projects more of a special case. But I recognize that that changes the current semantics and goals of Poetry a fair amount.

Also, happy to move this to a comment on #1132 if you do think it's a duplicate issue.

[thernstig]

I think this is a great feature reuqest, and have been thinking the same thing for a long while. It was one of the main reasons I did not move away from Pipenv (until now). We can probably attribute Pipenv's previous success to many things, but the main use of Pipenv was to "install packages/projects" and not "create packages". I am sure that attributed some to it success because it made package management easier for the Python ecosystem.

This is also an unsubstantiated guess but I bet the majority of Python users are looking for a packet/project manager, not a package creation tool when looking at Pipenv. Thus I feel like @gjoseph92 is right on point.

[danmur]

I personally don't see much difference between a project and a package. All of my works with poetry are 'projects', but they need to be deployed, and to be deployed they need to be packaged (in the general sense), and specifically a Python package has been a good way to do that for me.Then again, I use a templating system to make new 'projects' anyway rather than poetry init or poetry new, no entering of anything, so it's kind of moot.

(I don't object to your use case btw, I'm just not 100% on-board with the assertion that it's the most common one)

[thernstig]

@danmur Could you elaborate on why you think that packaging a project is more common? I think it is way off assuming people equate a project == package. I guess that an abundance of more users would know what pip the package installer is and use it, than packaging python projects with e.g. setuptools, wheels.

Many people do not deploy their own project by packaging it. E.g. in a container world you usually install with pip or pipenv --system. I know that most users I've ever come across also do this in VM's or directly into systems where they run production environments.

Again, I'm genuinely curious why you think packaging a Python project would be common, as I think it is not. Maybe it is I who am way off here, hence why I am curious.

[danmur]

I haven't taken a survey so it's just my experience that this is how projects are generally distributed and deployed. Like I said, if this workflow works for you and others I think that's fine. Perhaps we are from different worlds :). I also build container images by installing packages and using a private repo. I see what is happening in your install with pip example but to be honest I can't see why I'd ever want to build a container with a python file or two and not put them in a package so it's easy to reproduce the build, or share the packages with others (or make two builds, one for linux and one for windows).

I think since you're saying these are data science projects that your use case is quite different to mine, and that's fine, but there are lots of other use cases for building and deploying software into production systems and in my experience this is generally done by using the packaging tools available to the language (not just Python). I will add as well that the data scientists at my current job do something similar to what you're describing, but if it ever reaches production it's packaged.

Just my opinion.

[thernstig]

Thanks for your input. I suppose we are just from different worlds as you say 😃 I don't work in data projects myself, it was the original poster that did.

[0az]

As a Data Science student who's been using Poetry for projects (without packaging) and packaged projects [0], here's my 2¢.

Given that it's late, I'll skip most of what I'd like to say and focus on one point: documentation.

Poetry has a lot of space to improve, documentation-wise. This isn't just online -- I stumbled upon this issue while looking for how to use "package data" -- but also in the executable itself. Specifically, the errors are confusing.

As an example, when trying to find information on poetry run, I opened Kitty, typed in poetry run --help, and ended up with this:

~> poetry run --help

[RuntimeError]
Poetry could not find a pyproject.toml file in /Users/az or its parents

Now, this is a case of "whoops, wrong command", but it also illustrates another important point: Poetry currently doesn't fail educationally.

Contrast cargo, which is often cited as a gold standard:

The same command, but using Cargo

~> cargo run --help
cargo-run 
Run the main binary of the local package (src/main.rs)

USAGE:
    cargo run [OPTIONS] [--] [args]...
[...]
~> cargo help run
cargo-run 
Run the main binary of the local package (src/main.rs)
[...]

This is but one unrepresentative example, but it does show that there is space for Python's ecosystem to improve.

---

* `poetry add` creates a `pyproject.toml` in the current directory, if not already within a poetry project (similar to pipenv semantics)

I'm actually against this, because I've accidentally run yarn add $package / pipenv install $package in the wrong place enough times, and cleaned up the aftermath enough times, that I'd prefer a helpful error message instead.

* `poetry init` is non-interactive by default

I don't hold any particular opinions here, other than bouncing off the interactive deps tool [1], but cargo init is non-interactive.

* `poetry new` only creates a directory and `pyproject.toml` by default, `poetry new --package` would add the boilerplate scaffolding of README, tests, package directory, etc.

I could get behind this. Though, at this point, why not have a poetry create analogous to create-react-app?

---

Could you elaborate on why you think that packaging a project is more common? I think it is way off assuming people equate a project == package. I guess that an abundance of more users would know what pip the package installer is and use it, than packaging python projects with e.g. setuptools, wheels.

Another point is this: regardless of if packaging is more common or not, should Poetry (and the broader ecosystem) take an opinionated stance of "Yes, all nontrivial Python code should be packaged"?

Rust, as a compiled language, takes that stance. Python probably shouldn't make that declaration for all code -- one-off scripts are an easy counterexample. But should all nontrivial projects see packaging? If so, then packaging needs to be as frictionless as possible.

---

[0] This is for a variety of reasons, including

• It's just easier to use poetry install && poetry run python -m $package.
• poetry shell is easier to remember than fish -c ...
• venv has activate/deactivate, which provide a much worse UX because of the lack of separation.

[1] The interface didn't give a good way to cancel, I think.

Testing the interface:

Search for package to add (or leave blank to continue): requests
Found 20 packages matching requests

Enter package # to add, or the complete package name if it is not listed: 
 [0] requests
 [1] requests3
 [2] requests2
 [3] lambda_requests
 [4] requests-transition
 [5] micropython-requests
 [6] tornado-requests
 [7] requests_handler
 [8] requests-onshape
 [9] requests-lb
 > ^M
'NoneType' object has no attribute 'replace'

There's also an inconsistency between poetry new and poetry init here: only the latter uses this prompt interface. I'd argue that the expected behavior is the other way around, if not both.

---

I'll probably have some new input later, once this quarter finishes and I have time to actually organize my thoughts.

[epogrebnyak]

Sad there is no progress on no-package use, I think it is quite helpful!

My usecase is a documentation project with jupyter-book and sphinx, where I need to install dependecies in virtual environment. I want to list these deps in pyproject.toml and after poetry install use something like poetry run jupyter-book build . My work-around is to create a project with poetry init, but since there is no actual package, poetry install will issue a ton of warnings.

I could use pipenv, venv and probably conda, but prefer poetry for this (I had to explore a conflict of dependecies that was never apparent in requirements.txt, but rightfllly rejected by poetry).

Ideally there is some flag like poetry init --no-package that allows poetry add/install/run on project pyproject.toml.

[danmur]

I don't quite get the issue @epogrebnyak. I can create a new Poetry project using poetry init, with no package files (just a pyproject.toml and poetry.lock), and poetry install works fine for me. I can also install jupyter and poetry run that. What errors are you getting exactly?

[epogrebnyak]

@danmur - you are right, frsej init-install sequence works well. I had Cannot install package error, but can't replicate it now, must be some editing or corruption of pyproject.toml. Can confirm fresh init and install without package work fine, just as you say, so no need for --no-package flag.

[neersighted]

Duplicate #1132

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.