Native namespace package does not install properly when specifying a target directory (-t option)

[furechan]

Description

I cannot make native namespace packages to install properly when specifying a target directory (-t option).

This is important to me as I need to use the the target option to install some packages for an AWS Lambda.

This looks like a very old issue #1924. However I cannot find, after extensive search, any clarification anywhere and the case was closed without apparent resolution.

To confirm I tried it with the pypa samples from pypa/sample-namespace-packages and failed to install both native packages (pkg_a & pkg_b) with an error stating that the directory already exists.

Please note I tried this both on Windows and Linux and the other packaging methods 'pkgutil' and 'pkg_resources' had the same exact problem.

Thank You

Expected behavior

Both packages pkg_a & pkg_b should install in the same namespace directory.

pip version

21.1.3

Python version

3.8.5

OS

Windows/Linux

How to Reproduce

git clone https://github.com/pypa/sample-namespace-packages

cd sample-namespace-packages/native
mkdir temp

pip install ./pkg_a -t temp
pip install ./pkg_b -t temp

Output

The second install fails with a message like:

WARNING: Target directory .../sample-namespace-packages/native/temp/example_pkg already exists. Specify --upgrade to force replacement.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[DiddiLeija]

The warning is suggesting you to use --upgrade. Try to run on both commands:

pip install --upgrade ./pkg_a -t temp
pip install --upgrade ./pkg_b -t temp

Can you try it and tell us what happened?

[furechan]

With --upgrade both packages seem to install fine, but the second install actually replaces the first. Still only one one package installed (pkg_b).

tree shows

├───example_pkg
│ └───b
├───example_pkg_a-1.dist-info
└───example_pkg_b-1.dist-info

[DiddiLeija]

It is because both packages are being installed on temp/example_pkg. Seems like both packages inside the folders are named as the same: example_pkg (even when the containers have different names [1]). So pip will consider them as the same package, and upgrade pkg_a with pkg_b.

And you can reproduce it as follows: If pkg_a installs first, it will be replaced by pkg_b. But if you install pkg_b first, it must be replaced by pkg_a.

If you want both packages on temp/, maybe you must create 2 folders and separate the packages with the -t option.

[1]: To explain this, imagine a package named hello_world. Imagine that I copy it on 2 folders: hw_1 and hw_2. If I install them with the same target directory, pip will look inside the packages, and understand them as the same (because they're actually named hello_world).

[furechan]

I do not understand!

These are two distinct packages with different names.

example_pkg/a
example_pkg/b

example_pkg in this case is not even a package as it is never installed on its own.

When doing a local install, pip rightly sees these two as distinct, and they get installed alongside each other which is obviously the intent of using example_pkg as a namespace.

One of the ideas of using a target directory is to isolate/segregate a part of a distribution. Instead of installing the packages in [sitepackages] I need them in a folder that is imposed to me by the runtime. I have had no problem installing large distributions of packages in the same target directory, mainly in the context of AWS Lambda Layers. For namespace packages however here it fails. It seems to think example_pkg is a package that needs to be replaced, but there is no such package!

Your suggestion that I should create a separate target folder for each package I install is not only extremely cumbersome but it also means I would have to hack the import system to fudge the paths ...

[DiddiLeija]

When doing a local install, pip rightly sees these two as distinct, and they get installed alongside each other which is obviously the intent of using example_pkg as a namespace.

Well, the container can be named as different, but the inside "folders" are named as the same: example_pkg. Even when they are different, pip notices that the example_pkg destination folder is taken by both installs, so it writes example_pkg/b to replace example_pkg/a.

---

It has a simple logic: Imagine an example1 package with these files:

|-example
|-------hello

and an example2 with these others:

|-example
|-------world

When these are installed, will be installed into example. If pip finds that the folder already exists and you are using --upgrade, it will delete the existing one and replace it by the new one.

I believe that's why pkg_a is being replaced by pkg_b, because both are being installed in temp/example_pkg. It must be as follows:

1. pip install --upgrade ./pkg_a -t temp
2. pip creates example_pkg and example_pkg/a (with all their files)
3. pip install --upgrade ./pkg_b -t temp
4. pip deletes the whole example_pkg
5. pip creates example_pkg and example_pkg/b

---

Anyway, I'm not the best person to explain this 😞. Maybe another pip commiter can take a look on this if you don't understand.

[DiddiLeija]

Any questions or commentaries about this?

[pfmoore]

I think the OP's concern is legitimate. But both namespace packages and -t are "a bit weird"™ and in this case interact badly. I'm not sure whether I'd consider this as "not supported" or something we should/could fix, though, and I don't really have the time right now to do any deeper investigation.

[DiddiLeija]

But both namespace packages and -t are "a bit weird"™ and in this case interact badly.

Yeah, that's what I wanted to say since we started to discuss the issue 😆. Thanks @pfmoore.

I'm not sure whether I'd consider this as "not supported" or something we should/could fix, and I don't really have the time right now to do any deeper investigation.

I think we need more eyes on this. Is there any related label for this?

[furechan]

Hi @DiddiLeija,

Thanks for trying! But sorry to say I have trouble getting at your explanations...

First of all there is no need to use --upgrade as these are two different packages. So this already is wrong.

What I am asking is why pip cannot do the same thing in the [target] folder as it does in the [sitepackages] folder? This works fine for all packages except namespace packages.

The wanted tree structure is easy enough to achieve:

target/example_pkg/a
target/example_pkg/b

In the same way we get with a local install:

site-packages/example_pkg/a
site-packages/example_pkg/b

There is nothing magic about it. I could do the copies by hand but I am hoping pip can do the install on my behalf.

hi @pfmoore,

I understand there are some inner workings that may be complex. However I want to stress how important namespace packages are as a modern distribution pattern and also in the context of customized cloud installs.

Thanks

[DiddiLeija]

Thanks for trying! But sorry to say I have trouble getting at your explanations...

Yeah... The people doesn't know me as an "explanation expert" 😅. Don't worry.

I understand there are some inner workings that may be complex. However I want to stress how important namespace packages are as a modern distribution pattern and also in the context of customized cloud installs.

I understand your concerns. Maybe if we add a S: needs eyes label could help us to find a pip commiter who can help you better than me.

[furechan]

Hi @DiddiLeija,

Thank you!

I had a look at the source code to try and better understand ...

When using a target directory pip first creates a temporary environment where it installs all the packages in a typical fashion and then copies it back into the target directory, folder by folder. When copying any folder it raises a warning if the folder already exists in the target, regardless of the type of folder, hence the message to use --upgrade. With the --upgrade flag the folder is first erased before copy, without it the copy is skipped. This works for differential installs on most packages but not for namespace packages. For namespace packages the --upgrade flag is no remedy and the warning misleading.

One way to make this work reliably, it seems, is to make sure related namespace packages are always bundled in the same install.

pip install ./pkg_a ./pkg_b -t target

This works.

Not sure where to take this next. It'd probably be nice to have some clarification or documentation somewhere ...

Thanks again

[uranusjr]

I’m going to just classify it as a bug. Not sure I (or another maintainer) will be able to work on this in the newer future, so any contribution is welcomed.

[DiddiLeija]

Sounds fine.

Unfortunately, I will be really busy for these months, so I don't think I would be able to check it. But maybe someone else can take a closer look on this issue.

[pradyunsg]

Based on a trip down the issue tracker history (notably #4389), I don't think multiple-runs-with---target was ever an intended use case. Using --prefix is a much better fit for such uses cases, and I recommend using that.

I've relabelled this but I'm very much inclined to close this as "no, don't do that; use --prefix instead". If someone wants to add defenses to --target to prevent such usage, let's file a separate issue for that.

[furechan]

I agree that the target option does not seem to be designed for multiple runs... The warning message makes it seem so though.

However, the use of --prefix does not replace --target exactly here, as the files get installed in a hierarchy several levels below. With target you can simply add the path to the import path and it will work. With prefix you cannot. Also one of the use cases is that the target directory is imposed on the distribution mechanism (eg AWS Lambdas), on a system where you cannot alter the import path...

Thanks

[pradyunsg]

Well, if you control the Python interpreter and have changed import paths to use --target, then you can also use --prefix instead and use the locations it installs to.

As for the provider of the execution environment being a vendor, I suggest that you take that up with the relevant vendors that they're using the wrong option for the task at hand and ask on their support channels for help here.

[furechan]

Got it. Thank You!

[jaraco]

In #4116, I expressed: This issue is still needed and still a blocker to remove the suboptimal hack in pip-run that attempts to do the same thing (jaraco/pip-run#51). Is there a reason the duplicate is left open while this report is closed? It makes it more difficult to track if the issue is a moving target.

In that issue and also #5178, I express a use-case where it would be useful if --target could function as a "supplemental" install to the current environment.

One disadvantage to using prefix is that it creates a lot of unnecessary structure and variability (bin/Scripts directory, python3.NN directory), especially when all the installer wishes to do is install the packages for placing them on the PYTHONPATH. Those extra paths add unwanted noise to the installed environment and length to the package paths.

I vaguely recall that there were other problems with attempting to use --prefix, but if so, I can't find them now.

The issues I have are unrelated to namespace packages, but are instead related to desired behaviors around installing with --target but honoring already-installed packages.

[pfmoore]

The issues I have are unrelated to namespace packages, but are instead related to desired behaviors around installing with --target but honoring already-installed packages.

I have a long-term goal of making --target better integrated with the other "install schemes" (which may, for compatibility reasons, mean adding a new option to replace --target). That change may help here - I'm not sure because I haven't really read through the requirement described here. But I can't offer any timescale on when that will happen.

[dwoz]

The issues I have are unrelated to namespace packages, but are instead related to desired behaviors around installing with --target but honoring already-installed packages.

I have a long-term goal of making --target better integrated with the other "install schemes" (which may, for compatibility reasons, mean adding a new option to replace --target). That change may help here - I'm not sure because I haven't really read through the requirement described here. But I can't offer any timescale on when that will happen.

I took a stab at this in #12524

[pfmoore]

I don’t think that PR goes in the direction I want. We should never be simply dumping a new install over an old one. Instead, we should properly support the normal uninstall/reinstall mechanism that upgrades use.

[dwoz]

I don’t think that PR goes in the direction I want. We should never be simply dumping a new install over an old one. Instead, we should properly support the normal uninstall/reinstall mechanism that upgrades use.

I agree that dumping a new install over an old one isn't ideal by any means. Having said that, my understanding is that --prefix currently suffers from the same issue if the directory is not in the python path. Until such time that the upgrade logic for directories not in the path is solved. Would updating --target to work similarly as --prefix makes sense?

[dwoz]

@pfmoore What about adding the value of --prefix and --target to the path?

In my ideal world --target would be deprecated and replaced --prefix, --libs and --scripts which I believed would be more in-line with what python does when building from source.

[dwoz]

@pfmoore I've modified my PR, nothing will get overwritten without --upgrade