Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fail with "Error: No such file or directory (os error 2)" #553

Open
ikus060 opened this issue Feb 20, 2023 · 9 comments · May be fixed by #562
Open

Fail with "Error: No such file or directory (os error 2)" #553

ikus060 opened this issue Feb 20, 2023 · 9 comments · May be fixed by #562

Comments

@ikus060
Copy link

ikus060 commented Feb 20, 2023

I'm trying to use py-spy on a python process already running py-spy dump -p 219292.

That process make use of user name space. That probably explain why py-spy is failing.

  1. I'm question if it's possible to use py-spy cross namespace ?
  2. Would it be possible to handle the exception and continue the "spying" event if some file are not found ?

Running with strace:

# strace -e trace=open py-spy dump -p 219292
open("/proc/219292/maps", O_RDONLY|O_CLOEXEC) = 3
open("/proc/219292/exe", O_RDONLY|O_CLOEXEC) = 3
open("/proc/219292/root/tmp/minarca-jail-ku6ln2cg/opt/rdiff-backup-2.0/libpython3.7m.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
Error: No such file or directory (os error 2)
+++ exited with 1 +++

/tmp/minarca-jail-ku6ln2cg/opt/rdiff-backup-2.0/libpython3.7m.so.1.0 doesn't exists. It's probably only mounted in the namespace

@Jongy
Copy link
Contributor

Jongy commented Feb 20, 2023

py-spy accesses the files in the mount (mount, not user) namespace of the target process (that's the /proc/219292/root prefix). It works just fine for containerized processes.

In this case py-spy decided that libpython.so is needed (because the Python is built with --enable-shared, and thus the core of Python is implemented in libpython.so, not in the executable). The file is indeed needed.
Any chance the file is deleted? Can you please check inside the container?

@ikus060
Copy link
Author

ikus060 commented Feb 21, 2023

Not exists "/proc/219292/root/tmp/minarca-jail-ku6ln2cg/opt/rdiff-backup-2.0/libpython3.7m.so.1.0"
Exists "/proc/282195/root/opt/rdiff-backup-2.0/libpython3.7m.so.1.0"

"/tmp/minarca-jail-ku6ln2cg" the the root of the container

Looking at /proc/:

exe is a symlink to the executable.
exe -> /tmp/minarca-jail-g5905ukq/opt/rdiff-backup-2.0/rdiff-backup

root is also a symlink to a path.
root -> /tmp/minarca-jail-g5905ukq

So the real file is

sharelib = /proc/219292/root/tmp/minarca-jail-ku6ln2cg/opt/rdiff-backup-2.0/libpython3.7m.so.1.0
root ="/tmp/minarca-jail-g5905ukq"
/proc/282195/root/ + (sharelib[len(root):])

@ikus060
Copy link
Author

ikus060 commented Mar 1, 2023

@Jongy Is the information provided help you write a fix ?

@Jongy
Copy link
Contributor

Jongy commented Mar 6, 2023

Not quite yet but we're getting there.

py-spy attempts to access the path based on what it reads from /proc/pid/maps which contains the absolute path of the libpython, so no symlink resolving mistakes should / could occur. What might happens here, based on root pointing to non /, is py-spy mishandling a chrooted container. Possibly the correct handling is to remove the prefix of root from the path, if the maps file gives it when reading maps. I was under the impression it will not.

Let's try to walk it one by one - to have all data. Please run all these commands so we can see the exact files & directories along the way:

ls -ld /proc/219292/cwd/
ls -ld /proc/219292/root/
ls -ld /proc/219292/root/opt/rdiff-backup-2.0/
ls -ld /proc/219292/root/opt/rdiff-backup-2.0/libpython3.7m.so.1.0
ls -ld /proc/219292/root/tmp/
ls -ld /proc/219292/root/tmp/minarca-jail-ku6ln2cg/
ls -ld /proc/219292/root/tmp/minarca-jail-ku6ln2cg/opt/
ls -ld /proc/219292/root/tmp/minarca-jail-ku6ln2cg/opt/rdiff-backup-2.0/
ls -ld /proc/219292/root/tmp/minarca-jail-ku6ln2cg/opt/rdiff-backup-2.0/libpython3.7m.so.1.0

replace 219292 with the PID of the process, if it changed.

@ikus060
Copy link
Author

ikus060 commented Mar 6, 2023

# ls -ld /proc/982626/cwd/
    drwx------ 10 minarca minarca 10 Mar  6 07:40 /proc/982626/cwd/
# ls -ld /proc/982626/root/
    drwx------ 10 minarca minarca 10 Mar  6 07:40 /proc/982626/root/
# ls -ld /proc/982626/root/opt/rdiff-backup-2.0/
    drwxr-xr-x 5 root root 22 Jan 19 16:36 /proc/982626/root/opt/rdiff-backup-2.0/
# ls -ld /proc/982626/root/opt/rdiff-backup-2.0/libpython3.7m.so.1.0
    -rw-r--r-- 1 root root 3437576 Dec 20 06:53 /proc/982626/root/opt/rdiff-backup-2.0/libpython3.7m.so.1.0
# ls -ld /proc/982626/root/tmp/
    ls: cannot access '/proc/982626/root/tmp/': No such file or directory
# ls -ld /proc/982626/root/tmp/minarca-jail-*/
    ls: cannot access '/proc/982626/root/tmp/minarca-jail-*/': No such file or directory
# ls -ld /proc/982626/root/tmp/minarca-jail-*/opt/
    ls: cannot access '/proc/982626/root/tmp/minarca-jail-*/opt/': No such file or directory
# ls -ld /proc/982626/root/tmp/minarca-jail-*/opt/rdiff-backup-2.0/
    ls: cannot access '/proc/982626/root/tmp/minarca-jail-*/opt/rdiff-backup-2.0/': No such file or directory
# ls -ld /proc/982626/root/tmp/minarca-jail-*/opt/rdiff-backup-2.0/libpython3.7m.so.1.0
    ls: cannot access '/proc/982626/root/tmp/minarca-jail-*/opt/rdiff-backup-2.0/libpython3.7m.so.1.0': No such file or directory

Here the extra command I also ran:

root@ranculos:~# ls -la /proc/982626/root
lrwxrwxrwx 1 minarca minarca 0 Mar  6 07:42 /proc/982626/root -> /tmp/minarca-jail-okxuc76l

@Jongy
Copy link
Contributor

Jongy commented Mar 19, 2023

I managed to reproduce the problem locally. Interesting.

Possibly the correct handling is to remove the prefix of root from the path, if the maps file gives it when reading maps. I was under the impression it will not.

This is the fix that needs to be implemented in py-spy. It can be generalized to - instead of stripping the / that is usually the readlink(/proc/pid/root), you need to strip the entire readlink result, because that part is included in the /proc/pid/maps when viewed outside of the process.

I might have the time to fix it in py-spy soon but cannot promise.

@Jongy
Copy link
Contributor

Jongy commented Mar 23, 2023

I implemented a fix, pushed a PR - @ikus060 please try building it and see if it solves your problem :) I explained in the PR the steps I took and the way I see that it solved the issue.

@ikus060
Copy link
Author

ikus060 commented Apr 3, 2023

Ho, I forgot about this issue.
Will try to test this week

@ikus060
Copy link
Author

ikus060 commented Dec 13, 2023

Sorry for the long delay, this issue slip my mind and I only need py-spy once in a while.

Thans changes you made seams accurate with my analysis too and is working for my requirement.

Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants