Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Workaround missing main symbol in Python >= 3.9 #937

Merged
merged 1 commit into from Oct 25, 2021
Merged

Conversation

tkf
Copy link
Member

@tkf tkf commented Oct 25, 2021

This patch changes Libdl.dlsym(proc_handle, :main) to Libdl.dlsym(proc_handle, :_start) to fix JuliaPy/pyjulia#459

In Python >= 3.9, main cannot be accessed through the process handle (dlopen(NULL, ...) ):

$ python3.9 -c 'import ctypes; ctypes.CDLL(None).main'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/arakaki/.pyenv/versions/3.9.7/lib/python3.9/ctypes/__init__.py", line 387, in __getattr__
    func = self.__getitem__(name)
  File "/home/arakaki/.pyenv/versions/3.9.7/lib/python3.9/ctypes/__init__.py", line 392, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /home/arakaki/.pyenv/versions/3.9.7/bin/python3.9: undefined symbol: main

while it has been possible for < 3.9.

$ python3.8 -c 'import ctypes; ctypes.CDLL(None).main'  # no error

I'm not sure why this is the case since python3.9 binary does seem to have main:

$ objdump -dC ~/.pyenv/versions/3.9.7/bin/python3.9 | grep -e '<main>' -A5
0000000000000720 <main>:
 720:   e9 db ff ff ff          jmpq   700 <Py_BytesMain@plt>
 725:   66 2e 0f 1f 84 00 00    nopw   %cs:0x0(%rax,%rax,1)
 72c:   00 00 00
 72f:   90                      nop

--
 74d:   48 8d 3d cc ff ff ff    lea    -0x34(%rip),%rdi        # 720 <main>
 754:   ff 15 86 08 20 00       callq  *0x200886(%rip)        # 200fe0 <__libc_start_main@GLIBC_2.2.5>
 75a:   f4                      hlt
 75b:   0f 1f 44 00 00          nopl   0x0(%rax,%rax,1)

That said, JuliaPy/pyjulia#459 can now be solved by checking _start instead. (Should we check main first and then fallback to _start?)

A better fix probably is to use Preferences.jl as the build-time option (#835) and get rid of automatic detection. But I wonder if this is good enough for now?

@Keno It'd be great if you can look at this, as you wrote this originally in #419.

@codecov-commenter
Copy link

codecov-commenter commented Oct 25, 2021

Codecov Report

Merging #937 (b787aba) into master (4e5c12c) will not change coverage.
The diff coverage is 0.00%.

Impacted file tree graph

@@           Coverage Diff           @@
##           master     #937   +/-   ##
=======================================
  Coverage   68.24%   68.24%           
=======================================
  Files          20       20           
  Lines        2012     2012           
=======================================
  Hits         1373     1373           
  Misses        639      639           
Flag Coverage Δ
unittests 68.24% <0.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/startup.jl 53.12% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4e5c12c...b787aba. Read the comment docs.

@Keno
Copy link
Collaborator

Keno commented Oct 25, 2021

Seems reasonable to me. Any symbol should work.

@tkf
Copy link
Member Author

tkf commented Oct 25, 2021

Thanks! By any chance, do you have any idea why this is happening? I'm just puzzled that main is "gone," presumably after it's called via _start.

@Keno
Copy link
Collaborator

Keno commented Oct 25, 2021

Did you check the dynamic symbol table nm -D? They may have decided to hide it.

@tkf
Copy link
Member Author

tkf commented Oct 25, 2021

Indeed main does not show up in python3.9 but it does in python3.8!

$ nm -D ~/.pyenv/versions/3.9.7/bin/python3.9
0000000000201010 B __bss_start
                 w __cxa_finalize
0000000000201000 D __data_start
0000000000201000 W data_start
0000000000201010 D _edata
0000000000201018 B _end
00000000000008b4 T _fini
                 w __gmon_start__
00000000000006d0 T _init
00000000000008c0 R _IO_stdin_used
                 w _ITM_deregisterTMCloneTable
                 w _ITM_registerTMCloneTable
00000000000008b0 T __libc_csu_fini
0000000000000840 T __libc_csu_init
                 U __libc_start_main
                 U Py_BytesMain
0000000000000730 T _start
$ nm -D ~/.pyenv/versions/3.8.12/bin/python3.8
0000000000201010 B __bss_start
                 w __cxa_finalize
0000000000201000 D __data_start
0000000000201000 W data_start
0000000000201010 D _edata
0000000000201018 B _end
00000000000008e4 T _fini
                 w __gmon_start__
0000000000000700 T _init
00000000000008f0 R _IO_stdin_used
                 w _ITM_deregisterTMCloneTable
                 w _ITM_registerTMCloneTable
00000000000008e0 T __libc_csu_fini
0000000000000870 T __libc_csu_init
                 U __libc_start_main
0000000000000750 T main
                 U Py_BytesMain
0000000000000760 T _start

@tkf tkf merged commit 5bea9e1 into JuliaPy:master Oct 25, 2021
@tkf tkf deleted the nomain branch October 25, 2021 02:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

System image compilation and --compiled-modules=no is broken in Python >= 3.9
3 participants