pytorch segfaults when compiling nvidia-cuda-toolkit with lto

[Xephobia]

importing pytorch in python segfaults when compiling nvidia-cuda-toolkit with lto. GDB stacktrace :

gnuwu@anonynowo ~/koboldcpp $ gdb python
GNU gdb (Gentoo 13.2 vanilla) 13.2
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python...
(No debugging symbols found in python)
(gdb) run -c "import torch"
Starting program: /usr/bin/python -c "import torch"
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
process 22178 is executing new program: /usr/bin/python3.11
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x00007fffb7b47cb0 in ?? () from /opt/cuda/lib64/libcublasLt.so.11
(gdb) bt
#0  0x00007fffb7b47cb0 in ?? ()
   from /opt/cuda/lib64/libcublasLt.so.11
#1  0x00007ffff7fcf04e in ?? () from /lib64/ld-linux-x86-64.so.2
#2  0x00007ffff7fcf13c in ?? () from /lib64/ld-linux-x86-64.so.2
#3  0x00007ffff7fcb56e in _dl_catch_exception ()
   from /lib64/ld-linux-x86-64.so.2
#4  0x00007ffff7fd5aa6 in ?? () from /lib64/ld-linux-x86-64.so.2
#5  0x00007ffff7fcb4e1 in _dl_catch_exception ()
   from /lib64/ld-linux-x86-64.so.2
#6  0x00007ffff7fd5e3c in ?? () from /lib64/ld-linux-x86-64.so.2
#7  0x00007ffff784c8fc in ?? () from /lib64/libc.so.6
#8  0x00007ffff7fcb4e1 in _dl_catch_exception ()
   from /lib64/ld-linux-x86-64.so.2
#9  0x00007ffff7fcb603 in ?? () from /lib64/ld-linux-x86-64.so.2
#10 0x00007ffff784c3cf in ?? () from /lib64/libc.so.6
#11 0x00007ffff784c9b1 in dlopen () from /lib64/libc.so.6
#12 0x00007ffff7094607 in ?? ()
   from /usr/lib/python3.11/lib-dynload/_ctypes.cpython-311-x86_64-linux-gnu.so
#13 0x00007ffff7b9a0a8 in ?? () from /usr/lib64/libpython3.11.so.1.0
#14 0x00007ffff7b6c0dc in _PyObject_MakeTpCall ()
   from /usr/lib64/libpython3.11.so.1.0
#15 0x00007ffff7b7d7de in _PyEval_EvalFrameDefault ()
   from /usr/lib64/libpython3.11.so.1.0
#16 0x00007ffff7bae97d in _PyFunction_Vectorcall ()
   from /usr/lib64/libpython3.11.so.1.0
#17 0x00007ffff7bba439 in ?? () from /usr/lib64/libpython3.11.so.1.0
#18 0x00007ffff7b6c07a in _PyObject_MakeTpCall ()
   from /usr/lib64/libpython3.11.so.1.0
#19 0x00007ffff7b7d7de in _PyEval_EvalFrameDefault ()
   from /usr/lib64/libpython3.11.so.1.0
#20 0x00007ffff7b77c59 in PyEval_EvalCode ()
   from /usr/lib64/libpython3.11.so.1.0
#21 0x00007ffff7be38f2 in ?? () from /usr/lib64/libpython3.11.so.1.0
#22 0x00007ffff7b8e6ba in ?? () from /usr/lib64/libpython3.11.so.1.0
#23 0x00007ffff7b84dac in _PyEval_EvalFrameDefault ()
   from /usr/lib64/libpython3.11.so.1.0
#24 0x00007ffff7bae97d in _PyFunction_Vectorcall ()
   from /usr/lib64/libpython3.11.so.1.0
#25 0x00007ffff7b99cd0 in ?? () from /usr/lib64/libpython3.11.so.1.0
#26 0x00007ffff7bd6c34 in PyObject_CallMethodObjArgs ()
   from /usr/lib64/libpython3.11.so.1.0
#27 0x00007ffff7ac2096 in ?? () from /usr/lib64/libpython3.11.so.1.0
#28 0x00007ffff7b8209c in _PyEval_EvalFrameDefault ()
   from /usr/lib64/libpython3.11.so.1.0
#29 0x00007ffff7b77c59 in PyEval_EvalCode ()
   from /usr/lib64/libpython3.11.so.1.0
#30 0x00007ffff7c81da3 in ?? () from /usr/lib64/libpython3.11.so.1.0
#31 0x00007ffff7c7e66a in ?? () from /usr/lib64/libpython3.11.so.1.0
#32 0x00007ffff7c756dd in PyRun_StringFlags ()
   from /usr/lib64/libpython3.11.so.1.0
#33 0x00007ffff7c754ec in PyRun_SimpleStringFlags ()
   from /usr/lib64/libpython3.11.so.1.0
#34 0x00007ffff7c8c325 in Py_RunMain ()
   from /usr/lib64/libpython3.11.so.1.0
#35 0x00007ffff7c5be7b in Py_BytesMain ()
   from /usr/lib64/libpython3.11.so.1.0
#36 0x00007ffff77eda90 in ?? () from /lib64/libc.so.6
#37 0x00007ffff77edb49 in __libc_start_main () from /lib64/libc.so.6