library libshm.dylib is missing

[jskye]

i have basically the same problem but the library libshm.dylib is missing.

dlopen(/usr/local/lib/python3.9/site-packages/torch/_C.cpython-39-darwin.so, 2): Library not loaded: @loader_path/libshm.dylib
  Referenced from: /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_python.dylib
  Reason: image not found

all answers i found to fix that say to install libomp, which i reinstalled but it didnt fix the issue, though for others that seemed to work in previous years.

i cant find the library libshm in any lib folders on my machine. Im wondering if its included somewhere in libomp and how to point pytorch to it, or if it can be downloaded from somewhere else?

i have the current torch version installed 1.11.0 with Python 3.9 on Mac Big Sur.

Found the libshm library required for sharedmemory alloc on linux machines.
Does PyTorch require a bugfix to include this dylib? Or is there any other workaround? @malfet

Originally posted by @jskye in #36941 (comment)

cc @ezyang @gchanan @zou3519 @seemethere @malfet @albanD

[jskye]

Collecting environment information...
PyTorch version: N/A
Is debug build: N/A
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: N/A

OS: macOS 11.5.1 (x86_64)
GCC version: Could not collect
Clang version: 11.0.0 (clang-1100.0.33.16)
CMake version: Could not collect
Libc version: N/A

Python version: 3.9.13 (main, May 24 2022, 21:28:44) [Clang 13.0.0 (clang-1300.0.29.30)] (64-bit runtime)
Python platform: macOS-11.5.1-x86_64-i386-64bit
Is CUDA available: N/A
CUDA runtime version: Could not collect
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] clip-anytorch==2.4.0
[pip3] CoCa-pytorch==0.0.5
[pip3] dalle2-pytorch==0.6.8
[pip3] numpy==1.22.0
[pip3] pytorch-pretrained-biggan==0.1.1
[pip3] rotary-embedding-torch==0.1.5
[pip3] torch==1.11.0
[pip3] torch-fidelity==0.3.0
[pip3] torchaudio==0.11.0
[pip3] torchmetrics==0.9.0
[pip3] torchvision==0.12.0
[pip3] vector-quantize-pytorch==0.6.0
[conda] Could not collect

[albanD]

Hi,
The issue about openmp is unrelated to your issue I think.
Could you double check that you did install pytorch properly? That file should be bundled with the pip package that you download.
In particular, you can run pip show -f torch | grep dylib to show all the shared libraries bundled in the install. And you should see libshm.dylib in there.

[jskye]

pip show -f torch | grep dylib
torch/.dylibs/libiomp5.dylib
torch/lib/libc10.dylib
torch/lib/libiomp5.dylib
torch/lib/libshm.dylib
torch/lib/libtorch.dylib
torch/lib/libtorch_cpu.dylib
torch/lib/libtorch_global_deps.dylib
torch/lib/libtorch_python.dylib

[albanD]

All seem to be in order here.
Can you share more details on what you run to get that error?
Also I would try to uninstall torch (multiple times to make sure you don't have multiple conflicting installs) and once it says that torch is not installed, install it again.

[malfet]

Can you please run otool -l /usr/local/lib/python3.9/site-packages/torch/_C.cpython-39-darwin.so? Also, please type env and share its output (I'm curious if you have some DYLD_ variables there)
And correct me if I'm wrong, you've build Python from source, haven't you?

[vfdev-5]

I have a similar issue with libomp.dylib on Mac OSX 10.14.6 (maybe this OS is too old). I'm installing nightly torch on clean conda environment:

conda create -n vision python=3.9
conda activate vision
pip install --upgrade --pre torch --extra-index-url=https://download.pytorch.org/whl/nightly
pip list | grep torch
> torch             1.13.0.dev20220608
python -c "import torch"
> 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/user/opt/miniconda3/envs/vision/lib/python3.9/site-packages/torch/__init__.py", line 202, in <module>
    from torch._C import *  # noqa: F403
ImportError: dlopen(/Users/user/opt/miniconda3/envs/vision/lib/python3.9/site-packages/torch/_C.cpython-39-darwin.so, 2): Library not loaded: @loader_path/../.dylibs/libomp.dylib
  Referenced from: /Users/user/opt/miniconda3/envs/vision/lib/python3.9/site-packages/torch/lib/libtorch_cpu.dylib
  Reason: no suitable image found.  Did find:
        /Users/user/opt/miniconda3/envs/vision/lib/python3.9/site-packages/torch/lib/../.dylibs/libomp.dylib: cannot load 'libomp.dylib' (load command 0x80000034 is unknown)
        /Users/user/opt/miniconda3/envs/vision/lib/python3.9/site-packages/torch/lib/../.dylibs/libomp.dylib: cannot load 'libomp.dylib' (load command 0x80000034 is unknown)

otool -l /usr/local/lib/python3.9/site-packages/torch/_C.cpython-39-darwin.so

output

/Users/user/opt/miniconda3/envs/vision/lib/python3.9/site-packages/torch/_C.cpython-39-darwin.so:
Mach header
      magic cputype cpusubtype  caps    filetype ncmds sizeofcmds      flags
 0xfeedfacf 16777223          3  0x00           8    17       1328 0x00000085
Load command 0
      cmd LC_SEGMENT_64
  cmdsize 392
  segname __TEXT
   vmaddr 0x0000000000000000
   vmsize 0x0000000000004000
  fileoff 0
 filesize 16384
  maxprot 0x00000005
 initprot 0x00000005
   nsects 4
    flags 0x0
Section
  sectname __text
   segname __TEXT
      addr 0x0000000000003f80
      size 0x000000000000000a
    offset 16256
     align 2^4 (16)
    reloff 0
    nreloc 0
     flags 0x80000400
 reserved1 0
 reserved2 0
Section
  sectname __stubs
   segname __TEXT
      addr 0x0000000000003f8a
      size 0x0000000000000006
    offset 16266
     align 2^1 (2)
    reloff 0
    nreloc 0
     flags 0x80000408
 reserved1 0 (index into indirect symbol table)
 reserved2 6 (size of stubs)
Section
  sectname __stub_helper
   segname __TEXT
      addr 0x0000000000003f90
      size 0x000000000000001a
    offset 16272
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x80000400
 reserved1 0
 reserved2 0
Section
  sectname __unwind_info
   segname __TEXT
      addr 0x0000000000003fac
      size 0x0000000000000048
    offset 16300
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Load command 1
      cmd LC_SEGMENT_64
  cmdsize 312
  segname __DATA
   vmaddr 0x0000000000004000
   vmsize 0x0000000000004000
  fileoff 16384
 filesize 16384
  maxprot 0x00000003
 initprot 0x00000003
   nsects 3
    flags 0x0
Section
  sectname __nl_symbol_ptr
   segname __DATA
      addr 0x0000000000004000
      size 0x0000000000000008
    offset 16384
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000006
 reserved1 1 (index into indirect symbol table)
 reserved2 0
Section
  sectname __got
   segname __DATA
      addr 0x0000000000004008
      size 0x0000000000000008
    offset 16392
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000006
 reserved1 2 (index into indirect symbol table)
 reserved2 0
Section
  sectname __la_symbol_ptr
   segname __DATA
      addr 0x0000000000004010
      size 0x0000000000000008
    offset 16400
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000007
 reserved1 3 (index into indirect symbol table)
 reserved2 0
Load command 2
      cmd LC_SEGMENT_64
  cmdsize 72
  segname __LINKEDIT
   vmaddr 0x0000000000008000
   vmsize 0x0000000000008000
  fileoff 32768
 filesize 19072
  maxprot 0x00000001
 initprot 0x00000001
   nsects 0
    flags 0x0
Load command 3
            cmd LC_DYLD_INFO_ONLY
        cmdsize 48
     rebase_off 32768
    rebase_size 8
       bind_off 32776
      bind_size 24
  weak_bind_off 0
 weak_bind_size 0
  lazy_bind_off 32800
 lazy_bind_size 24
     export_off 32824
    export_size 24
Load command 4
     cmd LC_SYMTAB
 cmdsize 24
  symoff 32856
   nsyms 3
  stroff 32920
 strsize 48
Load command 5
            cmd LC_DYSYMTAB
        cmdsize 80
      ilocalsym 0
      nlocalsym 0
     iextdefsym 0
     nextdefsym 1
      iundefsym 1
      nundefsym 2
         tocoff 0
           ntoc 0
      modtaboff 0
        nmodtab 0
   extrefsymoff 0
    nextrefsyms 0
 indirectsymoff 32904
  nindirectsyms 4
      extreloff 0
        nextrel 0
      locreloff 0
        nlocrel 0
Load command 6
     cmd LC_UUID
 cmdsize 24
    uuid C0E2DC01-8518-33AC-B49F-E200D9BCF968
Load command 7
      cmd LC_VERSION_MIN_MACOSX
  cmdsize 16
  version 10.10
      sdk 12.3
Load command 8
      cmd LC_SOURCE_VERSION
  cmdsize 16
  version 0.0
Load command 9
          cmd LC_LOAD_DYLIB
      cmdsize 64
         name @loader_path/lib/libtorch_python.dylib (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 0.0.0
compatibility version 0.0.0
Load command 10
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name /usr/lib/libSystem.B.dylib (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 1311.100.3
compatibility version 1.0.0
Load command 11
          cmd LC_RPATH
      cmdsize 72
         path /Users/runner/work/_temp/anaconda/envs/wheel_py39/lib (offset 12)
Load command 12
          cmd LC_RPATH
      cmdsize 72
         path /Users/runner/work/_temp/anaconda/envs/wheel_py39/lib (offset 12)
Load command 13
          cmd LC_RPATH
      cmdsize 32
         path @loader_path/lib (offset 12)
Load command 14
      cmd LC_FUNCTION_STARTS
  cmdsize 16
  dataoff 32848
 datasize 8
Load command 15
      cmd LC_DATA_IN_CODE
  cmdsize 16
  dataoff 32856
 datasize 0
Load command 16
      cmd LC_CODE_SIGNATURE
  cmdsize 16
  dataoff 32976
 datasize 18864

pip show -f torch | grep dylib
>

  torch/.dylibs/libomp.dylib
  torch/lib/libc10.dylib
  torch/lib/libshm.dylib
  torch/lib/libtorch.dylib
  torch/lib/libtorch_cpu.dylib
  torch/lib/libtorch_global_deps.dylib
  torch/lib/libtorch_python.dylib

ls -all /Users/user/opt/miniconda3/envs/vision/lib/python3.9/site-packages/torch/.dylibs/
> 
total 1376
drwxr-xr-x   3 user  staff      96 Jun  9 10:50 .
drwxr-xr-x  84 user  staff    2688 Jun  9 10:50 ..
-rw-r--r--   1 user  staff  702928 Jun  9 10:50 libomp.dylib

Installing older nightly version works on this (rather old) OSX version:

pip install --upgrade --pre "torch==1.12.0.dev20220520" --extra-index-url=https://download.pytorch.org/whl/nightly

[xsacha]

Sounds like _C.cpython-39-darwin.so is in torch directory and looking for libshm in same directory. I've only been using libtorch on Mac so not entirely familiar with why this lib is there or why it needs libshm.

Does symlinking libshm from torch/lib/ to torch/ help? What does otool -L say for that library?

As for the old MacOS issues, maybe a dependency on something newer crept in with all the MPS changes?

[jskye]

i had the same prob installing stable diffusion, not requiring cuda. the problem turned out to be that that libshm isnt bundled into virtual env installs or isnt linked to local env. however it was in my system install which is the one that showed up for pip show -f torch above. so by copying the library over to the venv , i was able to fix this, but i also got a signature error and so i also needed to copy over libtorch_cpu. unfortunately i still run into problems with the S.D install, trying to install the torch-sparse module with an older xcode version. not sure why libshm is not being bundled or linked properly in torch for older envs.

[albanD]

Closing this as the information here most likely is stale after a year.
Feel free to open a new issue if you observe this with latest PyTorch (2.1+)!