Delocate "@rpath" when search paths include "/usr/lib/swift"

[timoffex]

Describe the bug
delocate works with /usr/lib/swift/libswiftCore.dylib paths, but fails with @rpath/libswiftCore.dylib when the search path includes "/usr/lib/swift".

Expected behavior
I'm not sure. This might not be a bug; I can understand why the latter might not work. Do you have any pointers as to why the Swift package manager (swift build) might output @rpath/ entries in one environment, and /usr/lib/swift/ entries in another? I would really appreciate some help.

See additional context.

Platform (please complete the following information):

• OS version: macOS 13
• Delocate version: 0.10.4

Additional context
When I build a Swift binary on my personal laptop, otool -L outputs a bunch of paths like

• /usr/lib/swift/libswiftCore.dylib
• /usr/lib/swift/libswiftCoreFoundation.dylib
• /usr/lib/swift/libswiftCoreImage.dylib
• ...

My "/usr/lib/swift" does not include those directories, but the binary works fine (I don't understand why) and delocate works fine as well.

When I do the same in a GitHub workflow, the otool -L output instead shows

• @rpath/libswiftCore.dylib
• @rpath/libswiftCoreFoundation.dylib
• @rpath/libswiftCoreImage.dylib
• ...

I don't know where the discrepancy comes from exactly, but I know that the macOS version is different and that the CI is using Xcode_15.1.app whereas I'm using the Xcode CommandLineTools.

Importantly, however, the error messages show that the RPATH includes /usr/lib/swift:

  ERROR:delocate.libsana:
  @rpath/libswiftCore.dylib not found:
    Needed by: /private/var/folders/14/hgs_fjmn5ms001tb8qtxxn5c0000gn/T/tmp3fkfmyzr/wheel/wandb_core/AppleStats
    Search path:
      /usr/lib/swift
      @loader_path
      /Applications/Xcode_15.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift-5.5/macosx

So I would expect delocate to have the same behavior in my CI as on my personal laptop.

[timoffex]

Note, using install_name_tool -change to manually rewrite the @rpath/ entries to /usr/lib/swift/ makes delocate work, which makes me think that delocate ignores certain paths, since /usr/lib/swift/ does not contain any of the listed libraries.

So the solution is probably to make swift build output the right paths, but I don't have a lot of experience here (I only learned about delocate, rpaths, and all that today). I'm asking in this repo since it seems like an edge case to a functionality (ignoring certain paths), so maybe others have run into this problem.

[HexDecimal]

/usr/lib/*.dylib are often system libraries such as /usr/lib/libstdc++.6.dylib which are not supposed to be bundled with the wheel. Many checks are path.startswith("/usr/lib") to filter these libraries. Delocate mainly expects external libraries to be treated the way Brew handles them.

A more verbose output from Delocate -vv might state why a library was ignored.

[timoffex]

Thank you! Any clue as to why these libraries might be output with "@rpath/" rather than "/usr/lib" by a compiler?

[HexDecimal]

Any clue as to why these libraries might be output with "@rpath/" rather than "/usr/lib" by a compiler?

If I remember correctly. Because the "install name" of libswiftCore.dylib is either /usr/lib/swift/libswiftCore.dylib or @rpath/libswiftCore.dylib. This is decided by the linker flags used to build libswiftCore.dylib itself.

An install name with @rpath/ is preferred for portable libraries. I suspect the issue is with Delocate being confused by non-system libraries being installed in /usr/lib and not your build process, as I've explained.

[HexDecimal]

Delocate would be less confused if these libraries were installed in /usr/local/lib for example.

[HexDecimal]

Looks like Swift installs to /usr/lib/swift because it is an official MacOS tool.

I'm considering changing Delocate to only exclude libraries directly inside of /usr/lib but I don't know the risks or side effects of doing this. I'd like the hear from someone more experienced on this matter.

[timoffex]

I see, thanks for looking into it, I really appreciate it! It's clear to me now this isn't a bug or unexpected behavior with Delocate, so I'll close the issue.

[HexDecimal]

It isn't completely unexpected, but it can still be a bug, one that's affecting bundling the Swift runtime. I'd prefer this issue was left open until a solution is found.

System libraries are ones which are always available on every MacOS system. If Swift has to be installed, then it isn't a system library and Delocate should not ignore it like it's doing now.

[timoffex]

I know a little more now. On my arm64 (aka aarch64) macOS device:

• when I invoke swiftc with --arch arm64, the output binary uses /usr/lib/swift/... paths for Swift standard libraries
• when I invoke swiftc with --arch x86_64, the output binary uses @rpath paths for Swift standard libraries

Is this a Swift compiler bug? Or is it something that delocate should work around? Maybe both?

I think the only way workaround for delocate would be to explicitly treat paths matching @rpath/libswift* the same as system libraries. Maybe it could be an option?

[isuruf]

Can you send the output with output -l (lowercase L)

[timoffex]

otool_l_aarch64.txt
otool_l_x86_64.txt

[isuruf]

Using @rpath paths isn't a bug, but it complicates the logic.

I think the only way workaround for delocate would be to explicitly treat paths matching @rpath/libswift* the same as system libraries.

Yes, I think that's the best way forward.

[timoffex]

A workaround for now is to use the --ignore-missing-dependencies option.