Unable to cross compile windows wheel with zstd as dependency

[Owen-CH-Leung]

Bug Description

When zstd exists as part of the project dependency, the command maturin build --profile dev --target x86_64-pc-windows-msvc will fail with the following warnings and error:

The following warnings were emitted during compilation:

warning: zstd-sys@2.0.10+zstd.1.5.6: clang: error: unknown argument: '-Brepro'
warning: zstd-sys@2.0.10+zstd.1.5.6: clang: error: unknown argument: '--target=x86_64-pc-windows-msvc'
warning: zstd-sys@2.0.10+zstd.1.5.6: clang: error: unknown argument: '-fuse-ld=lld-link'
warning: zstd-sys@2.0.10+zstd.1.5.6: clang: error: unknown argument: '-fvisibility=hidden'
warning: zstd-sys@2.0.10+zstd.1.5.6: clang: error: no such file or directory: '/imsvc/root/.cache/cargo-xwin/xwin/crt/include'
warning: zstd-sys@2.0.10+zstd.1.5.6: clang: error: no such file or directory: '/imsvc/root/.cache/cargo-xwin/xwin/sdk/include/ucrt'
warning: zstd-sys@2.0.10+zstd.1.5.6: clang: error: no such file or directory: '/imsvc/root/.cache/cargo-xwin/xwin/sdk/include/um'
warning: zstd-sys@2.0.10+zstd.1.5.6: clang: error: no such file or directory: '/imsvc/root/.cache/cargo-xwin/xwin/sdk/include/shared'

--- stderr


  error occurred: Command "clang-cl" "-nologo" "-MD" "-Z7" "-Brepro" "-m64" "--target=x86_64-pc-windows-msvc" "-Wno-unused-command-line-argument" "-fuse-ld=lld-link" "/imsvc/root/.cache/cargo-xwin/xwin/crt/include" "/imsvc/root/.cache/cargo-xwin/xwin/sdk/include/ucrt" "/imsvc/root/.cache/cargo-xwin/xwin/sdk/include/um" "/imsvc/root/.cache/cargo-xwin/xwin/sdk/include/shared" "-I" "zstd/lib/" "-I" "zstd/lib/common" "-I" "zstd/lib/legacy" "-fvisibility=hidden" "-DZSTD_DISABLE_ASM=" "-DZSTD_LIB_DEPRECATED=0" "-DXXH_PRIVATE_API=" "-DZSTDLIB_VISIBILITY=" "-DZDICTLIB_VISIBILITY=" "-DZSTDERRORLIB_VISIBILITY=" "-DZSTD_LEGACY_SUPPORT=1" "-Fo/io/target/x86_64-pc-windows-msvc/debug/build/zstd-sys-96a2e3968fd14727/out/zstd/lib/legacy/zstd_v07.o" "-c" "--" "zstd/lib/legacy/zstd_v07.c" with args "clang-cl" did not execute successfully (status code exit status: 1).

it's incorrectly looking for files at /imsvc/root/.cache/cargo-xwin/, where the correct path should be /root/.cache/cargo-xwin/.

However, even after fixing the path by mkdir -p /imsvc/root/.cache && cp -r /root/.cache/cargo-xwin/ /imsvc/root/.cache/, the following warnings & errors comes out:

warning: zstd-sys@2.0.10+zstd.1.5.6: clang: error: unknown argument: '-Brepro'
warning: zstd-sys@2.0.10+zstd.1.5.6: clang: error: unknown argument: '--target=x86_64-pc-windows-msvc'
warning: zstd-sys@2.0.10+zstd.1.5.6: clang: error: unknown argument: '-fuse-ld=lld-link'
warning: zstd-sys@2.0.10+zstd.1.5.6: clang: error: unknown argument: '-fvisibility=hidden'
warning: zstd-sys@2.0.10+zstd.1.5.6: clang: error: cannot specify '-Fo/io/target/x86_64-pc-windows-msvc/debug/build/zstd-sys-96a2e3968fd14727/out/zstd/lib/common/debug.o' when compiling multiple source files

exit status: 1
  cargo:warning=ToolExecError: Command "clang-cl" "-nologo" "-MD" "-Z7" "-Brepro" "-m64" "--target=x86_64-pc-windows-msvc" "-Wno-unused-command-line-argument" "-fuse-ld=lld-link" "/imsvc/root/.cache/cargo-xwin/xwin/crt/include" "/imsvc/root/.cache/cargo-xwin/xwin/sdk/include/ucrt" "/imsvc/root/.cache/cargo-xwin/xwin/sdk/include/um" "/imsvc/root/.cache/cargo-xwin/xwin/sdk/include/shared" "-I" "zstd/lib/" "-I" "zstd/lib/common" "-I" "zstd/lib/legacy" "-fvisibility=hidden" "-DZSTD_DISABLE_ASM=" "-DZSTD_LIB_DEPRECATED=0" "-DXXH_PRIVATE_API=" "-DZSTDLIB_VISIBILITY=" "-DZDICTLIB_VISIBILITY=" "-DZSTDERRORLIB_VISIBILITY=" "-DZSTD_LEGACY_SUPPORT=1" "-Fo/io/target/x86_64-pc-windows-msvc/debug/build/zstd-sys-96a2e3968fd14727/out/zstd/lib/legacy/zstd_v03.o" "-c" "--" "zstd/lib/legacy/zstd_v03.c" with args "clang-cl" did not execute successfully (status code exit status: 1).

Clearly, the compilation of the zstd source code failed.

Your maturin version (maturin --version)

0.15.1

Your Python version (python -V)

3.8.18

Your pip version (pip -V)

24.0

What bindings you're using

None

Does cargo build work?

• Yes, it works

If on windows, have you checked that you aren't accidentally using unix path (those with the forward slash /)?

• Yes

Steps to Reproduce

1. On a brand new project folder, create Cargo.toml with just zstd as dependency

[dependencies]
zstd = "*"

2. create src/main.rs, and put the following into the main function:

fn main() {
    println!("{}", zstd::DEFAULT_COMPRESSION_LEVEL);
}

3. run a docker container using maturin:v1.5.1 image. docker run -it --rm -v $(pwd):/io --entrypoint /bin/bash ghcr.io/pyo3/maturin:v1.5.1

4. Inside the terminal, run rustup target add x86_64-pc-windows-msvc, and then yum install clang

5. Run maturin build --profile dev --target x86_64-pc-windows-msvc. Then you will see the warnings and error I've posted

[messense]

The maturin Docker image uses old version of CentOS for native compilation, it's not intend for cross compilation usage.

When cross compiling to Windows, make sure you install a recent version of LLVM/clang. The stock version in CentOS 7 is too old.

[Owen-CH-Leung]

Ahh thanks. let me try to get a more updated toolchain version and retry