Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

how to build for powerpc openwrt musl libc 1.2.3 device featuring e500v1 core ? #1363

Open
thomas725 opened this issue Oct 28, 2023 · 24 comments

Comments

@thomas725
Copy link
Contributor

thomas725 commented Oct 28, 2023

For the last few weeks with the help of @Emilgardis I've tried to get rust compiled for my powerpc openwrt device.

TP-Link TL-WDR4900 v1.x

See also: #1349

Currently we're testing those images created by @Emilgardis:

We managed to get the first one of those to produce debug build using:

RUSTFLAGS="-C target-feature=-crt-static" cross +nightly build --target powerpc-unknown-linux-musl

but since the resulting binary is 15mb big I'd need a release build to actually test if it can run on my device, which failed to build so far:

$ RUSTFLAGS="-C target-feature=-crt-static" cross -Zbuild-std=std,panic_abort +nightly build --target powerpc-unknown-linux-musl --release
[cross] warning: using newer rustc `1.75.0-nightly (2f1bd0729 2023-10-27)` for the target. Current active rustc on the host is `rustc 1.73.0 (cc66ad468 2023-10-03)`.
 > Update with `rustup update`
   Compiling compiler_builtins v0.1.101
   Compiling core v0.0.0 (/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core)
   Compiling libc v0.2.149
   Compiling cc v1.0.79
   Compiling memchr v2.5.0
   Compiling std v0.0.0 (/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std)
   Compiling unwind v0.0.0 (/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/unwind)
   Compiling rustc-std-workspace-core v1.99.0 (/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/rustc-std-workspace-core)
   Compiling alloc v0.0.0 (/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc)
   Compiling cfg-if v1.0.0
   Compiling adler v1.0.2
   Compiling rustc-demangle v0.1.23
   Compiling rustc-std-workspace-alloc v1.99.0 (/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/rustc-std-workspace-alloc)
   Compiling panic_unwind v0.0.0 (/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/panic_unwind)
   Compiling panic_abort v0.0.0 (/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/panic_abort)
   Compiling gimli v0.28.0
   Compiling miniz_oxide v0.7.1
   Compiling std_detect v0.1.5 (/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/stdarch/crates/std_detect)
   Compiling object v0.32.0
   Compiling hashbrown v0.14.2
   Compiling addr2line v0.21.0
   Compiling proc_macro v0.0.0 (/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/proc_macro)
   Compiling my-project v0.1.0 (/home/user/Development/git/my-project)
error[E0463]: can't find crate for `panic_abort`

For more information about this error, try `rustc --explain E0463`.
error: could not compile `my-project` (bin "my-project") due to previous error
[cross] warning: rust-std is not available for powerpc-unknown-linux-musl
[cross] note: you may need to build components for the target via `-Z build-std=<components>` or in your cross configuration specify `target.powerpc-unknown-linux-musl.build-std`
              the available components are core, std, alloc, and proc_macro
@thomas725
Copy link
Contributor Author

thomas725 commented Oct 28, 2023

commenting out the panic= property in this Cargo.toml block

[profile.release]
opt-level = 'z'   # Optimize for size
lto = true        # Enable link-time optimization
codegen-units = 1 # Reduce number of codegen units to increase optimizations
#panic = 'abort'   # Abort on panic
strip = true # Strip symbols from binary*

makes this release build command work too:

CROSS_CONTAINER_ENGINE=podman RUSTFLAGS="-C target-feature=-crt-static" cross +nightly build --target powerpc-unknown-linux-musl --release

and results in a 321kb big binary:

ELF 32-bit MSB pie executable, PowerPC or cisco 4500, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-powerpc.so.1, stripped

Running that binary on my openwrt router sadly doesn't print "hello world" as expected, but instead:

-ash: /bin/my-project: not found

I guess that's because it's looking for the first while we only have the second on device:

root@openwrt:~# ls -alih /lib/ld-musl-powerpc.so.1
ls: /lib/ld-musl-powerpc.so.1: No such file or directory
root@openwrt:~# ls -alih /lib/ld-musl-powerpc-sf.so.1 
    305 lrwxrwxrwx    1 root     root           7 Oct 15  2022 /lib/ld-musl-powerpc-sf.so.1 -> libc.so

@thomas725
Copy link
Contributor Author

ohhh, but it actually is compatible! running

ln -s /lib/libc.so /lib/ld-musl-powerpc.so.1

makes my binary print the expected output!

root@openwrt:~# /bin/my-project
hello world

@thomas725
Copy link
Contributor Author

thomas725 commented Oct 28, 2023

@Emilgardis Can you help me translate our success into something I can put into a .gitlab-ci.yml task?
I'm reading #273 but am a bit confused in how complex that task seems to be. Do I really need to run docker in docker, can't I just use the build image directly?

@thomas725
Copy link
Contributor Author

also, will this solution blow up as soon as I do anything that uses float-point operations? since my musl-libc is actually soft-float but the compiler seems to think we have hardware-float?

Can you make the second = soft-float image public so I can use that one instead? Also can you merge what we managed to do into the regular cross-rs infrastructure?

@Emilgardis
Copy link
Member

Awesome!

Cross works by running in/starting a container, so unfortunately container in container is needed. If you want you can skip cross and just emulate what cross does but without cross. Shouldn't be too hard, just copy the dockerfile I've made and supply all the needed files and use that as the base image for the runner, then also install rustup and you should be all good :)

If you do want to use cross for Gitlab Runners, recommended usage is via CROSS_REMOTE now. see https://github.com/cross-rs/cross/wiki/FAQ#gitlab-ci

@Emilgardis
Copy link
Member

also, could you try adding a float operation to the binary, interested if the device actually can do hard floats.

something like

println!("{:?}", (1.0f32/10.0).sin());

@thomas725
Copy link
Contributor Author

thomas725 commented Oct 28, 2023

no, doesn't seem like it can:

hello world
Illegal instruction

running in dev environment this instead prints:

hello world
0.09983342

I will have a look at https://github.com/cross-rs/cross/wiki/FAQ#gitlab-ci tomorrow and see if I can make that work, thanks for your great support, you're amazing :) gotta get some sleep now.

@thomas725
Copy link
Contributor Author

thomas725 commented Oct 29, 2023

Okey, so I don't think what's described at https://github.com/cross-rs/cross/wiki/FAQ#gitlab-ci can work for me, since I'm not hosting a gitlab instance myself but just using the public gitlab.com instance which includes 400 free CI minutes myself - but I don't think (or know how) I can access a docker host from there like described at tcp://docker:2375/

I'm considering using github instead, but the section about github says it need's to be reworked because actions-rs is unmaintained.

@Emilgardis
Copy link
Member

You should be able to use tcp://docker:2375/ just fine, doesnt matter if you host the instance yourself or not.

So, for only using softfloat, you can do RUSTFLAGS="-C target-feature=-crt-static,-hard-float" and also use the muslsf image instead.

Another thing you could do is use a custom target, this way you don't have to specify any rustflags via env.

#Cross.toml
[target."powerpc-unknown-linux-muslsf.json"]
image = "ghcr.io/emilgardis/powerpc-unknown-linux-muslsf-cross:test"
build-std = true
# powerpc-unknown-linux-muslsf.json
{
    "arch": "powerpc",
    "crt-objects-fallback": "musl",
    "crt-static-default": false,
    "crt-static-respected": true,
    "data-layout": "E-m:e-p:32:32-Fn32-i64:64-n32",
    "dynamic-linking": true,
    "env": "musl",
    "has-rpath": true,
    "has-thread-local": true,
    "linker-flavor": "gnu-cc",
    "llvm-target": "powerpc-unknown-linux-musl",
    "max-atomic-width": 32,
    "os": "linux",
    "position-independent-executables": true,
    "pre-link-args": {
        "gnu-cc": [
            "-m32"
        ],
        "gnu-lld-cc": [
            "-m32"
        ]
    },
    "relro-level": "full",
    "stack-probes": {
        "kind": "inline"
    },
    "supported-split-debuginfo": [
        "packed",
        "unpacked",
        "off"
    ],
    "target-endian": "big",
    "target-family": [
        "unix"
    ],
    "target-mcount": "_mcount",
    "target-pointer-width": "32",
    "features": "-hard-float"
}

cross +nightly build --release --target powerpc-unknown-linux-muslsf.json

@Emilgardis
Copy link
Member

Emilgardis commented Oct 29, 2023

looking further into the MPC8540, you might be fine with SPE. this comment is a bit concerning though

The following CPUs are known to not working with this port:

MPC8540 (for this one, you can use Debian's regular powerpc port with activated FPU emulation in the kernel)

#Cross.toml
[target."powerpc-unknown-linux-muslspe.json"]
image = "ghcr.io/emilgardis/powerpc-unknown-linux-muslsf-cross:test"
build-std = true
# powerpc-unknown-linux-muslspe.json
{
    "arch": "powerpc",
    "crt-objects-fallback": "musl",
    "crt-static-default": false,
    "crt-static-respected": true,
    "data-layout": "E-m:e-p:32:32-i64:64-n32",
    "dynamic-linking": true,
    "env": "musl",
    "has-rpath": true,
    "has-thread-local": true,
    "linker-flavor": "gnu-cc",
    "llvm-target": "powerpc-unknown-linux-muslspe",
    "max-atomic-width": 32,
    "os": "linux",
    "position-independent-executables": true,
    "pre-link-args": {
        "gnu-cc": [
            "-m32 -mspe"
        ],
        "gnu-lld-cc": [
            "-m32 -mspe"
        ]
    },
    "relro-level": "full",
    "stack-probes": {
        "kind": "inline"
    },
    "supported-split-debuginfo": [
        "packed",
        "unpacked",
        "off"
    ],
    "target-endian": "big",
    "target-family": [
        "unix"
    ],
    "target-mcount": "_mcount",
    "target-pointer-width": "32",
    "features": "-fpu,+spe"
}

@thomas725
Copy link
Contributor Author

hmm, something with that custom target json didn't work as hoped:

$ cross +nightly build --target powerpc-unknown-linux-muslsf.json --release
[cross] warning: using newer rustc `1.75.0-nightly (2f1bd0729 2023-10-27)` for the target. Current active rustc on the host is `rustc 1.73.0 (cc66ad468 2023-10-03)`.
 > Update with `rustup update`
   Compiling my-project v0.1.0 (/home/user/Development/git/my-project)
error: linking with `cc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin" VSLANG="1033" "cc" "-m32" "/tmp/rustctgSOPN/symbols.o" "/target/powerpc-unknown-linux-muslsf/release/deps/my_project-b43545ee96f7e2e3.my_project.2be8c3d152745613-cgu.0.rcgu.o" "-Wl,--as-needed" "-L" "/target/powerpc-unknown-linux-muslsf/release/deps" "-L" "/target/release/deps" "-L" "/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/powerpc-unknown-linux-muslsf/lib" "-Wl,-Bstatic" "/target/powerpc-unknown-linux-muslsf/release/deps/libcompiler_builtins-ef09095d5d60da69.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/powerpc-unknown-linux-muslsf/lib" "-o" "/target/powerpc-unknown-linux-muslsf/release/deps/my_project-b43545ee96f7e2e3" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-Wl,--strip-all" "-nodefaultlibs"
  = note: /usr/bin/ld: cannot find Scrt1.o: No such file or directory
          /usr/bin/ld: cannot find crti.o: No such file or directory
          ...
          /usr/bin/ld: /target/powerpc-unknown-linux-muslsf/release/deps/my_project-b43545ee96f7e2e3.my_project.2be8c3d152745613-cgu.0.rcgu.o: relocations in generic ELF (EM: 20)
          /usr/bin/ld: /target/powerpc-unknown-linux-muslsf/release/deps/my_project-b43545ee96f7e2e3.my_project.2be8c3d152745613-cgu.0.rcgu.o: error adding symbols: file in wrong format
          collect2: error: ld returned 1 exit status

@thomas725
Copy link
Contributor Author

and the spe version fails like this:

error: linking with `cc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin" VSLANG="1033" "cc" "-m32 -mspe" "/tmp/rustcGLJTbP/symbols.o" "/target/powerpc-unknown-linux-muslspe/release/deps/my_project-5fc72a3857ce6d3c.my_project.d68e5b2406453db7-cgu.0.rcgu.o" "-Wl,--as-needed" "-L" "/target/powerpc-unknown-linux-muslspe/release/deps" "-L" "/target/release/deps" "-L" "/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/powerpc-unknown-linux-muslspe/lib" "-Wl,-Bstatic" "/target/powerpc-unknown-linux-muslspe/release/deps/libcompiler_builtins-5290246e7a0b36d3.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/powerpc-unknown-linux-muslspe/lib" "-o" "/target/powerpc-unknown-linux-muslspe/release/deps/my_project-5fc72a3857ce6d3c" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-Wl,--strip-all" "-nodefaultlibs"
  = note: cc: error: unrecognized command line option '-m32 -mspe'

@thomas725
Copy link
Contributor Author

thomas725 commented Oct 29, 2023

okey so the binary resulting from this command:

RUSTFLAGS="-C target-feature=-crt-static,-hard-float" CROSS_CONTAINER_ENGINE=podman cross +nightly build --target powerpc-unknown-linux-musl --release

gives good float output:

hello world
0.09983342

even though it does output this warning during building:

warning: unknown feature specified for `-Ctarget-feature`: `hard-float`
  |
  = note: it is still passed through to the codegen backend
  = help: consider filing a feature request

@Emilgardis
Copy link
Member

Emilgardis commented Oct 29, 2023

that warning is what I'm discussing in rust-lang/rust#117347

for the powerpc-unknown-linux-muslsf.json and powerpc-unknown-linux-muslspe.json targets, forgot that we also need to specify the linker

i'm not certain what the envvar would be called but you can set the linker in .cargo/config.toml

should be powerpc-linux-muslsf-gcc

@thomas725
Copy link
Contributor Author

I've tried putting those two into .cargo/config.toml:

[target."powerpc-unknown-linux-muslsf.json"]
linker = "powerpc-linux-muslsf-gcc"

[target.powerpc-unknown-linux-musl]
linker = "powerpc-linux-muslsf-gcc"

But it didn't change the output of cross +nightly build --target powerpc-unknown-linux-muslsf.json --release shown 4 posts above.

btw. huge thanks for helping me with this! :)

@thomas725
Copy link
Contributor Author

thomas725 commented Oct 30, 2023

Also, since you said using tcp://docker:2375/ on public gitlab.com should work just fine, I wanted to try that - but I need a docker image for the layer calling cross, and the guide at https://github.com/cross-rs/cross/wiki/FAQ#gitlab-ci doesn't specify one. Is there one provided by the cross-rs project for this purpose or do I need to do cargo install cross in a generic rust docker image?

@Emilgardis
Copy link
Member

we have a generic image https://github.com/cross-rs/cross/pkgs/container/cross

@thomas725
Copy link
Contributor Author

thomas725 commented Oct 30, 2023

thanks! is there also one with rust nightly pre-installed?

$ RUSTFLAGS="-C target-feature=-crt-static,-hard-float" cross +nightly build --target powerpc-unknown-linux-musl --release
[cross] warning: unable to get metadata for package
[cross] note: Falling back to `cargo` on the host.
error: toolchain 'nightly-x86_64-unknown-linux-gnu' is not installed

@Emilgardis
Copy link
Member

no wouldn't make sense to install nightly in the images since they are "nightly"

install nightly with rustup install nightly

@Emilgardis
Copy link
Member

also, can you answer my question in rust-lang/rust#117347 (comment) please :3

@thomas725
Copy link
Contributor Author

Okey, I did, but now get this strange error:

$ RUSTFLAGS="-C target-feature=-crt-static,-hard-float" cross +nightly build --target powerpc-unknown-linux-musl --release
[cross] warning: using newer rustc `1.75.0-nightly (608e9682f 2023-10-29)` for the target. Current active rustc on the host is `rustc 1.73.0 (cc66ad468 2023-10-03)`.
 > Update with `rustup update`
info: downloading component 'rust-src'
info: installing component 'rust-src'
Error: 
   0: `docker inspect runner-xs6vzpvo-project-51146970-concurrent-0` failed with exit status: 1
Stderr:
   Error: No such object: runner-xs6vzpvo-project-51146970-concurrent-0

@Emilgardis Emilgardis changed the title how to build for powerpc openwrt musl libc 1.2.3 device? how to build for powerpc openwrt musl libc 1.2.3 device featuring e500v1 core ? Oct 30, 2023
@Emilgardis
Copy link
Member

that's #1351, im still not sure why it happens since it has worked for a while before

@thomas725
Copy link
Contributor Author

thomas725 commented Nov 1, 2023

okey, thanks for showing me the connection!
So cross-rs build won't work on GitLab until there's a solution for #1351 - correct?

Can you help me get the json based build definition files working?
For the powerpc-unknown-linux-muslsf.json you posted I get this error:

/usr/bin/ld: cannot find Scrt1.o: No such file or directory
/usr/bin/ld: cannot find crti.o: No such file or directory

And for the *spe.json version's the error is:

cc: error: unrecognized command line option '-m32 -mspe'

@Emilgardis
Copy link
Member

the powerpc-unknown-linux-muslsf.json target is using the wrong c toolchain, you'd have to tweak the variables set in

https://github.com/cross-rs/cross-toolchains/pull/46/files#diff-d3289395a4fe24e408d63f42af8d51c23b61c51ec8974e090bb33c9a60c68a57

however, I'm not sure about the naming of the env-vars with .json targets, I think I remember something about cargo not supporting them

for the spe version, same story

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants