Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

error when compiling for x86_64-pc-windows-gnu #3

Open
ntzb opened this issue Sep 5, 2023 · 31 comments
Open

error when compiling for x86_64-pc-windows-gnu #3

ntzb opened this issue Sep 5, 2023 · 31 comments

Comments

@ntzb
Copy link

ntzb commented Sep 5, 2023

hi and thanks for rperf.
it is mentioned in the readme that rperf should build and work on all major platforms, though its development and usage focus is on Linux-based systems, so that is where it will be most feature-complete.
trying to switch from iperf to rperf, I need some binaries for Windows and Linux aarch64.

compiling for Windows fails with:

❯ cargo build --release --target x86_64-pc-windows-gnu
  Downloaded winapi-build v0.1.1
  Downloaded miow v0.2.2
  Downloaded kernel32-sys v0.2.2
  Downloaded ws2_32-sys v0.2.1
  Downloaded winapi-util v0.1.5
  Downloaded windows-targets v0.48.5
  Downloaded winapi v0.2.8
  Downloaded winapi v0.3.9
  Downloaded windows-sys v0.48.0
  Downloaded winapi-x86_64-pc-windows-gnu v0.4.0
  Downloaded windows_x86_64_gnu v0.48.5
  Downloaded windows v0.48.0
  Downloaded 12 crates (20.0 MB) in 2.60s (largest was `windows` at 11.9 MB)
   Compiling winapi-x86_64-pc-windows-gnu v0.4.0
   Compiling winapi v0.3.9
   Compiling winapi-build v0.1.1
   Compiling windows_x86_64_gnu v0.48.5
   Compiling winapi v0.2.8
   Compiling libc v0.2.147
   Compiling memchr v2.6.3
   Compiling bitflags v1.2.1
   Compiling kernel32-sys v0.2.2
   Compiling ws2_32-sys v0.2.1
   Compiling cfg-if v0.1.10
   Compiling cfg-if v1.0.0
   Compiling regex-syntax v0.7.5
   Compiling windows-targets v0.48.5
   Compiling aho-corasick v1.0.5
   Compiling serde v1.0.188
   Compiling num-traits v0.2.16
   Compiling memoffset v0.6.5
   Compiling slab v0.4.9
   Compiling regex-automata v0.3.8
   Compiling unicode-width v0.1.10
   Compiling log v0.4.20
   Compiling serde_json v1.0.105
   Compiling textwrap v0.11.0
   Compiling windows-sys v0.48.0
   Compiling getrandom v0.2.10
   Compiling num_cpus v1.16.0
   Compiling vec_map v0.8.2
   Compiling ryu v1.0.15
   Compiling regex v1.9.5
   Compiling net2 v0.2.39
   Compiling atty v0.2.14
   Compiling winapi-util v0.1.5
   Compiling termcolor v1.2.0
   Compiling time v0.1.45
   Compiling strsim v0.8.0
   Compiling itoa v1.0.9
   Compiling humantime v2.1.0
   Compiling iovec v0.1.4
   Compiling env_logger v0.8.4
   Compiling clap v2.33.4
   Compiling chrono v0.4.28
   Compiling miow v0.2.2
   Compiling mio v0.6.23
   Compiling core_affinity v0.5.10
   Compiling ctrlc v3.4.1
   Compiling uuid v0.8.2
   Compiling nix v0.20.2
   Compiling simple-error v0.2.3
   Compiling rperf v0.1.8 (/Users/nitzan/Downloads/rperf)
error[E0433]: failed to resolve: could not find `sys` in `nix`
  --> src/stream/tcp.rs:23:10
   |
23 | use nix::sys::socket::{setsockopt, sockopt::RcvBuf, sockopt::SndBuf};
   |          ^^^ could not find `sys` in `nix`

error[E0433]: failed to resolve: could not find `unix` in `os`
  --> src/stream/tcp.rs:70:18
   |
70 |     use std::os::unix::io::AsRawFd;
   |                  ^^^^ could not find `unix` in `os`

error[E0433]: failed to resolve: could not find `unix` in `os`
   --> src/stream/tcp.rs:416:18
    |
416 |     use std::os::unix::io::AsRawFd;
    |                  ^^^^ could not find `unix` in `os`

error[E0433]: failed to resolve: could not find `sys` in `nix`
  --> src/stream/udp.rs:26:10
   |
26 | use nix::sys::socket::{setsockopt, sockopt::RcvBuf, sockopt::SndBuf};
   |          ^^^ could not find `sys` in `nix`

error[E0433]: failed to resolve: could not find `unix` in `os`
  --> src/stream/udp.rs:72:18
   |
72 |     use std::os::unix::io::AsRawFd;
   |                  ^^^^ could not find `unix` in `os`

error[E0433]: failed to resolve: could not find `unix` in `os`
   --> src/stream/udp.rs:439:18
    |
439 |     use std::os::unix::io::AsRawFd;
    |                  ^^^^ could not find `unix` in `os`

error[E0599]: no method named `as_raw_fd` found for struct `mio::net::TcpStream` in the current scope
   --> src/stream/tcp.rs:261:90
    |
261 | ...                   super::setsockopt(stream.as_raw_fd(), super::RcvBuf, &self.receive_buffer)?;
    |                                                ^^^^^^^^^ method not found in `TcpStream`

error[E0599]: no method named `as_raw_fd` found for struct `mio::net::TcpStream` in the current scope
   --> src/stream/tcp.rs:492:46
    |
492 |                     super::setsockopt(stream.as_raw_fd(), super::SndBuf, &self.send_buffer)?;
    |                                              ^^^^^^^^^ method not found in `TcpStream`

error[E0599]: no method named `as_raw_fd` found for struct `std::net::UdpSocket` in the current scope
   --> src/stream/udp.rs:209:46
    |
209 |                     super::setsockopt(socket.as_raw_fd(), super::RcvBuf, receive_buffer)?;
    |                                              ^^^^^^^^^ method not found in `UdpSocket`

warning: use of deprecated associated function `chrono::NaiveDateTime::from_timestamp`: use `from_timestamp_opt()` instead
   --> src/stream/udp.rs:258:52
    |
258 |             let current_timestamp = NaiveDateTime::from_timestamp(now.as_secs() as i64, now.subsec_nanos());
    |                                                    ^^^^^^^^^^^^^^
    |
    = note: `#[warn(deprecated)]` on by default

warning: use of deprecated associated function `chrono::NaiveDateTime::from_timestamp`: use `from_timestamp_opt()` instead
   --> src/stream/udp.rs:302:55
    |
302 |                 let source_timestamp = NaiveDateTime::from_timestamp(origin_seconds, origin_nanoseconds);
    |                                                       ^^^^^^^^^^^^^^

error[E0599]: no method named `as_raw_fd` found for struct `std::net::UdpSocket` in the current scope
   --> src/stream/udp.rs:475:46
    |
475 |                     super::setsockopt(socket.as_raw_fd(), super::SndBuf, send_buffer)?;
    |                                              ^^^^^^^^^ method not found in `UdpSocket`

Some errors have detailed explanations: E0433, E0599.
For more information about an error, try `rustc --explain E0433`.
warning: `rperf` (bin "rperf") generated 2 warnings
error: could not compile `rperf` (bin "rperf") due to 10 previous errors; 2 warnings emitted

it seems that there are some dependencies on *nix tcp/udp

@flan
Copy link
Member

flan commented Sep 5, 2023

I've added a new branch, windows-cfg_if, which replaces my incorrect assumption that Rust would discard unused symbols during its build process, rather than trying to evaluate everything.

I also incorrectly assumed that nix and other things flagged in your build would still provide those symbols, even if their implementations were effectively just generating errors.

It seems to build in a limited cross-compilation environment here, but I haven't used Windows in over two decades, so I'm afraid I don't have any means of testing it myself.

@ntzb
Copy link
Author

ntzb commented Sep 5, 2023

thanks for looking into this! I'll happily test.
it compiles fine now. tried a simple usage, but getting an error.
server:

C:\Apps>rperf -s
[2023-09-05T16:50:41Z INFO  rperf::server] server listening on 0.0.0.0:5199
[2023-09-05T16:50:54Z INFO  rperf::server] connection from 192.168.1.150:57412
[2023-09-05T16:50:54Z ERROR rperf::server] error in client-handler: connection lost
[2023-09-05T16:50:54Z INFO  rperf::server] 192.168.1.150:57412 disconnected

client:

C:\Apps\rperf>rperf -c htpc.lan
[2023-09-05T16:50:53Z INFO  rperf::client] connecting to server at htpc.lan:5199...
[2023-09-05T16:50:53Z INFO  rperf::client] connected to server
[2023-09-05T16:50:53Z ERROR rperf] unable to run client: operation would block

p.s. I tried cross compiling for aarch64 (target aarch64-unknown-linux-gnu, but seems to hit brick walls, on macos m1. if it's possible, can you try and build for that target as well?)

@flan
Copy link
Member

flan commented Sep 5, 2023

Can you re-run this with --debug (or -d)?

I suspect the block is happening at

Ok(stream.write_all(&output_buffer)?)
, though I'm unsure of the circumstances under which the Windows networking subsystem would generate an EWOULDBLOCK response to an established, not-yet-used TCP connection, especially since write_all() is supposed to take care of overflows. So it might be happening somewhere else.

@ntzb
Copy link
Author

ntzb commented Sep 5, 2023

server:

C:\Apps>rperf -s -d
[2023-09-05T17:22:28Z DEBUG rperf] registering SIGINT handler...
[2023-09-05T17:22:28Z DEBUG rperf] beginning normal operation...
[2023-09-05T17:22:28Z DEBUG rperf::stream::tcp::receiver] using OS assignment for IPv4 TCP ports
[2023-09-05T17:22:28Z DEBUG rperf::stream::tcp::receiver] using OS assignment for IPv6 TCP ports
[2023-09-05T17:22:28Z DEBUG rperf::stream::udp::receiver] using OS assignment for IPv4 UDP ports
[2023-09-05T17:22:28Z DEBUG rperf::stream::udp::receiver] using OS assignment for IPv6 UDP ports
[2023-09-05T17:22:28Z DEBUG rperf::utils::cpu_affinity] enumerated CPU cores: [0, 1, 2, 3, 4, 5]
[2023-09-05T17:22:28Z DEBUG rperf::utils::cpu_affinity] not applying CPU core affinity
[2023-09-05T17:22:28Z INFO  rperf::server] server listening on 0.0.0.0:5199
[2023-09-05T17:22:36Z INFO  rperf::server] connection from 192.168.1.150:57630
[2023-09-05T17:22:36Z DEBUG rperf::protocol::communication] awaiting length-value from 192.168.1.150:57630...
[2023-09-05T17:22:36Z ERROR rperf::server] error in client-handler: connection lost
[2023-09-05T17:22:36Z INFO  rperf::server] 192.168.1.150:57630 disconnected

client:

C:\Apps\rperf>rperf -c htpc.lan -d
[2023-09-05T17:22:35Z DEBUG rperf] registering SIGINT handler...
[2023-09-05T17:22:35Z DEBUG rperf] connecting to server...
[2023-09-05T17:22:35Z DEBUG rperf::stream::tcp::receiver] using OS assignment for IPv4 TCP ports
[2023-09-05T17:22:35Z DEBUG rperf::stream::tcp::receiver] using OS assignment for IPv6 TCP ports
[2023-09-05T17:22:35Z DEBUG rperf::stream::udp::receiver] using OS assignment for IPv4 UDP ports
[2023-09-05T17:22:35Z DEBUG rperf::stream::udp::receiver] using OS assignment for IPv6 UDP ports
[2023-09-05T17:22:35Z DEBUG rperf::utils::cpu_affinity] enumerated CPU cores: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
[2023-09-05T17:22:35Z DEBUG rperf::utils::cpu_affinity] not applying CPU core affinity
[2023-09-05T17:22:35Z DEBUG rperf::protocol::messaging] preparing TCP upload config...
[2023-09-05T17:22:35Z DEBUG rperf::protocol::messaging] preparing TCP download config...
[2023-09-05T17:22:35Z INFO  rperf::client] connecting to server at htpc.lan:5199...
[2023-09-05T17:22:35Z INFO  rperf::client] connected to server
[2023-09-05T17:22:35Z DEBUG rperf::client] running in forward-mode: server will be receiving data
[2023-09-05T17:22:35Z DEBUG rperf::protocol::communication] sending message of length 172, Object {"family": String("tcp"), "kind": String("configuration"), "length": Number(32768), "receive_buffer": Number(0), "role": String("download"), "streams": Number(1), "test_id": Array [Number(107), Number(61), Number(17), Number(48), Number(58), Number(130), Number(65), Number(229), Number(177), Number(53), Number(253), Number(22), Number(160), Number(115), Number(249), Number(105)]}, to 192.168.1.181:5199...
[2023-09-05T17:22:35Z ERROR rperf] unable to run client: operation would block

@flan
Copy link
Member

flan commented Sep 5, 2023

I'll hopefully be able to do some testing directly against M1/M2 hardware later this year, running native Mac OS, but for now, I'm just trusting that nix provides the facilities I read in its documentation.

aarch64 for Linux should work, though: we do a lot of testing with SBCs, even down to some 8-bit controllers. I'll need to know more about what you mean by "brick walls".

@flan
Copy link
Member

flan commented Sep 5, 2023

Yeah... sending 176 bytes really shouldn't be a problem with the socket in the state I expect it's in.

But it looks like mio's implementation changed since 0.6 (in a manner that may have been later retconned by changing the function's name) and now it expects the stream to already be in non-blocking mode, rather than performing that conversion itself. It expects everything to be non-blocking, actually.

Creating another branch off of the Windows one to build this, though I'm afraid I'm working on a number of other things today, so I can't promise super-quick responses.

@flan
Copy link
Member

flan commented Sep 5, 2023

Ah, it's not that mio's interface changed within 0.6, it's that it behaves very differently on Windows.

And mio 0.7+ removed set_keepalive(), which was the only reason I had introduced it in the first place.

Learning what I can.

@ntzb
Copy link
Author

ntzb commented Sep 5, 2023

I'll hopefully be able to do some testing directly against M1/M2 hardware later this year, running native Mac OS, but for now, I'm just trusting that nix provides the facilities I read in its documentation.

aarch64 for Linux should work, though: we do a lot of testing with SBCs, even down to some 8-bit controllers. I'll need to know more about what you mean by "brick walls".

I don't think it's rperf related, just probably cross compilation rust stuff (missing linker libs, things like that) - I've already burnt a few hours on it today, it's not exactly a well documented procedure (macos m1 targeting aarch64-unknown-linux-gnu, tried 3 different linkers, ld cannot find needed libs).
it's just if you have cross compilation working, I'd be happy to see a aarch64 test binary as a shortcut :)

@ntzb
Copy link
Author

ntzb commented Sep 5, 2023

Ah, it's not that mio's interface changed within 0.6, it's that it behaves very differently on Windows.

Learning what I can.

it's fine, surely nothing urgent about this Windows build, any effort is appreciated

@flan
Copy link
Member

flan commented Sep 6, 2023

I've excised mio in the socket2 branch. While everything works (at least under Linux), there's a timing mismatch in the handoff for TCP that doesn't affect total transmission time or data sent, but does skew the data-received window.

I'm trying to figure out how to minimise its impact, but for now, you should be able to build that for Windows to see if it works.

@flan
Copy link
Member

flan commented Sep 6, 2023

I ended up putting bits of mio back in place, but it should all be working now. Possibly a little more accurate than before, though on the order of microseconds, nothing too significant.

@ntzb
Copy link
Author

ntzb commented Sep 6, 2023

tested with socket2 clean build, same exact logs unfortunately.
testing between two windows machines.
the server is only using port 5199 right?

@flan
Copy link
Member

flan commented Sep 6, 2023

This morning, I observed a Windows build linked through cross-compilation from Linux (same process you were using) succeed as a traffic-sending client without issue, so the "operation would block" issue may be something configuration or policy-based (a firewall, perhaps). It's possible the previous implementation would have worked, too.

However, there is a problem when receiving traffic on Windows: only the first interval (one second) of traffic gets through with TCP and nothing with UDP, seemingly because the sender assesses a timeout, concludes the receiver isn't present, and gives up.

I'll look into this when I have time, but it's a lot less obvious. I think I already have access to all the data I need, though (trace-level logs, packet-captures).

As for aarch64/Linux, I'm still able to build without issue (Debian host), but because there's a dependence on dynamic linking (libc specifically), even if I felt comfortable providing precompiled binaries (and you'd feel comfortable accepting them), there's a high chance of a mismatch, so it's probably best to leave resolving the conflict to your own build environment (maybe use a VM on your M1 if you want to build with a native instruction-set), to avoid desync.

@ntzb
Copy link
Author

ntzb commented Sep 7, 2023

interesting, I just (cross) compiled for macos, linux and it works between then on my x86 openwrt machine and m1 mac.
Windows machine still fails as both client and server (tried multiple branches).

regarding the aarch64, I get it, it's ok. I'll try cross

@flan
Copy link
Member

flan commented Sep 8, 2023

Windows should now be able to receive UDP; still working on TCP.

I may have also implemented a working handler for the would-block case you're experiencing, though nobody I've asked to help me test has reported it themselves.

@flan
Copy link
Member

flan commented Sep 8, 2023

TCP reception should now work, too.

Aside from being able to configure TCP keepalive and rend/receive-buffers, there should be functional parity between POSIX and Windows.

If this checks out, I'll merge things and increase the version-number.

@ntzb
Copy link
Author

ntzb commented Sep 9, 2023

since the windows build is running as client, I'm not really sure what could be blocking.
something in my build process might be bad then.
file size is always the same, and the error remains (operation would block).
if you test it on a different machine, and multiple other people confirmed it's working, I might have raised a false flag here, not sure if this issue should still be even open

@flan
Copy link
Member

flan commented Sep 9, 2023

Well, it was in a state where it wouldn't compile for Windows and I did need to rework time-accounting to make it deal with Microsoft's system clock semantics, so the issue definitely has merit.

I'll hopefully be able to see it running on more Windows systems on Monday (I'm in Western Canada, so it's about 3am on Saturday now), and maybe one of them will stall so I can get a live testing candidate.

For whatever it's worth, the Windows environment where it was observed to be working is Windows 10, x64, with very little in the way of additional security.

I don't suppose Windows has any facilities like strace, does it? I'd be curious to see just what is going into the WinSock call and what it's returning, since I should be handling both the POSIX and WSAPI "not yet able to send" cases now (

Err(ref e) if e.kind() == std::io::ErrorKind::WouldBlock || e.kind() == std::io::ErrorKind::TimedOut => {
), which would seem to indicate that there's another class of error that is being overlooked here, but that has the same textual presentation as WouldBlock.

@ntzb
Copy link
Author

ntzb commented Sep 9, 2023

I don't know anything about strace unfortunately. perhaps one of the suggested tools here https://stackoverflow.com/questions/3847745/systrace-for-windows could be helpful.
I have 2 Windows PCs, 10 x64 and 11 x64, and both do not work (with Windows firewall disabled).

p.s. can you provide the Windows test binary that worked for you, or for anyone you gave it to? or I can provide the .exe that I built, and maybe you can test that one, just to eliminate cross compilation shenanigans

@flan
Copy link
Member

flan commented Sep 9, 2023

It's temporarily available in [pre-stable-binary link redacted].

md5sum (XZ compressed): b2795f70cb2c9ecc8a6f202ace37121f

@ntzb
Copy link
Author

ntzb commented Sep 9, 2023

your binary works! client AND server (linux <> windows, simple testing).
now I really wonder what goes wrong in my cross compilation efforts.

you can take it down as far as I'm concerned..

@flan
Copy link
Member

flan commented Sep 9, 2023

My build environment is Debian Trixie (Testing)

All I'm doing is cross build --target x86_64-pc-windows-gnu --release.

Prior steps included apt-get install mingw-w64-x86-64-dev and cargo install cross, though it seems like something else isn't present in my bash-history file.

I recall that cross built a container around the mingw libraries and that cross (cargo) is using it to do the actual compilation, but I haven't touched anything inside.

@ntzb
Copy link
Author

ntzb commented Sep 9, 2023

I'm on macOS 13, something in the Apple+rust, aarch64 -> x86, is probably not mature enough.
the regular build (installing linkers and setting up targets) produces an unusable binary (as we've seen), and using cross fails for every target I'm trying with sh: 1: cargo: not found which probably means I have to build my own images etc.

@flan
Copy link
Member

flan commented Sep 10, 2023

I'm making arrangements to buy a Mac to figure this out (I have other things to port to ARM and could use the rough edges of Asahi to learn anyway), though it may take a couple of weeks before I have one available.

@ntzb
Copy link
Author

ntzb commented Sep 11, 2023

I also considered Ashai linux on my m1, but it's got a way to go.. no external monitors is a huge problem for me.
if you haven't already, I suggest you take a look at https://github.com/AsahiLinux/docs/wiki/Feature-Support and https://www.reddit.com/r/AsahiLinux/comments/11t2yt9/asahi_linux_users_what_has_been_your_reallife/
btw it's my work laptop, I've already asked for a replacement, and will probably dump it in a few months anyway (hopefully you don't plan on buying a mac because of my cross compilation issues.. honestly, don't worry about it)

@flan
Copy link
Member

flan commented Sep 11, 2023

I've only been looking at Asahi for the past week (I have an entire Debian-based operating system to explore porting to ARM in general, not specifically to Apple Silicon, but just at a scale where I need to be able to rebuild binaries and things at decent speeds, without cross-compilation surprises), but it looks sufficient for my needs.

I'm going to wait to see if M3 minis are part of the Apple announcement on Tuesday, then make a decision from there, but it's going to happen in some capacity, both for work (like the OS) and projects I maintain outside of work, like this.

Kinda sucks that ARM is still so bound to U-BOOT and board-specific bring-up processes that Apple is the best option for a general-purpose dev-and-build system right now. I'd really love to just be able to put some reasonably specced mITX-like build together.

Although, on that note, how much memory do you usually see your system watermark at when doing any sort of compilation task?

@ntzb
Copy link
Author

ntzb commented Sep 12, 2023

my code is backend JS, so no much compilation going on here :)
I agree that M1/2/3 are the best ARM the world has to offer right now. if that's what you need, go for it..

@thewh1teagle
Copy link

Tried in windows inside msys2 in ucrt64 and get the following error:

User@DESKTOP-HPEE9O3 UCRT64 ~/rperf
$ ./target/release/rperf.exe -s
[2024-01-19T00:33:32Z INFO  rperf::server] server listening on 0.0.0.0:5199
[2024-01-19T00:34:26Z INFO  rperf::server] connection from 127.0.0.1:52004
[2024-01-19T00:34:26Z ERROR rperf::server] error in client-handler: connection lost
[2024-01-19T00:34:26Z INFO  rperf::server] 127.0.0.1:52004 disconnected
User@DESKTOP-HPEE9O3 UCRT64 ~/rperf
$ ./target/release/rperf.exe -c 127.0.0.1 -p 5199
[2024-01-19T00:34:26Z INFO  rperf::client] connecting to server at 127.0.0.1:5199...
[2024-01-19T00:34:26Z INFO  rperf::client] connected to server
[2024-01-19T00:34:26Z ERROR rperf] unable to run client: operation would block

@flan
Copy link
Member

flan commented Jan 19, 2024

I'm expecting to merge a PR in the next day or two that has been tested on Windows, though it's largely the same solution as in the socket2 branch.

Can I ask which branch you used that led to that problem?

@thewh1teagle
Copy link

@flan
cfg_if branch
main branch didn't compiled

@flan
Copy link
Member

flan commented Jan 19, 2024

socket2 was where I took the code after cfg_if upon encountering that timeout

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants