New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
uv_run hangs on macOS arm64/Rosetta in x86_64 docker container #4279
Comments
For reference, I narrowed this example down from CMake Issue 25562. CMake uses libuv to run child processes, and works on many platforms, but hangs in libuv on this one. |
If you run the program with |
I just ran the program under
These lines are identical every time except for the |
That's... unusual. strace is saying the process's personality changed between syscalls. It checks bit 30 in the syscall number (i.e. I can't really make heads or tails of that trace. Can you perhaps trying dumping core or attaching with gdb when the process hangs and a) inspect registers, and b) obtain a backtrace? |
I think strace's heuristics are getting confused. It prints the x32/64 bit messages repeatedly. It's also printing most of the syscalls in raw form. gdb doesn't work well in the container either. I found some posts about using ROSETTA_DEBUGSERVER_PORT to run a process in a gdbserver and debug it from another terminal, but that doesn't actually run the process until a remote debugger attaches to the session. When I do that and run the "continue" step in gdb the process hangs immediately in gdb Immediate Hang Backtrace (click to expand)
I've not been able to get a backtrace for the real hang, but I patched libuv as follows to print more information: debug-prints.patch (Click to expand)diff --git a/src/unix/linux.c b/src/unix/linux.c
index 8eeb352e..64a6d351 100644
--- a/src/unix/linux.c
+++ b/src/unix/linux.c
@@ -1427,7 +1427,9 @@ void uv__io_poll(uv_loop_t* loop, int timeout) {
*/
lfields->current_timeout = timeout;
+ fprintf(stderr, "before epoll_pwait\n");
nfds = epoll_pwait(epollfd, events, ARRAY_SIZE(events), timeout, sigmask);
+ fprintf(stderr, "after epoll_pwait\n");
/* Update loop->time unconditionally. It's tempting to skip the update when
* timeout == 0 (i.e. non-blocking poll) but there is no guarantee that the
diff --git a/src/unix/process.c b/src/unix/process.c
index dd58c18d..12b6b02c 100644
--- a/src/unix/process.c
+++ b/src/unix/process.c
@@ -76,6 +76,7 @@ extern char **environ;
#ifdef UV_USE_SIGCHLD
static void uv__chld(uv_signal_t* handle, int signum) {
+ write(2, "c\n", 2);
assert(signum == SIGCHLD);
uv__wait_children(handle->loop);
}
diff --git a/src/unix/signal.c b/src/unix/signal.c
index bc4206e6..46139c56 100644
--- a/src/unix/signal.c
+++ b/src/unix/signal.c
@@ -185,6 +185,8 @@ static void uv__signal_handler(int signum) {
uv_signal_t* handle;
int saved_errno;
+ write(2, "a\n", 2);
+
saved_errno = errno;
memset(&msg, 0, sizeof msg);
@@ -197,6 +199,7 @@ static void uv__signal_handler(int signum) {
handle != NULL && handle->signum == signum;
handle = RB_NEXT(uv__signal_tree_s, &uv__signal_tree, handle)) {
int r;
+ write(2, "b\n", 2);
msg.signum = signum;
msg.handle = handle; The changes print before and after
This shows that the signal handling and process reaping works several times (omitted in
Then the hang doesn't occur again until I Ctrl-C the |
The code to handle signals does look like there is an illegal data race here, since this variable update might need to be seq-cst (it is read outside the lock, but needs to be sequenced-with the read from the previous epoll read--but this variable is written after the pipe write, so it would appear to be a data race even with seq-cst annotations). It currently isn't even marked atomic on read or write: Line 325 in a7cbda9
|
We should maybe consider refactoring the code so that we can share this with uv_async, since that code is correctly sequenced for use from signals, and this signal-handler code does not look correct for use even from threads: Lines 65 to 70 in a7cbda9
|
I assume this == handle->caught_signals? It's "protected" by a syscall-induced memory barrier but yeah, non-atomic stores can take a while to propagate to other cores so it's quite possible for events to get lost. I'll open a pull request to fix that. handle->dispatched_signals is only accessed from the event loop thread. There should be no synchronization issues there. |
Changes to the `caught_signals` field should be immediately visible to other threads, otherwise "we get signal" events can get lost. Tentative fix for the linked issue. I say "tentative" because the cause is not fully understood but all signs point to libuv's handling of SIGCHLD signals, and they only manifest on architectures with weaker memory models than x86. Fixes: libuv#4279
Changes to the `caught_signals` field should be immediately visible to other threads, otherwise "we get signal" events can get lost. Tentative fix for the linked issue. I say "tentative" because the cause is not fully understood but all signs point to libuv's handling of SIGCHLD signals, and they only manifest on architectures with weaker memory models than x86. Fixes: libuv#4279
Ah, I see now that yes, it has a lock exclusion around it, so non-atomic access is okay, as it only gets accessed on the main thread when the lock is released and it is not in the list of handlers, but changing the list requires that lock, and releasing the lock provides the needed memory barrier |
I guess the trace above does seem to indicate a kernel bug, since the signal itself appears to be missing until something triggers the kernel to check for it? Although it might be a issue with the rosetta/docker emulator also? I ran the test inside an ubuntu-aarch64 vm, and did not run into issue. |
The simplest fix might be to change to use signalfd instead (since linux 2.6.27 / glibc 2.8), which works almost identically to the libuv emulation of it. We already did this on macOS for essentially identical reasons (though kevent supports waitpid natively for fast native work, while poll/epoll seems to require some inefficient workarounds) |
I saw a similar hang when signals are delivered while in a signal handler: https://github.com/libuv/libuv/pull/3755/files does the added test look similar to the issue you are seeing? |
No that is a different issue. In that example, you showed there is a case where EDIT: misread slightly the PR diff. This case seems to be an issue with the PR itself, but not an issue with v1.x code |
I'm leaning towards this. The fact that it manifests neither on native ARM or native x86 is suggestive. Switching to signalfd won't work. Any thread can be the receiver of a signal. signalfd only works correctly when combined with pthread_sigmask. |
Oh, yeah I see now the design of signalfd seems to be just as bad as for sigwaitinfo, which it states it is identical to (and which was the other possible option for fixing this). I assumed they had designed the handling better, to be like kevent, but alas, no. The problem with most alternative options (except kevent, which doesn't have this issue) is indeed that the signal must be blocked on all threads, as setting it to SIG_IGN globally also breaks signalfd. I am also not certain if this is actually considered a kernel bug though, or just undefined behavior in the posix implementation on linux--I have observed that the handling of signals is such that the kernel picks a thread at random, and then waits to run the signal handler if that thread later is eligible to be scheduled and isn't blocking signals. But if that thread dies or doesn't need to be rescheduled, then the kernel may choose to delay delivery of that signal indefinitely. (I have run into this exact problem in the past because I had blocked SIGCHLD on a high-priority thread once, and it caused libuv to miss the occasional SIGCHLD notifications, because the kernel happened to pick that thread to choose to handle them and that one thread blocked them) |
I've had a few reports of libuv hanging in
|
That must be unrelated, since it is a different kernel and does not use signals, but yes, it sounds similar (c.f. https://gitlab.kitware.com/cmake/cmake/-/issues/25562) |
Version: 1.43.0, 1.47.0, others
Platform: macOS arm64 + Rosetta + Docker + Linux/x86_64
On a macOS Apple Silicon host using Docker and Rosetta to run a Linux/x86_64 container, the following test program hangs and leaves a defunct child process. The same program works well in containers with native architectures, or on plain Linux and macOS hosts.
The example in demo.tar.gz is a small C program using libuv to run
sh -c "echo echo"
, read its output, wait for exit, cleanup, and repeat. It should iterate 10000 times, but hangs with a defunct childsh
. For convenience, the contents ofdemo.tar.gz
include:test-uv.c (click to expand)
Dockerfile (click to expand)
README.md (click to expand)
On a macOS Apple Silicon host:
The test process in the container eventually hangs.
The text was updated successfully, but these errors were encountered: