GitHub Community
Self-hosted runners now always hang on jobs? No errors, no changes... #122999
-
Select Topic AreaQuestion BodyI am looking after two self-hosted Ubuntu runners which have been working fine for months until a few hours ago. Now they are seemingly stuck on build jobs that never go anywhere. No errors in the worker logs. I restarted both runners and they pick up jobs fine, but they will never finish them. I verified we're using the latest agent, it self-upgraded. v2.316.1 I saw one failed web call in the runner log but when I tried to curl it I was able to reach just fine. Is there anything else I can look at to troubleshoot further? The Worker log just keeps showing this over and over: [2024-05-08 16:47:40Z INFO HostContext] Well known directory 'Work': '/home/ubuntu/runner/_work' The runner log just shows the job keeps renewing every minute. [2024-05-08 16:48:23Z INFO JobDispatcher] Successfully renew job request 206793, job is valid till 05/08/2024 16:58:22 We haven't made any changes to either the runners or the actions configurations in the repo so I'm not exactly sure what is going on here. |
Beta Was this translation helpful? Give feedback.
Replies: 10 comments 22 replies
-
|
I forgot to write that I looked at the usual system things; disk space is fine, for example. No containers listed in docker, but docker is running for our build job... The last successful job was about 18 hours ago and to my knowledge, nothing changed that would cause a job to just hang in the middle. I'm not having much luck in searching for similar issues either, it's usually hung waiting for a job to be picked up. This is stuck inside the workflow and it just refuses to ever error out. It's also happening at the same exact place on both runners which are two different architectures (ARM and X86). |
Beta Was this translation helpful? Give feedback.
-
|
It seems to hang running actions/checkout@V4. No errors, and I think it runs everything properly. At the end of that step it just spins and spins and never finishes. We have to eventually cancel the jobs as they seemingly never timeout or exit! Nothing in the error logs to indicate a problem that I can tell. I tried using actions/checkout@V3 but it gives the same result, unfortunately. We're using just the bare checkout, no options at all... |
Beta Was this translation helpful? Give feedback.
-
|
[2024-05-09 01:13:18Z INFO ProcessInvokerWrapper] Starting process: $ ps aux | grep [n]ode It just hangs... I see no open issues for checkoutV4... |
Beta Was this translation helpful? Give feedback.
This comment was marked as spam.
This comment was marked as spam.
-
|
Thank you, I have completed steps 1 through 6... |
Beta Was this translation helpful? Give feedback.
-
|
We are also experiencing this issue. During the checkout it just hangs at the last step when running checkout@v3 or checkout@v4. All we see is the node16 or node20 process running the dist/index file in the checkout which gets hung. |
Beta Was this translation helpful? Give feedback.
-
|
Unfortunately, 2.315.0 does not seem to work either. |
Beta Was this translation helpful? Give feedback.
-
|
I am starting to suspect an issue with GHA infrastructure combined with insufficient timeout function in the runner itself... |
Beta Was this translation helpful? Give feedback.
-
|
Our most recent attempt was to replace the node16 in the externals dir with node20 and that also hung on the ./run.sh --check at the same spot. Screen shot to verify that we replaced the pathing 
|
Beta Was this translation helpful? Give feedback.
-
|
Perhaps, but anything that is running node is causing the issue. The weird part is that we have dozens and dozens of runners still working that have been running prior to this week (approx prior to May 3). Every runner we have spun up since then using terraform using the same AMI, github runner version and node versions are now suffering from the forever pause when they try do to anything in node16 or node20 |
Beta Was this translation helpful? Give feedback.
-
|
We have now also switched out the externals node for v22.1.0 and the checkout action is still stuck. We are going to try setting the github runner process to force it to node20 vs node16 as it seems the runner process itself is still using 16 so might be something with the process node version?!?! |
Beta Was this translation helpful? Give feedback.
-
|
Incidentally the post checkout step also runs the checkout node step and gets stuck... 😿 |
Beta Was this translation helpful? Give feedback.
-
|
@jefflightweb do you have a firewall or egress white/black listing where you host your runners? We certainly do and are wondering if we are missing a whitelist IP/DNS or something new that was introduced recently that could be getting "stuck" without a retry or error handler. |
Beta Was this translation helpful? Give feedback.
-
|
No firewall issues here, our runners are in a separate VPC with proper security group and no outbound ACLs that would cause interference here... And they've basically been the same for months now. No changes. |
Beta Was this translation helpful? Give feedback.
-
|
Another day and this is still broken. :-( |
Beta Was this translation helpful? Give feedback.
-
|
We have 5-6 of our senior engineers working on this and nothing has presented itself and GitHub support has been engaged and their theory and comments seem to imply its something we did, but all we did was start runners using terraform the same way we always have on the same AMI and runner libraries. These were set to autoupdate the runners and we can see working runners started on 2.314 and are now on 2.316 through that auto-updating process. However even starting new runners on older versions with auto-update turned off still do not work. New versions of the runner 2.316.0 or 1 are also still not working on newly started machines. To be clear the process starts and reports home. They pick up jobs but checkout or any jobs running node still hang indefinitely. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for sending an update. I'm glad you are engaged with GH Support, we have been tied up with other unrelated fires and haven't been able to go that route yet, but will if necessary. This is a strange situation, indeed. |
Beta Was this translation helpful? Give feedback.
-
|
FWIW, one of our engineers just verified that we can issue a new PAT from GH console and install on a fresh instance and then the runners work. I have a feeling that it won't last for very long, but we shall see. |
Beta Was this translation helpful? Give feedback.
-
|
We are trying the same thing from a clean ec2 on our end as well. Our prevailing theory is that the process callbacks to the DLLs that live in github runner are not working on Ubuntu 22.04. Sadly the issue prevailed when we reverted to an ubuntu 20 instance. |
Beta Was this translation helpful? Give feedback.
-
|
what is your
|
Beta Was this translation helpful? Give feedback.
-
|
The exact same kernel (Ubuntu 22.04) on both X86 and aarch64. |
Beta Was this translation helpful? Give feedback.
-
|
It isn't even checkout that is broken. Any action that runs via node is breaking. We have tried node16 default and using the flag to force node20. Here is a ps from doing a jfrog action that uses node that also gets stuck. We didnt even try a checkout. We KNOW it is node just now why yet. This is the tree for the stuck process that is just hanging... ⬇️ |
Beta Was this translation helpful? Give feedback.
-
|
It looks about the same for checkout action just replace the action path |
Beta Was this translation helpful? Give feedback.
-
|
So we finally traced this down to an issue with datadog-apm-library-js:5.12.0-1 which on Ubunutu 22 and Ubuntu 20 injects itself in such a way that no js process will exit unless we SIGTERM. Downgrading this library to the last stable we had working (4.19.0-1) is what solved the issue for us. |
Beta Was this translation helpful? Give feedback.
-
|
Wow, we are using Datadog here also... I would not have expected that. |
Beta Was this translation helpful? Give feedback.
-
|
Hrmm, I stopped and disabled all datadog agents for now, and restarted the runner processes, but this hasn't fixed it yet here. |
Beta Was this translation helpful? Give feedback.
-
|
OK. I had to remove all the injection libraries and rebooted and everything works again. I wasn't able to find an older deb package for that library, only the source code from GitHub releases, so I think we'll just do without tracing for now (we weren't using it anyway). |
Beta Was this translation helpful? Give feedback.
So we finally traced this down to an issue with datadog-apm-library-js:5.12.0-1 which on Ubunutu 22 and Ubuntu 20 injects itself in such a way that no js process will exit unless we SIGTERM. Downgrading this library to the last stable we had working (4.19.0-1) is what solved the issue for us.