Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Automatically restart runner after a crash #30662

Closed
kerem0comert opened this issue Apr 23, 2024 · 7 comments
Closed

Automatically restart runner after a crash #30662

kerem0comert opened this issue Apr 23, 2024 · 7 comments
Labels
topic/gitea-actions related to the actions of Gitea

Comments

@kerem0comert
Copy link

Feature Description

Hello,

I have an Action Runner, which crashes every now and then due to some exception. For instance the latest logs show:

[Python Code Checks/code-check] ⭐ Run Post Checkout repository
[Python Code Checks/code-check] [DEBUG] run post step for 'Checkout repository'
[Python Code Checks/code-check] [DEBUG] executing remote job container: [node /var/run/act/actions/actions-checkout@v2/dist/index.js]
[Python Code Checks/code-check] Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.43/images/catthehacker/ubuntu:act-latest/json": context canceled
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x48 pc=0xb5e0e1]
 
goroutine 434942 [running]:

where this container is the one that I use for Python-based Actions, the one you can find here.

Once any kind of crash such as this one happens, the Runner goes Offline and requires a manual restart. Is there a recommended way to auto-restart the runner? I guess one can setup a cronjob or something to check the runner is alive routinely, but I wanted to create this issue anyway to see if this is a problem that others face too or what's the best way to approach it.

Thanks!

Screenshots

No response
@kerem0comert kerem0comert added the type/proposal The new feature has not been accepted yet but needs to be discussed first. label Apr 23, 2024
@lunny
Copy link
Member

lunny commented Apr 23, 2024

You can run it with systemd or docker then you can make it restart easier.

Can you paste more details so that we can know what made the runner panic.

@kerem0comert
Copy link
Author

I see. Here are more detailed logs:


[Python Code Checks/code-check] [DEBUG] evaluating expression 'always()'

[Python Code Checks/code-check] [DEBUG] expression 'always()' evaluated to 'true'

[Python Code Checks/code-check] ⭐ Run Post Checkout repository

[Python Code Checks/code-check] [DEBUG] run post step for 'Checkout repository'

[Python Code Checks/code-check] [DEBUG] executing remote job container: [node /var/run/act/actions/actions-checkout@v2/dist/index.js]

[Python Code Checks/code-check] Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.43/images/catthehacker/ubuntu:act-latest/json": context canceled

panic: runtime error: invalid memory address or nil pointer dereference

[signal SIGSEGV: segmentation violation code=0x1 addr=0x48 pc=0xb5e0e1]
 
goroutine 338871 [running]:

github.com/nektos/act/pkg/container.(*containerReference).UpdateFromImageEnv.(*containerReference).extractFromImageEnv.func1({0xefd3a8, 0xc000314150})

        /go/pkg/mod/gitea.com/gitea/act@v0.2.51/pkg/container/docker_run.go:528 +0x1e1

github.com/nektos/act/pkg/container.(*containerReference).UpdateFromImageEnv.Executor.IfNot.func2({0xefd3a8, 0xc000314150})

        /go/pkg/mod/gitea.com/gitea/act@v0.2.51/pkg/common/executor.go:166 +0x4f

github.com/nektos/act/pkg/runner.(*RunContext).ApplyExtraPath(0xc0007de7e0, {0xefd3a8, 0xc000314150}, 0xc0000e27b8)

        /go/pkg/mod/gitea.com/gitea/act@v0.2.51/pkg/runner/run_context.go:416 +0x204

github.com/nektos/act/pkg/runner.(*stepActionRemote).post.runPostStep.func1({0xefd3a8, 0xc000314150})

        /go/pkg/mod/gitea.com/gitea/act@v0.2.51/pkg/runner/action.go:656 +0x891

github.com/nektos/act/pkg/runner.(*stepActionRemote).post.runStepExecutor.func2({0xefd3a8, 0xc000314150})

        /go/pkg/mod/gitea.com/gitea/act@v0.2.51/pkg/runner/step.go:141 +0xd83

github.com/nektos/act/pkg/runner.(*stepActionRemote).post.Executor.If.func4({0xefd3a8, 0xc000314150})

        /go/pkg/mod/gitea.com/gitea/act@v0.2.51/pkg/common/executor.go:156 +0x45

github.com/nektos/act/pkg/runner.(*stepActionRemote).post.Executor.If.func6({0xefd3a8, 0xc000314150})

        /go/pkg/mod/gitea.com/gitea/act@v0.2.51/pkg/common/executor.go:156 +0x45

github.com/nektos/act/pkg/runner.newJobExecutor.useStepLogger.func12({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/act@v0.2.51/pkg/runner/job_executor.go:225 +0x2e2

github.com/nektos/act/pkg/runner.newJobExecutor.Executor.Finally.func13({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/act@v0.2.51/pkg/common/executor.go:183 +0x4f

github.com/nektos/act/pkg/runner.newJobExecutor.Executor.Finally.func13({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/act@v0.2.51/pkg/common/executor.go:183 +0x4f

github.com/nektos/act/pkg/runner.newJobExecutor.Executor.Finally.func13({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/act@v0.2.51/pkg/common/executor.go:183 +0x4f

github.com/nektos/act/pkg/runner.newJobExecutor.Executor.Finally.func13({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/act@v0.2.51/pkg/common/executor.go:183 +0x4f

github.com/nektos/act/pkg/runner.newJobExecutor.Executor.Finally.func13({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/act@v0.2.51/pkg/common/executor.go:183 +0x4f

github.com/nektos/act/pkg/runner.newJobExecutor.Executor.Finally.func13({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/act@v0.2.51/pkg/common/executor.go:183 +0x4f

github.com/nektos/act/pkg/runner.newJobExecutor.Executor.Finally.func13({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/act@v0.2.51/pkg/common/executor.go:183 +0x4f

github.com/nektos/act/pkg/runner.newJobExecutor.Executor.Finally.func13({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/act@v0.2.51/pkg/common/executor.go:183 +0x4f

github.com/nektos/act/pkg/runner.newJobExecutor.Executor.Finally.func13({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/act@v0.2.51/pkg/common/executor.go:183 +0x4f

github.com/nektos/act/pkg/runner.newJobExecutor.Executor.Finally.func14({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/act@v0.2.51/pkg/common/executor.go:182 +0x31

github.com/nektos/act/pkg/runner.newJobExecutor.func7({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/act@v0.2.51/pkg/runner/job_executor.go:159 +0xe2

github.com/nektos/act/pkg/runner.newJobExecutor.Executor.Finally.func16({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/act@v0.2.51/pkg/common/executor.go:183 +0x4f

github.com/nektos/act/pkg/runner.newJobExecutor.Executor.Finally.func17({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/act@v0.2.51/pkg/common/executor.go:182 +0x31

github.com/nektos/act/pkg/runner.newJobExecutor.Executor.Finally.func18({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/act@v0.2.51/pkg/common/executor.go:182 +0x31

github.com/nektos/act/pkg/runner.newJobExecutor.NewPipelineExecutor.Executor.Then.func21({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/act@v0.2.51/pkg/common/executor.go:148 +0x142

github.com/nektos/act/pkg/runner.(*RunContext).Executor.func1({0xefd3a8, 0xc000b5b230})

        /go/pkg/mod/gitea.com/gitea/act@v0.2.51/pkg/runner/run_context.go:595 +0x50

github.com/nektos/act/pkg/runner.(*runnerImpl).NewPlanExecutor.func1.1({0xefd3a8, 0xc000b5a240})

        /go/pkg/mod/gitea.com/gitea/act@v0.2.51/pkg/runner/runner.go:214 +0x19a

github.com/nektos/act/pkg/runner.(*runnerImpl).NewPlanExecutor.func1.NewParallelExecutor.func2.1(0x8f4ba5?, 0xc00c67bb00?)

        /go/pkg/mod/gitea.com/gitea/act@v0.2.51/pkg/common/executor.go:107 +0x52

created by github.com/nektos/act/pkg/runner.(*runnerImpl).NewPlanExecutor.func1.NewParallelExecutor.func2 in goroutine 338870

        /go/pkg/mod/gitea.com/gitea/act@v0.2.51/pkg/common/executor.go:105 +0xf7
[Python Code Checks/code-check] Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.43/images/catthehacker/ubuntu:act-latest/json": context canceled

panic: runtime error: invalid memory address or nil pointer dereference

[signal SIGSEGV: segmentation violation code=0x1 addr=0x48 pc=0xb5e0e1]

@silverwind silverwind added the topic/gitea-actions related to the actions of Gitea label Apr 27, 2024
@wolfogre
Copy link
Member

wolfogre commented May 6, 2024 •
edited

@kerem0comert Which version of act runner are you using?

I think I have already fixed the panic: nektos/act#2071

@wolfogre wolfogre removed the type/proposal The new feature has not been accepted yet but needs to be discussed first. label May 6, 2024
@kerem0comert
Copy link
Author

I run the latest via:

gitea/act_runner:latest

@wolfogre
Copy link
Member

wolfogre commented May 7, 2024

I run the latest via:

gitea/act_runner:latest

latest cannot identify the version; it could be very old if you pulled it a long time ago. You can go to /admin/actions/runners to check the version of runners.

@kerem0comert
Copy link
Author

I see. Well in that case, the version seems to be v0.2.6.

@wolfogre
Copy link
Member

wolfogre commented May 8, 2024

I see. Well in that case, the version seems to be v0.2.6.

It's quite an old version; you can upgrade it. And as Lunny said, run it with systemd or Docker, then you can restart it more easily.

@wolfogre wolfogre closed this as completed May 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
topic/gitea-actions related to the actions of Gitea
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@lunny @silverwind @wolfogre @kerem0comert