Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

workaround for "runc list" returning "no such file or directory" #17977

Open
wants to merge 9 commits into
base: master
Choose a base branch
from
Open

workaround for "runc list" returning "no such file or directory" #17977

Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
60d404a
workaround for "runc list" returning "no such file or directory"
prezha Jan 17, 2024
c2f1b81
retry on "runc list" error
prezha Jan 17, 2024
95de446
Merge branch 'kubernetes:master' into fixRuncList
prezha Jan 18, 2024
d2fce20
Merge branch 'kubernetes:master' into fixRuncList
prezha Jan 18, 2024
063e669
Merge branch 'kubernetes:master' into fixRuncList
prezha Mar 3, 2024
2ed9bcf
Merge branch 'kubernetes:master' into fixRuncList
prezha Mar 7, 2024
148a0c9
Merge branch 'kubernetes:master' into fixRuncList
prezha Mar 14, 2024
116ff72
Merge branch 'kubernetes:master' into fixRuncList
prezha Mar 14, 2024
afe6199
Merge branch 'kubernetes:master' into fixRuncList
prezha Apr 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
18 changes: 16 additions & 2 deletions pkg/minikube/cruntime/cri.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,11 +25,13 @@ import (
"os/exec"
"path"
"strings"
"time"

"github.com/blang/semver/v4"
"github.com/pkg/errors"
"k8s.io/klog/v2"
"k8s.io/minikube/pkg/minikube/command"
"k8s.io/minikube/pkg/util/retry"
)

// container maps to 'runc list -f json'
Expand Down Expand Up @@ -108,10 +110,22 @@ func listCRIContainers(cr CommandRunner, root string, o ListContainersOptions) (
}

args = append(args, "list", "-f", "json")
rr, err = cr.RunCmd(exec.Command("sudo", args...))
if err != nil {

// avoid "no such file or directory" runc list error by retrying
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am generally not a big fan of retyring till it works...I rather waiting for a condition to be met,
if runc is gonna include that in their release soon opencontainers/runc#3349
I rather wait for that

Copy link
Collaborator Author

@prezha prezha Mar 26, 2024 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in a less likely, but still occasional scenario where the race condition happens, runc currently exits with an error and we don't have a specific condition to wait for, so we retry and the odds to hit race condition again are minor (but we could repeat a couple of times, if that's needed)

i agree it would be better if this would be handled internally by runc, which that pr aims to address, so i asked author & approver if there are plans to release it (given that it was merged to the main 2 years ago)

// TODO (prezha): consider removing this retry workaround when #17976 is addressed upstream and we've updated to that runc version
list := func() error {
rr, err = cr.RunCmd(exec.Command("sudo", args...))
if err != nil {
klog.Infof("temporary error listing containers using runc (will retry): %v", err)
return err
}
// bail out
return nil
}
if err := retry.Expo(list, 100*time.Millisecond, 5*time.Second); err != nil {
return nil, errors.Wrap(err, "runc")
}

content := rr.Stdout.Bytes()
klog.Infof("JSON = %s", content)
d := json.NewDecoder(bytes.NewReader(content))
Expand Down