v3.1.1 ks-console crashbackoff very often

[FireLv]

Describe the Bug
Recently,I found ks-console pod restart many times,and I use command "kubectl logs " to look the pod log,and it only have the logs after restarting,so I can't get the reason why it restart.
I use "kubectl describe pod" ,and I got the msg:"Liveness probe failed: dial tcp 172.19.0.157:8000: connect: connection timed out"

Versions Used
KubeSphere: v3.1.1
Kubernetes: v1.18

Environment
How many nodes and their hardware configuration:
OS: CentOS 7.6
master: 1 node 4cpu/8g
worker: 1 node 4cpu/8g

How To Reproduce
It often happens without reason,so I don't know how to reproduce it

Expected behavior
I want it no restart

[FireLv]

the liveness settings is:
timeout: 30s
delay: 15s
probe frequency: 10s

[zheng1]

Can you provide the return value of the following command on the node for ks-console?

journalctl -u kubelet --since=-12h > kubelet.log

[FireLv]

kubelet.log.tar.gz
ok,the file I have upload

[FireLv]

Can you provide the return value of the following command on the node for ks-console?

journalctl -u kubelet --since=-12h > kubelet.log

I have upload the kubelet.log

[zheng1]

It looks like ks-console-646dd6dcb4-j4ts4_kubesphere-system is not cleaned up, it's probably a docker issue, can you provide the return value of the following command?

kubectl version
docker version
kubectl get sc 

[FireLv]

It looks like ks-console-646dd6dcb4-j4ts4_kubesphere-system is not cleaned up, it's probably a docker issue, can you provide the return value of the following command?

kubectl version
docker version
kubectl get sc 

kubectl version

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.4", GitCommit:"c96aede7b5205121079932896c4ad89bb93260af", GitTreeState:"clean", BuildDate:"2020-06-17T11:41:22Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.4", GitCommit:"c96aede7b5205121079932896c4ad89bb93260af", GitTreeState:"clean", BuildDate:"2020-06-17T11:33:59Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

docker version

Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:23:03 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:25:29 2018
  OS/Arch:          linux/amd64
  Experimental:     false

kubectl get sc

NAME                 PROVISIONER       RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nfs-data (default)   nfs-provisioner   Delete                   Immediate                           true                                        99d

[zheng1]

You can refer to this link: kubernetes/kubernetes#74135

In our experience, this may be resolved by upgrading docker and kubelet, but the most complete solution is to stop using nfs in K8s.

[FireLv]

ks-console-646dd6dcb4-j4ts4_kubesphere-system

ks-console-646dd6dcb4-j4ts4_kubesphere-system is the pod that often restart,and I remove the healthy probe,then k8s remove it automatic,now the pod is "ks-console-79fc8668d7-fgxx2 "

[FireLv]

You can refer to this link: kubernetes/kubernetes#74135

In our experience, this may be resolved by upgrading docker and kubelet, but the most complete solution is to stop using nfs in K8s.

Do you have some suggest for the version of docker and kubelet?

[zheng1]

k8s v1.23.10
docker the latest version