RKE cluster was not created with Failed to bring up Etcd Plane: Error: No such container: etcd-rolling-snapshots

[martinat2604]

RKE Version v1.5.1

docker version: -

Client: Docker Engine - Community
Version:           23.0.6
API version:       1.42
Go version:        go1.19.9
Git commit:        ef23cbc
Built:             Fri May  5 21:19:37 2023
OS/Arch:           linux/amd64
Context:           default
Server: Docker Engine - Community
Engine:
Version:          23.0.6
API version:      1.42 (minimum version 1.12)
Go version:       go1.19.9
Git commit:       9dbdbd4
Built:            Fri May  5 21:18:11 2023
OS/Arch:          linux/amd64
Experimental:     false
containerd:
Version:          1.6.21
GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
runc:
Version:          1.1.7
GitCommit:        v1.1.7-0-g860f061
docker-init:
Version:          0.19.0
GitCommit:        de40ad0

docker info: -

Client:
Context:    default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
	Version:  v0.11.2
	Path:     /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
	Version:  v2.20.2
	Path:     /usr/libexec/docker/cli-plugins/docker-compose
scan: Docker Scan (Docker Inc.)
	Version:  v0.23.0
	Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
Containers: 2
Running: 1
Paused: 0
Stopped: 1
Images: 47
Server Version: 23.0.6
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
runc version: v1.1.7-0-g860f061
init version: de40ad0
Security Options:
seccomp
Profile: builtin
cgroupns
Kernel Version: 5.14.0-362.8.1.el9_3.x86_64
Operating System: Red Hat Enterprise Linux 9.3 (Plow)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.36GiB
Name: sdrpptwrkl01
ID: 08430d64-2f91-433a-bb3e-c1898638da7d
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

Operating system: -

We are using CIS Red Hat Enterprise Linux 9 Benchmark - Level 1(RHEL9.3)
selinux status disabled.

cluster.yml file: -

nodes:
- address: 172.19.123.132
port: "22"
internal_address: ""
role:
- controlplane
- worker
- etcd
hostname_override: ""
user: rke
docker_socket: /var/run/docker.sock
ssh_key: ""
ssh_key_path: /home/osssupport/.ssh/id_rsa
ssh_cert: ""
ssh_cert_path: ""
labels: {}
taints: []
- address: 172.19.123.133
port: "22"
internal_address: ""
role:
- controlplane
- worker
- etcd
hostname_override: ""
user: rke
docker_socket: /var/run/docker.sock
ssh_key: ""
ssh_key_path: /home/osssupport/.ssh/id_rsa
ssh_cert: ""
ssh_cert_path: ""
labels: {}
taints: []
- address: 172.19.123.134
port: "22"
internal_address: ""
role:
- controlplane
- worker
- etcd
hostname_override: ""
user: rke
docker_socket: /var/run/docker.sock
ssh_key: ""
ssh_key_path: /home/osssupport/.ssh/id_rsa
ssh_cert: ""
ssh_cert_path: ""
labels: {}
taints: []
services:
etcd:
	image: ""
	extra_args: {}
	extra_args_array: {}
	extra_binds: []
	extra_env: []
	win_extra_args: {}
	win_extra_args_array: {}
	win_extra_binds: []
	win_extra_env: []
	external_urls: []
	ca_cert: ""
	cert: ""
	key: ""
	path: ""
	uid: 0
	gid: 0
	snapshot: null
	retention: ""
	creation: ""
	backup_config: null
kube-api:
	image: ""
	extra_args: {}
	extra_args_array: {}
	extra_binds: []
	extra_env: []
	win_extra_args: {}
	win_extra_args_array: {}
	win_extra_binds: []
	win_extra_env: []
	service_cluster_ip_range: 10.43.0.0/16
	service_node_port_range: ""
	pod_security_policy: false
	pod_security_configuration: ""
	always_pull_images: false
	secrets_encryption_config: null
	audit_log: null
	admission_configuration: null
	event_rate_limit: null
kube-controller:
	image: ""
	extra_args: {}
	extra_args_array: {}
	extra_binds: []
	extra_env: []
	win_extra_args: {}
	win_extra_args_array: {}
	win_extra_binds: []
	win_extra_env: []
	cluster_cidr: 10.42.0.0/16
	service_cluster_ip_range: 10.43.0.0/16
scheduler:
	image: ""
	extra_args: {}
	extra_args_array: {}
	extra_binds: []
	extra_env: []
	win_extra_args: {}
	win_extra_args_array: {}
	win_extra_binds: []
	win_extra_env: []
kubelet:
	image: ""
	extra_args: {}
	extra_args_array: {}
	extra_binds: []
	extra_env: []
	win_extra_args: {}
	win_extra_args_array: {}
	win_extra_binds: []
	win_extra_env: []
	cluster_domain: cluster.local
	infra_container_image: ""
	cluster_dns_server: 10.43.0.10
	fail_swap_on: false
	generate_serving_certificate: false
kubeproxy:
	image: ""
	extra_args: {}
	extra_args_array: {}
	extra_binds: []
	extra_env: []
	win_extra_args: {}
	win_extra_args_array: {}
	win_extra_binds: []
	win_extra_env: []
network:
plugin: canal
options: {}
mtu: 0
node_selector: {}
update_strategy: null
tolerations: []
authentication:
strategy: x509
sans: []
webhook: null
addons: ""
addons_include: []
system_images:
etcd: rancher/mirrored-coreos-etcd:v3.5.9
alpine: rancher/rke-tools:v0.1.96
nginx_proxy: rancher/rke-tools:v0.1.96
cert_downloader: rancher/rke-tools:v0.1.96
kubernetes_services_sidecar: rancher/rke-tools:v0.1.96
kubedns: rancher/mirrored-k8s-dns-kube-dns:1.22.28
dnsmasq: rancher/mirrored-k8s-dns-dnsmasq-nanny:1.22.28
kubedns_sidecar: rancher/mirrored-k8s-dns-sidecar:1.22.28
kubedns_autoscaler: rancher/mirrored-cluster-proportional-autoscaler:1.8.6
coredns: rancher/mirrored-coredns-coredns:1.10.1
coredns_autoscaler: rancher/mirrored-cluster-proportional-autoscaler:1.8.6
nodelocal: rancher/mirrored-k8s-dns-node-cache:1.22.28
kubernetes: rancher/hyperkube:v1.27.8-rancher2
flannel: rancher/mirrored-flannel-flannel:v0.21.4
flannel_cni: rancher/flannel-cni:v0.3.0-rancher8
calico_node: rancher/mirrored-calico-node:v3.26.3
calico_cni: rancher/calico-cni:v3.26.3-rancher1
calico_controllers: rancher/mirrored-calico-kube-controllers:v3.26.3
calico_ctl: rancher/mirrored-calico-ctl:v3.26.3
calico_flexvol: rancher/mirrored-calico-pod2daemon-flexvol:v3.26.3
canal_node: rancher/mirrored-calico-node:v3.26.3
canal_cni: rancher/calico-cni:v3.26.3-rancher1
canal_controllers: rancher/mirrored-calico-kube-controllers:v3.26.3
canal_flannel: rancher/mirrored-flannel-flannel:v0.21.4
canal_flexvol: rancher/mirrored-calico-pod2daemon-flexvol:v3.26.3
weave_node: weaveworks/weave-kube:2.8.1
weave_cni: weaveworks/weave-npc:2.8.1
pod_infra_container: rancher/mirrored-pause:3.7
ingress: rancher/nginx-ingress-controller:nginx-1.9.4-rancher1
ingress_backend: rancher/mirrored-nginx-ingress-controller-defaultbackend:1.5-rancher1
ingress_webhook: rancher/mirrored-ingress-nginx-kube-webhook-certgen:v20231011-8b53cabe0
metrics_server: rancher/mirrored-metrics-server:v0.6.3
windows_pod_infra_container: rancher/mirrored-pause:3.7
aci_cni_deploy_container: noiro/cnideploy:6.0.3.1.81c2369
aci_host_container: noiro/aci-containers-host:6.0.3.1.81c2369
aci_opflex_container: noiro/opflex:6.0.3.1.81c2369
aci_mcast_container: noiro/opflex:6.0.3.1.81c2369
aci_ovs_container: noiro/openvswitch:6.0.3.1.81c2369
aci_controller_container: noiro/aci-containers-controller:6.0.3.1.81c2369
aci_gbp_server_container: ""
aci_opflex_server_container: ""
ssh_key_path: ~/.ssh/id_rsa
ssh_cert_path: ""
ssh_agent_auth: false
authorization:
mode: rbac
options: {}
ignore_docker_version: null
enable_cri_dockerd: null
kubernetes_version: ""
private_registries: []
ingress:
provider: ""
options: {}
node_selector: {}
extra_args: {}
dns_policy: ""
extra_envs: []
extra_volumes: []
extra_volume_mounts: []
update_strategy: null
http_port: 0
https_port: 0
network_mode: ""
tolerations: []
default_backend: null
default_http_backend_priority_class_name: ""
nginx_ingress_controller_priority_class_name: ""
default_ingress_class: null
cluster_name: ""
cloud_provider:
name: ""
prefix_path: ""
win_prefix_path: ""
addon_job_timeout: 0
bastion_host:
address: ""
port: ""
user: ""
ssh_key: ""
ssh_key_path: ""
ssh_cert: ""
ssh_cert_path: ""
ignore_proxy_env_vars: false
monitoring:
provider: ""
options: {}
node_selector: {}
update_strategy: null
replicas: null
tolerations: []
metrics_server_priority_class_name: ""
restore:
restore: false
snapshot_name: ""
rotate_encryption_key: false
dns: null

We run the command "rke up"

Results: -

INFO[0000] Running RKE version: v1.5.1
INFO[0000] Initiating Kubernetes cluster
INFO[0000] [dialer] Setup tunnel for host [172.19.123.134]
INFO[0000] [dialer] Setup tunnel for host [172.19.123.132]
INFO[0000] [dialer] Setup tunnel for host [172.19.123.133]
INFO[0000] Finding container [cluster-state-deployer] on host [172.19.123.134], try #1
INFO[0000] Finding container [cluster-state-deployer] on host [172.19.123.132], try #1
INFO[0000] Finding container [cluster-state-deployer] on host [172.19.123.133], try #1
INFO[0000] [certificates] Generating CA kubernetes certificates
INFO[0000] [certificates] Generating Kubernetes API server aggregation layer requestheader client CA certificates
INFO[0000] [certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates
INFO[0000] [certificates] Generating Kubernetes API server certificates
INFO[0000] [certificates] Generating Service account token key
INFO[0000] [certificates] Generating Kube Controller certificates
INFO[0000] [certificates] Generating Kube Scheduler certificates
INFO[0001] [certificates] Generating Kube Proxy certificates
INFO[0001] [certificates] Generating Node certificate
INFO[0001] [certificates] Generating admin certificates and kubeconfig
INFO[0002] [certificates] Generating Kubernetes API server proxy client certificates
INFO[0002] [certificates] Generating kube-etcd-172-19-123-132 certificate and key
INFO[0002] [certificates] Generating kube-etcd-172-19-123-133 certificate and key
INFO[0002] [certificates] Generating kube-etcd-172-19-123-134 certificate and key
INFO[0003] Successfully Deployed state file at [./cluster.rkestate]
INFO[0003] Building Kubernetes cluster
INFO[0003] [dialer] Setup tunnel for host [172.19.123.134]
INFO[0003] [dialer] Setup tunnel for host [172.19.123.132]
INFO[0003] [dialer] Setup tunnel for host [172.19.123.133]
INFO[0003] [network] Deploying port listener containers
INFO[0003] Image [rancher/rke-tools:v0.1.96] exists on host [172.19.123.133]
INFO[0003] Image [rancher/rke-tools:v0.1.96] exists on host [172.19.123.134]
INFO[0003] Image [rancher/rke-tools:v0.1.96] exists on host [172.19.123.132]
INFO[0003] Starting container [rke-etcd-port-listener] on host [172.19.123.134], try #1
INFO[0003] Starting container [rke-etcd-port-listener] on host [172.19.123.133], try #1
INFO[0003] Starting container [rke-etcd-port-listener] on host [172.19.123.132], try #1
INFO[0003] [network] Successfully started [rke-etcd-port-listener] container on host [172.19.123.134]
INFO[0004] [network] Successfully started [rke-etcd-port-listener] container on host [172.19.123.132]
INFO[0004] [network] Successfully started [rke-etcd-port-listener] container on host [172.19.123.133]
INFO[0004] Image [rancher/rke-tools:v0.1.96] exists on host [172.19.123.133]
INFO[0004] Image [rancher/rke-tools:v0.1.96] exists on host [172.19.123.134]
INFO[0004] Image [rancher/rke-tools:v0.1.96] exists on host [172.19.123.132]
INFO[0004] Starting container [rke-cp-port-listener] on host [172.19.123.134], try #1
INFO[0004] Starting container [rke-cp-port-listener] on host [172.19.123.133], try #1
INFO[0004] [network] Successfully started [rke-cp-port-listener] container on host [172.19.123.134]
INFO[0004] [network] Successfully started [rke-cp-port-listener] container on host [172.19.123.133]
INFO[0004] Starting container [rke-cp-port-listener] on host [172.19.123.132], try #1
INFO[0005] [network] Successfully started [rke-cp-port-listener] container on host [172.19.123.132]
INFO[0005] Image [rancher/rke-tools:v0.1.96] exists on host [172.19.123.132]
INFO[0005] Image [rancher/rke-tools:v0.1.96] exists on host [172.19.123.133]
INFO[0005] Image [rancher/rke-tools:v0.1.96] exists on host [172.19.123.134]
INFO[0005] Starting container [rke-worker-port-listener] on host [172.19.123.132], try #1
INFO[0005] Starting container [rke-worker-port-listener] on host [172.19.123.134], try #1
INFO[0005] Starting container [rke-worker-port-listener] on host [172.19.123.133], try #1
INFO[0005] [network] Successfully started [rke-worker-port-listener] container on host [172.19.123.132]
INFO[0005] [network] Successfully started [rke-worker-port-listener] container on host [172.19.123.133]
INFO[0005] [network] Successfully started [rke-worker-port-listener] container on host [172.19.123.134]
INFO[0005] [network] Port listener containers deployed successfully
INFO[0005] [network] Running etcd <-> etcd port checks
INFO[0005] [network] Checking if host [172.19.123.132] can connect to host(s) [172.19.123.132 172.19.123.133 172.19.123.134] on port(s) [2379 2380], try #1
INFO[0005] [network] Checking if host [172.19.123.133] can connect to host(s) [172.19.123.132 172.19.123.133 172.19.123.134] on port(s) [2379 2380], try #1
INFO[0005] [network] Checking if host [172.19.123.134] can connect to host(s) [172.19.123.132 172.19.123.133 172.19.123.134] on port(s) [2379 2380], try #1
INFO[0005] Image [rancher/rke-tools:v0.1.96] exists on host [172.19.123.133]
INFO[0005] Image [rancher/rke-tools:v0.1.96] exists on host [172.19.123.132]
INFO[0005] Image [rancher/rke-tools:v0.1.96] exists on host [172.19.123.134]
INFO[0006] Starting container [rke-port-checker] on host [172.19.123.133], try #1
INFO[0006] Starting container [rke-port-checker] on host [172.19.123.134], try #1
INFO[0006] Starting container [rke-port-checker] on host [172.19.123.132], try #1
INFO[0006] [network] Successfully started [rke-port-checker] container on host [172.19.123.132]
INFO[0006] [network] Successfully started [rke-port-checker] container on host [172.19.123.134]
INFO[0006] Removing container [rke-port-checker] on host [172.19.123.132], try #1
INFO[0006] Removing container [rke-port-checker] on host [172.19.123.134], try #1
INFO[0006] [network] Successfully started [rke-port-checker] container on host [172.19.123.133]
INFO[0006] Removing container [rke-port-checker] on host [172.19.123.133], try #1
INFO[0006] [network] Running control plane -> etcd port checks
INFO[0006] [network] Checking if host [172.19.123.132] can connect to host(s) [172.19.123.132 172.19.123.133 172.19.123.134] on port(s) [2379], try #1
INFO[0006] [network] Checking if host [172.19.123.133] can connect to host(s) [172.19.123.132 172.19.123.133 172.19.123.134] on port(s) [2379], try #1
INFO[0006] [network] Checking if host [172.19.123.134] can connect to host(s) [172.19.123.132 172.19.123.133 172.19.123.134] on port(s) [2379], try #1
INFO[0006] Image [rancher/rke-tools:v0.1.96] exists on host [172.19.123.133]
INFO[0006] Image [rancher/rke-tools:v0.1.96] exists on host [172.19.123.132]
INFO[0006] Image [rancher/rke-tools:v0.1.96] exists on host [172.19.123.134]
INFO[0006] Starting container [rke-port-checker] on host [172.19.123.132], try #1
INFO[0006] [network] Successfully started [rke-port-checker] container on host [172.19.123.132]
INFO[0007] Removing container [rke-port-checker] on host [172.19.123.132], try #1
INFO[0007] Starting container [rke-port-checker] on host [172.19.123.133], try #1
INFO[0007] [network] Successfully started [rke-port-checker] container on host [172.19.123.133]
INFO[0007] Removing container [rke-port-checker] on host [172.19.123.133], try #1
INFO[0007] Starting container [rke-port-checker] on host [172.19.123.134], try #1
INFO[0007] [network] Successfully started [rke-port-checker] container on host [172.19.123.134]
INFO[0007] Removing container [rke-port-checker] on host [172.19.123.134], try #1
INFO[0007] [network] Running control plane -> worker port checks
INFO[0007] [network] Checking if host [172.19.123.132] can connect to host(s) [172.19.123.132 172.19.123.133 172.19.123.134] on port(s) [10250], try #1
INFO[0007] [network] Checking if host [172.19.123.133] can connect to host(s) [172.19.123.132 172.19.123.133 172.19.123.134] on port(s) [10250], try #1
INFO[0007] [network] Checking if host [172.19.123.134] can connect to host(s) [172.19.123.132 172.19.123.133 172.19.123.134] on port(s) [10250], try #1
INFO[0007] Image [rancher/rke-tools:v0.1.96] exists on host [172.19.123.134]
INFO[0007] Image [rancher/rke-tools:v0.1.96] exists on host [172.19.123.132]
INFO[0007] Image [rancher/rke-tools:v0.1.96] exists on host [172.19.123.133]
INFO[0007] Starting container [rke-port-checker] on host [172.19.123.133], try #1
INFO[0007] Starting container [rke-port-checker] on host [172.19.123.132], try #1
INFO[0007] [network] Successfully started [rke-port-checker] container on host [172.19.123.133]
INFO[0007] [network] Successfully started [rke-port-checker] container on host [172.19.123.132]
INFO[0007] Removing container [rke-port-checker] on host [172.19.123.133], try #1
INFO[0007] Removing container [rke-port-checker] on host [172.19.123.132], try #1
INFO[0008] Starting container [rke-port-checker] on host [172.19.123.134], try #1
INFO[0008] [network] Successfully started [rke-port-checker] container on host [172.19.123.134]
INFO[0008] Removing container [rke-port-checker] on host [172.19.123.134], try #1
INFO[0008] [network] Running workers -> control plane port checks
INFO[0008] [network] Checking if host [172.19.123.132] can connect to host(s) [172.19.123.132 172.19.123.133 172.19.123.134] on port(s) [6443], try #1
INFO[0008] [network] Checking if host [172.19.123.133] can connect to host(s) [172.19.123.132 172.19.123.133 172.19.123.134] on port(s) [6443], try #1
INFO[0008] [network] Checking if host [172.19.123.134] can connect to host(s) [172.19.123.132 172.19.123.133 172.19.123.134] on port(s) [6443], try #1
INFO[0008] Image [rancher/rke-tools:v0.1.96] exists on host [172.19.123.132]
INFO[0008] Image [rancher/rke-tools:v0.1.96] exists on host [172.19.123.133]
INFO[0008] Image [rancher/rke-tools:v0.1.96] exists on host [172.19.123.134]
INFO[0008] Starting container [rke-port-checker] on host [172.19.123.133], try #1
INFO[0008] [network] Successfully started [rke-port-checker] container on host [172.19.123.133]
INFO[0008] Removing container [rke-port-checker] on host [172.19.123.133], try #1
INFO[0008] Starting container [rke-port-checker] on host [172.19.123.132], try #1
INFO[0008] [network] Successfully started [rke-port-checker] container on host [172.19.123.132]
INFO[0008] Removing container [rke-port-checker] on host [172.19.123.132], try #1
INFO[0008] Starting container [rke-port-checker] on host [172.19.123.134], try #1
INFO[0008] [network] Successfully started [rke-port-checker] container on host [172.19.123.134]
INFO[0008] Removing container [rke-port-checker] on host [172.19.123.134], try #1
INFO[0008] [network] Checking KubeAPI port Control Plane hosts
INFO[0008] [network] Removing port listener containers
INFO[0008] Removing container [rke-etcd-port-listener] on host [172.19.123.132], try #1
INFO[0008] Removing container [rke-etcd-port-listener] on host [172.19.123.134], try #1
INFO[0008] Removing container [rke-etcd-port-listener] on host [172.19.123.133], try #1
INFO[0009] [remove/rke-etcd-port-listener] Successfully removed container on host [172.19.123.132]
INFO[0009] [remove/rke-etcd-port-listener] Successfully removed container on host [172.19.123.133]
INFO[0009] [remove/rke-etcd-port-listener] Successfully removed container on host [172.19.123.134]
INFO[0009] Removing container [rke-cp-port-listener] on host [172.19.123.134], try #1
INFO[0009] Removing container [rke-cp-port-listener] on host [172.19.123.133], try #1
INFO[0009] Removing container [rke-cp-port-listener] on host [172.19.123.132], try #1
INFO[0009] [remove/rke-cp-port-listener] Successfully removed container on host [172.19.123.133]
INFO[0009] [remove/rke-cp-port-listener] Successfully removed container on host [172.19.123.132]
INFO[0009] [remove/rke-cp-port-listener] Successfully removed container on host [172.19.123.134]
INFO[0009] Removing container [rke-worker-port-listener] on host [172.19.123.132], try #1
INFO[0009] Removing container [rke-worker-port-listener] on host [172.19.123.134], try #1
INFO[0009] Removing container [rke-worker-port-listener] on host [172.19.123.133], try #1
INFO[0009] [remove/rke-worker-port-listener] Successfully removed container on host [172.19.123.132]
INFO[0009] [remove/rke-worker-port-listener] Successfully removed container on host [172.19.123.133]
INFO[0009] [remove/rke-worker-port-listener] Successfully removed container on host [172.19.123.134]
INFO[0009] [network] Port listener containers removed successfully
INFO[0009] [certificates] Deploying kubernetes certificates to Cluster nodes
INFO[0009] Finding container [cert-deployer] on host [172.19.123.134], try #1
INFO[0009] Finding container [cert-deployer] on host [172.19.123.133], try #1
INFO[0009] Finding container [cert-deployer] on host [172.19.123.132], try #1
INFO[0009] Image [rancher/rke-tools:v0.1.96] exists on host [172.19.123.133]
INFO[0009] Image [rancher/rke-tools:v0.1.96] exists on host [172.19.123.132]
INFO[0009] Image [rancher/rke-tools:v0.1.96] exists on host [172.19.123.134]
INFO[0009] Starting container [cert-deployer] on host [172.19.123.132], try #1
INFO[0009] Starting container [cert-deployer] on host [172.19.123.134], try #1
INFO[0010] Finding container [cert-deployer] on host [172.19.123.134], try #1
INFO[0010] Starting container [cert-deployer] on host [172.19.123.133], try #1
INFO[0010] Finding container [cert-deployer] on host [172.19.123.132], try #1
INFO[0010] Finding container [cert-deployer] on host [172.19.123.133], try #1
INFO[0015] Finding container [cert-deployer] on host [172.19.123.134], try #1
INFO[0015] Removing container [cert-deployer] on host [172.19.123.134], try #1
INFO[0015] Finding container [cert-deployer] on host [172.19.123.132], try #1
INFO[0015] Removing container [cert-deployer] on host [172.19.123.132], try #1
INFO[0015] Finding container [cert-deployer] on host [172.19.123.133], try #1
INFO[0015] Removing container [cert-deployer] on host [172.19.123.133], try #1
INFO[0015] [reconcile] Rebuilding and updating local kube config
INFO[0015] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml]
WARN[0015] [reconcile] host [172.19.123.132] is a control plane node without reachable Kubernetes API endpoint in the cluster
INFO[0015] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml]
WARN[0015] [reconcile] host [172.19.123.133] is a control plane node without reachable Kubernetes API endpoint in the cluster
INFO[0015] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml]
WARN[0015] [reconcile] host [172.19.123.134] is a control plane node without reachable Kubernetes API endpoint in the cluster
WARN[0015] [reconcile] no control plane node with reachable Kubernetes API endpoint in the cluster found
INFO[0015] [certificates] Successfully deployed kubernetes certificates to Cluster nodes
INFO[0015] [file-deploy] Deploying file [/etc/kubernetes/admission.yaml] to node [172.19.123.132]
INFO[0015] Image [rancher/rke-tools:v0.1.96] exists on host [172.19.123.132]
INFO[0015] Starting container [file-deployer] on host [172.19.123.132], try #1
INFO[0015] Successfully started [file-deployer] container on host [172.19.123.132]
INFO[0015] Waiting for [file-deployer] container to exit on host [172.19.123.132]
INFO[0015] Waiting for [file-deployer] container to exit on host [172.19.123.132]
INFO[0015] Container [file-deployer] is still running on host [172.19.123.132]: stderr: [], stdout: []
INFO[0016] Removing container [file-deployer] on host [172.19.123.132], try #1
INFO[0016] [remove/file-deployer] Successfully removed container on host [172.19.123.132]
INFO[0016] [file-deploy] Deploying file [/etc/kubernetes/admission.yaml] to node [172.19.123.133]
INFO[0016] Image [rancher/rke-tools:v0.1.96] exists on host [172.19.123.133]
INFO[0017] Starting container [file-deployer] on host [172.19.123.133], try #1
INFO[0017] Successfully started [file-deployer] container on host [172.19.123.133]
INFO[0017] Waiting for [file-deployer] container to exit on host [172.19.123.133]
INFO[0017] Waiting for [file-deployer] container to exit on host [172.19.123.133]
INFO[0017] Container [file-deployer] is still running on host [172.19.123.133]: stderr: [], stdout: []
INFO[0018] Removing container [file-deployer] on host [172.19.123.133], try #1
INFO[0018] [remove/file-deployer] Successfully removed container on host [172.19.123.133]
INFO[0018] [file-deploy] Deploying file [/etc/kubernetes/admission.yaml] to node [172.19.123.134]
INFO[0018] Image [rancher/rke-tools:v0.1.96] exists on host [172.19.123.134]
INFO[0018] Starting container [file-deployer] on host [172.19.123.134], try #1
INFO[0019] Successfully started [file-deployer] container on host [172.19.123.134]
INFO[0019] Waiting for [file-deployer] container to exit on host [172.19.123.134]
INFO[0019] Waiting for [file-deployer] container to exit on host [172.19.123.134]
INFO[0019] Container [file-deployer] is still running on host [172.19.123.134]: stderr: [], stdout: []
INFO[0020] Removing container [file-deployer] on host [172.19.123.134], try #1
INFO[0020] [remove/file-deployer] Successfully removed container on host [172.19.123.134]
INFO[0020] [/etc/kubernetes/admission.yaml] Successfully deployed admission control config to Cluster control nodes
INFO[0020] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [172.19.123.132]
INFO[0020] Image [rancher/rke-tools:v0.1.96] exists on host [172.19.123.132]
INFO[0020] Starting container [file-deployer] on host [172.19.123.132], try #1
INFO[0020] Successfully started [file-deployer] container on host [172.19.123.132]
INFO[0020] Waiting for [file-deployer] container to exit on host [172.19.123.132]
INFO[0020] Waiting for [file-deployer] container to exit on host [172.19.123.132]
INFO[0020] Container [file-deployer] is still running on host [172.19.123.132]: stderr: [], stdout: []
INFO[0021] Removing container [file-deployer] on host [172.19.123.132], try #1
INFO[0021] [remove/file-deployer] Successfully removed container on host [172.19.123.132]
INFO[0021] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [172.19.123.133]
INFO[0021] Image [rancher/rke-tools:v0.1.96] exists on host [172.19.123.133]
INFO[0021] Starting container [file-deployer] on host [172.19.123.133], try #1
INFO[0021] Successfully started [file-deployer] container on host [172.19.123.133]
INFO[0021] Waiting for [file-deployer] container to exit on host [172.19.123.133]
INFO[0021] Waiting for [file-deployer] container to exit on host [172.19.123.133]
INFO[0021] Container [file-deployer] is still running on host [172.19.123.133]: stderr: [], stdout: []
INFO[0022] Removing container [file-deployer] on host [172.19.123.133], try #1
INFO[0023] [remove/file-deployer] Successfully removed container on host [172.19.123.133]
INFO[0023] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [172.19.123.134]
INFO[0023] Image [rancher/rke-tools:v0.1.96] exists on host [172.19.123.134]
INFO[0023] Starting container [file-deployer] on host [172.19.123.134], try #1
INFO[0023] Successfully started [file-deployer] container on host [172.19.123.134]
INFO[0023] Waiting for [file-deployer] container to exit on host [172.19.123.134]
INFO[0023] Waiting for [file-deployer] container to exit on host [172.19.123.134]
INFO[0023] Container [file-deployer] is still running on host [172.19.123.134]: stderr: [], stdout: []
INFO[0024] Removing container [file-deployer] on host [172.19.123.134], try #1
INFO[0024] [remove/file-deployer] Successfully removed container on host [172.19.123.134]
INFO[0024] [/etc/kubernetes/audit-policy.yaml] Successfully deployed audit policy file to Cluster control nodes
INFO[0024] [reconcile] Reconciling cluster state
INFO[0024] [reconcile] This is newly generated cluster
INFO[0024] Pre-pulling kubernetes images
INFO[0024] Image [rancher/hyperkube:v1.27.8-rancher2] exists on host [172.19.123.134]
INFO[0024] Image [rancher/hyperkube:v1.27.8-rancher2] exists on host [172.19.123.132]
INFO[0024] Image [rancher/hyperkube:v1.27.8-rancher2] exists on host [172.19.123.133]
INFO[0024] Image [rancher/mirrored-pause:3.7] exists on host [172.19.123.134]
INFO[0024] Image [rancher/mirrored-pause:3.7] exists on host [172.19.123.132]
INFO[0024] Image [rancher/mirrored-pause:3.7] exists on host [172.19.123.133]
INFO[0024] Kubernetes images pulled successfully
INFO[0024] [etcd] Building up etcd plane..
INFO[0024] Image [rancher/rke-tools:v0.1.96] exists on host [172.19.123.132]
INFO[0024] Starting container [etcd-fix-perm] on host [172.19.123.132], try #1
INFO[0025] Successfully started [etcd-fix-perm] container on host [172.19.123.132]
INFO[0025] Waiting for [etcd-fix-perm] container to exit on host [172.19.123.132]
INFO[0025] Waiting for [etcd-fix-perm] container to exit on host [172.19.123.132]
INFO[0025] Removing container [etcd-fix-perm] on host [172.19.123.132], try #1
INFO[0025] [remove/etcd-fix-perm] Successfully removed container on host [172.19.123.132]
INFO[0025] Image [rancher/mirrored-coreos-etcd:v3.5.9] exists on host [172.19.123.132]
INFO[0025] Starting container [etcd] on host [172.19.123.132], try #1
INFO[0025] [etcd] Successfully started [etcd] container on host [172.19.123.132]
INFO[0025] [etcd] Running rolling snapshot container [etcd-rolling-snapshots] on host [172.19.123.132]
INFO[0025] Image [rancher/rke-tools:v0.1.96] exists on host [172.19.123.132]
INFO[0025] Starting container [etcd-rolling-snapshots] on host [172.19.123.132], try #1
INFO[0025] [etcd] Successfully started [etcd-rolling-snapshots] container on host [172.19.123.132]
WARN[0031] [etcd] etcd rolling snapshot container failed to start correctly
INFO[0031] Removing container [etcd-rolling-snapshots] on host [172.19.123.132], try #1
INFO[0031] Image [rancher/rke-tools:v0.1.96] exists on host [172.19.123.132]
INFO[0031] Starting container [rke-bundle-cert] on host [172.19.123.132], try #1
INFO[0031] [certificates] Successfully started [rke-bundle-cert] container on host [172.19.123.132]
INFO[0031] Waiting for [rke-bundle-cert] container to exit on host [172.19.123.132]
INFO[0031] Container [rke-bundle-cert] is still running on host [172.19.123.132]: stderr: [], stdout: []
INFO[0032] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [172.19.123.132]
INFO[0032] Removing container [rke-bundle-cert] on host [172.19.123.132], try #1
FATA[0032] [etcd] Failed to bring up Etcd Plane: Error: No such container: etcd-rolling-snapshots

[Gopi876]

Increase the CPU

[martinat2604]

We encountered an issue while attempting to install RKE on a hardened RHEL 9.3 within our PPT environment. The PPT environment shares the same CPU and memory specifications as our LAB environment, where we successfully created a cluster using the base OS RHEL 9.3. However, despite these similarities, we encountered issues during the installation process on the hardened RHEL system.

[github-actions]

This repository uses an automated workflow to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the workflow can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the workflow will automatically close the issue in 14 days. Thank you for your contributions.