Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cannot be started “statefulset.apps/rec. ” #249

Open
zhenyu-lan opened this issue Oct 17, 2022 · 1 comment
Open

Cannot be started “statefulset.apps/rec. ” #249

zhenyu-lan opened this issue Oct 17, 2022 · 1 comment

Comments

@zhenyu-lan
Copy link

zhenyu-lan commented Oct 17, 2022 •
edited 

[zhenyu.lan-a@----s` ~]$ kubectl  get all
NAME                                             READY   STATUS    RESTARTS   AGE
pod/rec-0                                        1/2     Running   0          2m37s
pod/rec-services-rigger-6ddcd5796f-5tcjc         1/1     Running   0          4d18h
pod/redis-enterprise-operator-69d566468c-m2p6k   3/3     Running   4          4d18h

NAME                TYPE           CLUSTER-IP                         EXTERNAL-IP   PORT(S)             AGE
service/admission   ClusterIP      240b:c0e0:202:554f:b414:2:0:cd4e   <none>        443/TCP             4d18h
service/rec         ClusterIP      None                               <none>        9443/TCP,8001/TCP   4d18h
service/rec-prom    ClusterIP      None                               <none>        8070/TCP            4d18h
service/rec-ui      LoadBalancer   240b:c0e0:202:554f:b414:2:0:dfc2   <pending>     8443:31981/TCP      4d18h

NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/rec-services-rigger         1/1     1            1           4d18h
deployment.apps/redis-enterprise-operator   1/1     1            1           4d18h

NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/rec-services-rigger-6ddcd5796f         1         1         1       4d18h
replicaset.apps/redis-enterprise-operator-69d566468c   1         1         1       4d18h

NAME                   READY   AGE
statefulset.apps/rec   0/3     4d18h


[zhenyu.lan-a@uhn5---t-s ~]$ kubectl  logs rec-0 redis-enterprise-node -f
Generating RSA private key, 4096 bit long modulus (2 primes)
.....++++
................................++++
e is 65537 (0x010001)
Can't load /opt/redislabs/.rnd into RNG
140321274876352:error:2406F079:random number generator:RAND_load_file:Cannot open file:../crypto/rand/randfile.c:88:Filename=/opt/redislabs/.rnd
Signature ok
subject=CN = rec-0.rec.agile.svc.cluster.local, OU = Admin UI, O = RedisLabs Enterprise Cluster
Getting Private key
Generating RSA private key, 4096 bit long modulus (2 primes)
..................................................................................++++
.......................................................................................++++
e is 65537 (0x010001)
Can't load /opt/redislabs/.rnd into RNG
140394671956416:error:2406F079:random number generator:RAND_load_file:Cannot open file:../crypto/rand/randfile.c:88:Filename=/opt/redislabs/.rnd
Signature ok
subject=CN = rec-0.rec.agile.svc.cluster.local, OU = Cluster API, O = RedisLabs Enterprise Cluster
Getting Private key
Generating RSA private key, 4096 bit long modulus (2 primes)
........................................................................................................++++
........................................................................++++
e is 65537 (0x010001)
Can't load /opt/redislabs/.rnd into RNG
139849269596608:error:2406F079:random number generator:RAND_load_file:Cannot open file:../crypto/rand/randfile.c:88:Filename=/opt/redislabs/.rnd
Signature ok
subject=CN = rec-0.rec.agile.svc.cluster.local, OU = redisdb, O = RedisLabs Enterprise Cluster
Getting Private key
Generating RSA private key, 4096 bit long modulus (2 primes)
.................................................................................................................................................................................................................................................................................++++
........................................................................++++
e is 65537 (0x010001)
Can't load /opt/redislabs/.rnd into RNG
139763628327360:error:2406F079:random number generator:RAND_load_file:Cannot open file:../crypto/rand/randfile.c:88:Filename=/opt/redislabs/.rnd
Signature ok
subject=CN = rec-0.rec.agile.svc.cluster.local, OU = metrics exporter, O = RedisLabs Enterprise Cluster
Getting Private key
Generating RSA private key, 4096 bit long modulus (2 primes)
.................................................++++
...................................................................++++
e is 65537 (0x010001)
Can't load /opt/redislabs/.rnd into RNG
139991659815360:error:2406F079:random number generator:RAND_load_file:Cannot open file:../crypto/rand/randfile.c:88:Filename=/opt/redislabs/.rnd
Signature ok
subject=CN = rec-0.rec.agile.svc.cluster.local, OU = syncer, O = RedisLabs Enterprise Cluster
Getting Private key
Generating RSA private key, 4096 bit long modulus (2 primes)
...++++
....................................................................................................................................++++
e is 65537 (0x010001)
Can't load /opt/redislabs/.rnd into RNG
140530423501248:error:2406F079:random number generator:RAND_load_file:Cannot open file:../crypto/rand/randfile.c:88:Filename=/opt/redislabs/.rnd
Signature ok
subject=CN = node.mesh_ca_signed, OU = mesh, O = RedisLabs Enterprise Cluster
Getting Private key
Generating RSA private key, 4096 bit long modulus (2 primes)
........++++
..++++
e is 65537 (0x010001)
Can't load /opt/redislabs/.rnd into RNG
139994297110976:error:2406F079:random number generator:RAND_load_file:Cannot open file:../crypto/rand/randfile.c:88:Filename=/opt/redislabs/.rnd
Signature ok
subject=CN = node.gossip_ca_signed, OU = gossip, O = RedisLabs Enterprise Cluster
Getting Private key
Generating RSA private key, 4096 bit long modulus (2 primes)
...................................................................++++
........................................................................++++
e is 65537 (0x010001)
Can't load /opt/redislabs/.rnd into RNG
139931683258816:error:2406F079:random number generator:RAND_load_file:Cannot open file:../crypto/rand/randfile.c:88:Filename=/opt/redislabs/.rnd
Signature ok
subject=CN = node.ccs_internode_encryption, OU = ccs_internode_encryption, O = RedisLabs Enterprise Cluster
Getting Private key
Generating RSA private key, 4096 bit long modulus (2 primes)
...........................................................................................................................................................................................................................++++
...................................................++++
e is 65537 (0x010001)
Can't load /opt/redislabs/.rnd into RNG
140524452495808:error:2406F079:random number generator:RAND_load_file:Cannot open file:../crypto/rand/randfile.c:88:Filename=/opt/redislabs/.rnd
Signature ok
subject=CN = node.data_internode_encryption, OU = Data plane internode encryption, O = RedisLabs Enterprise Cluster
Getting Private key
Run command: ['/opt/redislabs/sbin/generate_cert.sh', '--type', 'cm', '--org', 'Admin UI']
Run command: ['/opt/redislabs/sbin/generate_cert.sh', '--type', 'api', '--org', 'Cluster API']
Run command: ['/opt/redislabs/sbin/generate_cert.sh', '--type', 'proxy', '--org', 'redisdb']
Run command: ['/opt/redislabs/sbin/generate_cert.sh', '--type', 'metrics_exporter', '--org', 'metrics exporter']
Run command: ['/opt/redislabs/sbin/generate_cert.sh', '--type', 'syncer', '--org', 'syncer']
Run command: ['/opt/redislabs/sbin/generate_cert.sh', '--type', 'mesh_ca_signed', '--org', 'mesh', '--CN', 'node.mesh_ca_signed', '--days', '90']
Run command: ['/opt/redislabs/sbin/generate_cert.sh', '--type', 'gossip_ca_signed', '--org', 'gossip', '--CN', 'node.gossip_ca_signed', '--days', '90']
Run command: ['/opt/redislabs/sbin/generate_cert.sh', '--type', 'ccs_internode_encryption', '--org', 'ccs_internode_encryption', '--CN', 'node.ccs_internode_encryption', '--days', '90']
Run command: ['/opt/redislabs/sbin/generate_cert.sh', '--type', 'data_internode_encryption', '--org', 'Data plane internode encryption', '--CN', 'node.data_internode_encryption', '--days', '90']
Done.
/opt/redislabs/sbin/supervisord_prestart_script.sh: line 42: type: getenforce: not found
2022-10-17 02:15:41,631 INFO Included extra file "/etc/opt/redislabs/supervisord.conf.d/supervisord_cm_server.conf" during parsing
2022-10-17 02:15:41,633 INFO Included extra file "/etc/opt/redislabs/supervisord.conf.d/supervisord_crdb_coordinator.conf" during parsing
2022-10-17 02:15:41,633 INFO Included extra file "/etc/opt/redislabs/supervisord.conf.d/supervisord_crdb_worker.conf" during parsing
2022-10-17 02:15:41,633 INFO Included extra file "/etc/opt/redislabs/supervisord.conf.d/supervisord_mdns_server.conf" during parsing
2022-10-17 02:15:41,633 INFO Included extra file "/etc/opt/redislabs/supervisord.conf.d/supervisord_pdns_server.conf" during parsing
2022-10-17 02:15:41,633 INFO Included extra file "/etc/opt/redislabs/supervisord.conf.d/supervisord_saslauthd.conf" during parsing
2022-10-17 02:15:41,633 INFO Included extra file "/etc/opt/redislabs/supervisord.conf.d/supervisord_stats_archiver.conf" during parsing
2022-10-17 02:15:41,698 INFO __main__ MainThread: Waiting for REST API to be accessible
2022-10-17 02:15:41,698 INFO RPC interface 'supervisor' initialized
2022-10-17 02:15:41,699 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2022-10-17 02:15:41,699 INFO supervisord started with pid 173
2022-10-17 02:15:42,703 INFO spawned: 'ccs' with pid 182
2022-10-17 02:15:42,707 INFO spawned: 'cluster_wd' with pid 183
2022-10-17 02:15:42,712 INFO spawned: 'envoy' with pid 184
2022-10-17 02:15:42,716 INFO spawned: 'envoy_control_plane' with pid 185
2022-10-17 02:15:42,720 INFO spawned: 'gossip_envoy' with pid 186
2022-10-17 02:15:42,725 INFO spawned: 'heartbeatd' with pid 187
2022-10-17 02:15:42,728 INFO spawned: 'ldap_agent_mgr' with pid 188
2022-10-17 02:15:42,730 INFO spawned: 'alert_mgr' with pid 189
2022-10-17 02:15:42,733 INFO spawned: 'cnm_exec' with pid 190
2022-10-17 02:15:42,776 INFO spawned: 'cnm_http' with pid 197
2022-10-17 02:15:42,780 INFO spawned: 'crdb_coordinator' with pid 198
2022-10-17 02:15:42,872 INFO spawned: 'crdb_worker_0' with pid 199
2022-10-17 02:15:42,876 INFO spawned: 'crdb_worker_1' with pid 206
2022-10-17 02:15:42,880 INFO spawned: 'pdns_ccs_backend' with pid 214
2022-10-17 02:15:42,973 INFO spawned: 'redis_mgr' with pid 215
2022-10-17 02:15:42,978 INFO spawned: 'resource_mgr' with pid 216
2022-10-17 02:15:43,071 INFO spawned: 'rl_info_provider' with pid 217
2022-10-17 02:15:43,118 INFO spawned: 'stats_archiver' with pid 219
2022-10-17 02:15:43,174 INFO spawned: 'cm_server' with pid 220
2022-10-17 02:15:43,178 INFO spawned: 'dmcproxy' with pid 222
2022-10-17 02:15:43,271 INFO spawned: 'mdns_server' with pid 223
2022-10-17 02:15:43,274 INFO spawned: 'pdns_server' with pid 224
2022-10-17 02:15:43,371 INFO spawned: 'saslauthd' with pid 225
2022-10-17 02:15:43,400 INFO spawned: 'sentinel_service' with pid 226
2022-10-17 02:15:43,474 INFO spawned: 'job_scheduler' with pid 227
2022-10-17 02:15:43,573 INFO spawned: 'metrics_exporter' with pid 228
2022-10-17 02:15:43,577 INFO spawned: 'node_wd' with pid 229
2022-10-17 02:15:43,677 INFO spawned: 'start_redis_servers' with pid 232
2022-10-17 02:15:43,782 INFO success: ccs entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:43,783 INFO success: cluster_wd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:43,783 INFO success: envoy entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:43,783 INFO success: envoy_control_plane entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:43,783 INFO success: gossip_envoy entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:43,783 INFO success: heartbeatd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:43,783 INFO success: ldap_agent_mgr entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:43,783 INFO success: alert_mgr entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:43,783 INFO success: cnm_exec entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:43,783 INFO success: cnm_http entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:43,783 INFO success: crdb_coordinator entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:43,871 INFO success: crdb_worker_0 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:43,872 INFO success: start_redis_servers entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2022-10-17 02:15:44,071 INFO success: crdb_worker_1 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:44,071 INFO success: pdns_ccs_backend entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:44,071 INFO success: redis_mgr entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:44,071 INFO success: resource_mgr entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:44,071 INFO success: rl_info_provider entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:44,072 INFO success: stats_archiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:44,175 INFO success: cm_server entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:44,175 INFO success: dmcproxy entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:44,272 INFO success: mdns_server entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:44,273 INFO success: pdns_server entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:44,680 INFO success: saslauthd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:44,680 INFO success: sentinel_service entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:44,680 INFO success: metrics_exporter entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:44,680 INFO success: node_wd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:53,972 INFO success: job_scheduler entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2022-10-17 02:18:03,445 INFO __main__ MainThread: Done, moving to bootstrapping
2022-10-17 02:18:03,447 INFO __main__ MainThread: Bootstrapping node with action 'None'
2022-10-17 02:18:03,448 INFO __main__ MainThread: Done
@laurentdroin
Copy link
Contributor

There could be many reasons for this.
Can you generate a diagnostic package using the log collector script (https://docs.redis.com/latest/kubernetes/logs/collect-logs/) and open a Support ticket with Redis?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@laurentdroin @zhenyu-lan