Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] apecloud-mysql cluster hscale offline instance then stop and start, pod CrashLoopBackOff #7319

Open
JashBook opened this issue May 13, 2024 · 0 comments
Open

[BUG] apecloud-mysql cluster hscale offline instance then stop and start, pod CrashLoopBackOff #7319

JashBook opened this issue May 13, 2024 · 0 comments
Assignees
@xuriwuyun
Labels
kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Milestone
Release 0.9.0

Comments

@JashBook
Copy link
Collaborator

Describe the bug
A clear and concise description of what the bug is.

kbcli version
Kubernetes: v1.27.9
KubeBlocks: 0.9.0-beta.23
kbcli: 0.9.0-beta.9

To Reproduce
Steps to reproduce the behavior:

  1. create cluster
kbcli cluster create  mysql-cluster --termination-policy=WipeOut --cluster-definition=apecloud-mysql --cluster-version=ac-mysql-8.0.30 --set cpu=500m,memory=0.5Gi,replicas=5,storage=1Gi
  1. hscale offline instance
kubectl create -f -<<EOF
apiVersion: apps.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  generateName: mysql-cluster-hscaleoffinstance-
spec:
  clusterRef: mysql-cluster
  horizontalScaling:
  - componentName: mysql
    replicas: 4
    offlineInstances: ["mysql-cluster-mysql-0"]
  ttlSecondsAfterSucceed: 0
  type: HorizontalScaling
EOF
  1. stop -- start
kbcli cluster stop mysql-cluster --auto-approve 
kbcli cluster start mysql-cluster
  1. See error
kubectl get pod -l app.kubernetes.io/instance=mysql-cluster
NAME                    READY   STATUS             RESTARTS      AGE
mysql-cluster-mysql-1   4/5     Error              5 (97s ago)   3m22s
mysql-cluster-mysql-2   4/5     Error              5 (88s ago)   3m22s
mysql-cluster-mysql-3   4/5     Error              5 (97s ago)   3m22s
mysql-cluster-mysql-4   4/5     CrashLoopBackOff   4 (91s ago)   3m22s

describe pod

kubectl describe pod mysql-cluster-mysql-1
Name:         mysql-cluster-mysql-1
Namespace:    default
Priority:     0
Node:         aks-largepool-26178435-vmss000000/10.224.0.5
Start Time:   Mon, 13 May 2024 16:50:14 +0800
Labels:       app.kubernetes.io/component=mysql
              app.kubernetes.io/instance=mysql-cluster
              app.kubernetes.io/managed-by=kubeblocks
              app.kubernetes.io/name=apecloud-mysql
              app.kubernetes.io/version=
              apps.kubeblocks.io/cluster-uid=13765ed0-d1eb-49a3-a821-6c8902997235
              apps.kubeblocks.io/component-name=mysql
              apps.kubeblocks.io/pod-name=mysql-cluster-mysql-1
              clusterdefinition.kubeblocks.io/name=apecloud-mysql
              clusterversion.kubeblocks.io/name=ac-mysql-8.0.30
              controller-revision-hash=77f5b8dd4c
              workloads.kubeblocks.io/instance=mysql-cluster-mysql
              workloads.kubeblocks.io/managed-by=InstanceSet
Annotations:  apps.kubeblocks.io/component-replicas: 4
Status:       Running
IP:           10.244.1.15
IPs:
  IP:           10.244.1.15
Controlled By:  InstanceSet/mysql-cluster-mysql
Containers:
  mysql:
    Container ID:  containerd://8d9ac9ccb5a3161caf1f4da9e434eb1826e0dcde465b76043157f9f762e7d694
    Image:         infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.beta3.20231215.ge77d836.14
    Image ID:      infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/apecloud-mysql-server@sha256:c2615eeb26e39065e8aba3983c422bb2a170282073e3842a3bf4086a05de9e9a
    Ports:         3306/TCP, 13306/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /scripts/setup.sh
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 13 May 2024 16:53:30 +0800
      Finished:     Mon, 13 May 2024 16:53:30 +0800
    Ready:          False
    Restart Count:  5
    Limits:
      cpu:     500m
      memory:  512Mi
    Requests:
      cpu:     500m
      memory:  512Mi
    Environment Variables from:
      mysql-cluster-mysql-env      ConfigMap  Optional: false
      mysql-cluster-mysql-its-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:            mysql-cluster-mysql-1 (v1:metadata.name)
      KB_POD_UID:              (v1:metadata.uid)
      KB_NAMESPACE:           default (v1:metadata.namespace)
      KB_SA_NAME:              (v1:spec.serviceAccountName)
      KB_NODENAME:             (v1:spec.nodeName)
      KB_HOST_IP:              (v1:status.hostIP)
      KB_POD_IP:               (v1:status.podIP)
      KB_POD_IPS:              (v1:status.podIPs)
      KB_HOSTIP:               (v1:status.hostIP)
      KB_PODIP:                (v1:status.podIP)
      KB_PODIPS:               (v1:status.podIPs)
      KB_POD_FQDN:            $(KB_POD_NAME).mysql-cluster-mysql-headless.$(KB_NAMESPACE).svc
      SERVICE_PORT:           3306
      MYSQL_ROOT_HOST:        %
      MYSQL_ROOT_USER:        <set to the key 'username' in secret 'mysql-cluster-conn-credential'>  Optional: false
      MYSQL_ROOT_PASSWORD:    <set to the key 'password' in secret 'mysql-cluster-conn-credential'>  Optional: false
      MYSQL_DATABASE:         mydb
      MYSQL_USER:             u1
      MYSQL_PASSWORD:         u1
      CLUSTER_ID:             1
      CLUSTER_START_INDEX:    1
      REPLICATION_USER:       replicator
      REPLICATION_PASSWORD:   
      MYSQL_TEMPLATE_CONFIG:  
      MYSQL_CUSTOM_CONFIG:    
      MYSQL_DYNAMIC_CONFIG:   
      KB_EMBEDDED_WESQL:      1
    Mounts:
      /data/mysql from data (rw)
      /etc/annotations from annotations (rw)
      /opt/mysql from mysql-config (rw)
      /scripts from scripts (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wlkxb (ro)
  vttablet:
    Container ID:  containerd://60fbcd087552bb84c5501c03e87f4e4d7395e6ef79e7eff2644cddd39db6ceb1
    Image:         infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/apecloud-mysql-scale:0.2.7
    Image ID:      infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/apecloud-mysql-scale@sha256:0c39da3d1e542661e7783c125a9560614911de466532b9d1b6b06c66183bfedf
    Ports:         15100/TCP, 16100/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /scripts/vttablet.sh
    State:          Running
      Started:      Mon, 13 May 2024 16:50:25 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     0
      memory:  0
    Requests:
      cpu:     0
      memory:  0
    Environment Variables from:
      mysql-cluster-mysql-env      ConfigMap  Optional: false
      mysql-cluster-mysql-its-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:          mysql-cluster-mysql-1 (v1:metadata.name)
      KB_POD_UID:            (v1:metadata.uid)
      KB_NAMESPACE:         default (v1:metadata.namespace)
      KB_SA_NAME:            (v1:spec.serviceAccountName)
      KB_NODENAME:           (v1:spec.nodeName)
      KB_HOST_IP:            (v1:status.hostIP)
      KB_POD_IP:             (v1:status.podIP)
      KB_POD_IPS:            (v1:status.podIPs)
      KB_HOSTIP:             (v1:status.hostIP)
      KB_PODIP:              (v1:status.podIP)
      KB_PODIPS:             (v1:status.podIPs)
      KB_POD_FQDN:          $(KB_POD_NAME).mysql-cluster-mysql-headless.$(KB_NAMESPACE).svc
      CELL:                 zone1
      ETCD_SERVER:          $(KB_CLUSTER_NAME)-vtcontroller-headless
      ETCD_PORT:            2379
      TOPOLOGY_FLAGS:       --topo_implementation etcd2 --topo_global_server_address $(ETCD_SERVER):$(ETCD_PORT) --topo_global_root /vitess/global
      VTTABLET_PORT:        15100
      VTTABLET_GRPC_PORT:   16100
      VTCTLD_HOST:          $(KB_CLUSTER_NAME)-vtcontroller-headless
      VTCTLD_WEB_PORT:      15000
      SERVICE_PORT:         $(VTTABLET_PORT)
      MYSQL_ROOT_USER:      <set to the key 'username' in secret 'mysql-cluster-conn-credential'>  Optional: false
      MYSQL_ROOT_PASSWORD:  <set to the key 'password' in secret 'mysql-cluster-conn-credential'>  Optional: false
    Mounts:
      /conf from mysql-scale-config (rw)
      /scripts from scripts (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wlkxb (ro)
      /vtdataroot from data (rw)
  mysql-exporter:
    Container ID:  containerd://510a594a013bf4ea4a2a21930ad5a4e993c0f9964ab19fdf48b8fb5af69ac685
    Image:         docker.io/bitnami/mysqld-exporter:0.15.1
    Image ID:      docker.io/bitnami/mysqld-exporter@sha256:8ef6beaf6c77157e679e237fad34e08730156c7e5b24d447500d530b7489d9ad
    Port:          9104/TCP
    Host Port:     0/TCP
    Command:
      /scripts/exporter_start.sh
    State:          Running
      Started:      Mon, 13 May 2024 16:50:26 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     0
      memory:  0
    Requests:
      cpu:     0
      memory:  0
    Environment Variables from:
      mysql-cluster-mysql-env      ConfigMap  Optional: false
      mysql-cluster-mysql-its-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:               mysql-cluster-mysql-1 (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:              default (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_POD_FQDN:               $(KB_POD_NAME).mysql-cluster-mysql-headless.$(KB_NAMESPACE).svc
      MYSQLD_EXPORTER_USER:      <set to the key 'username' in secret 'mysql-cluster-conn-credential'>  Optional: false
      MYSQLD_EXPORTER_PASSWORD:  <set to the key 'password' in secret 'mysql-cluster-conn-credential'>  Optional: false
      EXPORTER_WEB_PORT:         9104
    Mounts:
      /scripts from scripts (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wlkxb (ro)
  lorry:
    Container ID:  containerd://2145c3ed229b436e67ab5c06c0b46b2411627dfe977eb235790cc2eecae6cf0d
    Image:         docker.io/apecloud/kubeblocks-tools:0.9.0-beta.23
    Image ID:      docker.io/apecloud/kubeblocks-tools@sha256:390128fc76a201d0ef49402c2e15e7160cc35b28000950edadf1da8dae6e13c6
    Ports:         3501/TCP, 50001/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      lorry
      --port
      3501
      --grpcport
      50001
    State:          Running
      Started:      Mon, 13 May 2024 16:50:26 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     0
      memory:  0
    Requests:
      cpu:      0
      memory:   0
    Readiness:  http-get http://:3501/v1.0/checkrole delay=0s timeout=1s period=1s #success=1 #failure=3
    Startup:    tcp-socket :3501 delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      mysql-cluster-mysql-env      ConfigMap  Optional: false
      mysql-cluster-mysql-its-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:                   mysql-cluster-mysql-1 (v1:metadata.name)
      KB_POD_UID:                     (v1:metadata.uid)
      KB_NAMESPACE:                  default (v1:metadata.namespace)
      KB_SA_NAME:                     (v1:spec.serviceAccountName)
      KB_NODENAME:                    (v1:spec.nodeName)
      KB_HOST_IP:                     (v1:status.hostIP)
      KB_POD_IP:                      (v1:status.podIP)
      KB_POD_IPS:                     (v1:status.podIPs)
      KB_HOSTIP:                      (v1:status.hostIP)
      KB_PODIP:                       (v1:status.podIP)
      KB_PODIPS:                      (v1:status.podIPs)
      KB_POD_FQDN:                   $(KB_POD_NAME).mysql-cluster-mysql-headless.$(KB_NAMESPACE).svc
      KB_RSM_ROLE_PROBE_PERIOD:      1
      KB_BUILTIN_HANDLER:            wesql
      KB_SERVICE_USER:               <set to the key 'username' in secret 'mysql-cluster-conn-credential'>  Optional: false
      KB_SERVICE_PASSWORD:           <set to the key 'password' in secret 'mysql-cluster-conn-credential'>  Optional: false
      KB_SERVICE_PORT:               3306
      KB_DATA_PATH:                  /data/mysql
      KB_RSM_ACTION_SVC_LIST:        null
      KB_RSM_ROLE_UPDATE_MECHANISM:  DirectAPIServerEventUpdate
      KB_RSM_ROLE_PROBE_TIMEOUT:     1
      KB_CLUSTER_NAME:                (v1:metadata.labels['app.kubernetes.io/instance'])
      KB_COMP_NAME:                   (v1:metadata.labels['apps.kubeblocks.io/component-name'])
      KB_SERVICE_CHARACTER_TYPE:     wesql
    Mounts:
      /data/mysql from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wlkxb (ro)
  config-manager:
    Container ID:  containerd://dac09636cc3b2398c7fdca730eae8b746e61adad620135626ee8b28762c0bab5
    Image:         docker.io/apecloud/kubeblocks-tools:0.9.0-beta.23
    Image ID:      docker.io/apecloud/kubeblocks-tools@sha256:390128fc76a201d0ef49402c2e15e7160cc35b28000950edadf1da8dae6e13c6
    Port:          <none>
    Host Port:     <none>
    Command:
      env
    Args:
      PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$(TOOLS_PATH)
      /bin/reloader
      --log-level
      info
      --operator-update-enable
      --tcp
      9901
      --config
      /opt/config-manager/config-manager.yaml
    State:          Running
      Started:      Mon, 13 May 2024 16:50:26 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     0
      memory:  0
    Requests:
      cpu:     0
      memory:  0
    Environment Variables from:
      mysql-cluster-mysql-env      ConfigMap  Optional: false
      mysql-cluster-mysql-its-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:            mysql-cluster-mysql-1 (v1:metadata.name)
      KB_POD_UID:              (v1:metadata.uid)
      KB_NAMESPACE:           default (v1:metadata.namespace)
      KB_SA_NAME:              (v1:spec.serviceAccountName)
      KB_NODENAME:             (v1:spec.nodeName)
      KB_HOST_IP:              (v1:status.hostIP)
      KB_POD_IP:               (v1:status.podIP)
      KB_POD_IPS:              (v1:status.podIPs)
      KB_HOSTIP:               (v1:status.hostIP)
      KB_PODIP:                (v1:status.podIP)
      KB_PODIPS:               (v1:status.podIPs)
      KB_POD_FQDN:            $(KB_POD_NAME).mysql-cluster-mysql-headless.$(KB_NAMESPACE).svc
      CONFIG_MANAGER_POD_IP:   (v1:status.podIP)
      DB_TYPE:                mysql
      MYSQL_USER:             <set to the key 'username' in secret 'mysql-cluster-conn-credential'>  Optional: false
      MYSQL_PASSWORD:         <set to the key 'password' in secret 'mysql-cluster-conn-credential'>  Optional: false
      DATA_SOURCE_NAME:       $(MYSQL_USER):$(MYSQL_PASSWORD)@(localhost:3306)/
      TOOLS_PATH:             /opt/kb-tools/reload/mysql-consensusset-config:/opt/config-manager
    Mounts:
      /conf from mysql-scale-config (rw)
      /opt/config-manager from config-manager-config (rw)
      /opt/kb-tools/reload/mysql-consensusset-config from cm-script-mysql-consensusset-config (rw)
      /opt/mysql from mysql-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wlkxb (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  annotations:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.annotations['cs.apps.kubeblocks.io/leader'] -> leader
      metadata.annotations['apps.kubeblocks.io/component-replicas'] -> component-replicas
  scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      mysql-cluster-mysql-apecloud-mysql-scripts
    Optional:  false
  mysql-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      mysql-cluster-mysql-mysql-consensusset-config
    Optional:  false
  mysql-scale-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      mysql-cluster-mysql-vttablet-config
    Optional:  false
  cm-script-mysql-consensusset-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      sidecar-mysql-reload-script-mysql-cluster
    Optional:  false
  config-manager-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      sidecar-mysql-cluster-mysql-config-manager-config
    Optional:  false
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-mysql-cluster-mysql-1
    ReadOnly:   false
  kube-api-access-wlkxb:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 kb-data=true:NoSchedule
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                    From                     Message
  ----     ------                  ----                   ----                     -------
  Normal   Scheduled               3m35s                  default-scheduler        Successfully assigned default/mysql-cluster-mysql-1 to aks-largepool-26178435-vmss000000
  Normal   Created                 3m25s                  kubelet                  Created container vttablet
  Normal   SuccessfulAttachVolume  3m25s                  attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-40c70beb-9eff-4cc2-b26a-970dce8e8e15"
  Normal   Pulled                  3m25s                  kubelet                  Container image "docker.io/bitnami/mysqld-exporter:0.15.1" already present on machine
  Normal   Started                 3m25s                  kubelet                  Started container vttablet
  Normal   Pulled                  3m25s                  kubelet                  Container image "infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/apecloud-mysql-scale:0.2.7" already present on machine
  Normal   Created                 3m25s                  kubelet                  Created container mysql-exporter
  Normal   Started                 3m24s                  kubelet                  Started container lorry
  Normal   Pulled                  3m24s                  kubelet                  Container image "docker.io/apecloud/kubeblocks-tools:0.9.0-beta.23" already present on machine
  Normal   Created                 3m24s                  kubelet                  Created container config-manager
  Normal   Started                 3m24s                  kubelet                  Started container mysql-exporter
  Normal   Pulled                  3m24s                  kubelet                  Container image "docker.io/apecloud/kubeblocks-tools:0.9.0-beta.23" already present on machine
  Normal   Created                 3m24s                  kubelet                  Created container lorry
  Normal   Started                 3m24s                  kubelet                  Started container config-manager
  Normal   Started                 3m23s (x2 over 3m25s)  kubelet                  Started container mysql
  Warning  BackOff                 3m13s (x5 over 3m22s)  kubelet                  Back-off restarting failed container mysql in pod mysql-cluster-mysql-1_default(13b29a97-6031-4cb6-9435-23b8035ef694)
  Normal   Created                 2m58s (x3 over 3m25s)  kubelet                  Created container mysql
  Normal   Pulled                  2m58s (x3 over 3m25s)  kubelet                  Container image "infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.beta3.20231215.ge77d836.14" already present on machine
➜  ~

logs pod

kubectl logs mysql-cluster-mysql-1 mysql
+ rmdir /docker-entrypoint-initdb.d
+ mkdir -p /data/mysql/auditlog
+ mkdir -p /data/mysql/binlog
+ mkdir -p /data/mysql/docker-entrypoint-initdb.d
+ ln -s /data/mysql/docker-entrypoint-initdb.d /docker-entrypoint-initdb.d
+ generate_cluster_info
+ local pod_name=mysql-cluster-mysql-1
+ local cluster_members=
+ export MYSQL_PORT=3306
+ MYSQL_PORT=3306
+ export MYSQL_CONSENSUS_PORT=13306
+ MYSQL_CONSENSUS_PORT=13306
+ export KB_MYSQL_VOLUME_DIR=/data/mysql/
+ KB_MYSQL_VOLUME_DIR=/data/mysql/
+ export KB_MYSQL_CONF_FILE=/opt/mysql/my.cnf
+ KB_MYSQL_CONF_FILE=/opt/mysql/my.cnf
+ '[' -z 4 ']'
+ echo KB_MYSQL_N=4
+ '[' -z 315ecb61-8f82-4ebb-84bc-49660b80398b ']'
KB_MYSQL_N=4
+ echo KB_MYSQL_CLUSTER_UID=315ecb61-8f82-4ebb-84bc-49660b80398b
+ (( i = 0 ))
+ (( i < KB_REPLICA_COUNT ))
KB_MYSQL_CLUSTER_UID=315ecb61-8f82-4ebb-84bc-49660b80398b
+ '[' 0 -gt 0 ']'
++ get_replica_host 0
++ local id=0
++ '[' -z '' ']'
++ host=KB_0_HOSTNAME
++ echo ''
+ host=
/scripts/setup.sh: line 50: host: missing member hostname
➜  ~ 
➜  ~ kubectl logs mysql-cluster-mysql-2 mysql
+ rmdir /docker-entrypoint-initdb.d
+ mkdir -p /data/mysql/auditlog
+ mkdir -p /data/mysql/binlog
+ mkdir -p /data/mysql/docker-entrypoint-initdb.d
+ ln -s /data/mysql/docker-entrypoint-initdb.d /docker-entrypoint-initdb.d
KB_MYSQL_N=4
KB_MYSQL_CLUSTER_UID=315ecb61-8f82-4ebb-84bc-49660b80398b
+ generate_cluster_info
+ local pod_name=mysql-cluster-mysql-2
+ local cluster_members=
+ export MYSQL_PORT=3306
+ MYSQL_PORT=3306
+ export MYSQL_CONSENSUS_PORT=13306
+ MYSQL_CONSENSUS_PORT=13306
+ export KB_MYSQL_VOLUME_DIR=/data/mysql/
+ KB_MYSQL_VOLUME_DIR=/data/mysql/
+ export KB_MYSQL_CONF_FILE=/opt/mysql/my.cnf
+ KB_MYSQL_CONF_FILE=/opt/mysql/my.cnf
+ '[' -z 4 ']'
+ echo KB_MYSQL_N=4
+ '[' -z 315ecb61-8f82-4ebb-84bc-49660b80398b ']'
+ echo KB_MYSQL_CLUSTER_UID=315ecb61-8f82-4ebb-84bc-49660b80398b
+ (( i = 0 ))
+ (( i < KB_REPLICA_COUNT ))
+ '[' 0 -gt 0 ']'
++ get_replica_host 0
++ local id=0
++ '[' -z '' ']'
++ host=KB_0_HOSTNAME
++ echo ''
+ host=
/scripts/setup.sh: line 50: host: missing member hostname
➜  ~ 
➜  ~ kubectl logs mysql-cluster-mysql-3 mysql
+ rmdir /docker-entrypoint-initdb.d
+ mkdir -p /data/mysql/auditlog
+ mkdir -p /data/mysql/binlog
+ mkdir -p /data/mysql/docker-entrypoint-initdb.d
+ ln -s /data/mysql/docker-entrypoint-initdb.d /docker-entrypoint-initdb.d
+ generate_cluster_info
+ local pod_name=mysql-cluster-mysql-3
+ local cluster_members=
+ export MYSQL_PORT=3306
+ MYSQL_PORT=3306
+ export MYSQL_CONSENSUS_PORT=13306
+ MYSQL_CONSENSUS_PORT=13306
+ export KB_MYSQL_VOLUME_DIR=/data/mysql/
+ KB_MYSQL_VOLUME_DIR=/data/mysql/
+ export KB_MYSQL_CONF_FILE=/opt/mysql/my.cnf
+ KB_MYSQL_CONF_FILE=/opt/mysql/my.cnf
+ '[' -z 4 ']'
+ echo KB_MYSQL_N=4
+ '[' -z 315ecb61-8f82-4ebb-84bc-49660b80398b ']'
+ echo KB_MYSQL_CLUSTER_UID=315ecb61-8f82-4ebb-84bc-49660b80398b
+ (( i = 0 ))
+ (( i < KB_REPLICA_COUNT ))
+ '[' 0 -gt 0 ']'
KB_MYSQL_N=4
KB_MYSQL_CLUSTER_UID=315ecb61-8f82-4ebb-84bc-49660b80398b
++ get_replica_host 0
++ local id=0
++ '[' -z '' ']'
++ host=KB_0_HOSTNAME
++ echo ''
+ host=
/scripts/setup.sh: line 50: host: missing member hostname
➜  ~ 
➜  ~ kubectl logs mysql-cluster-mysql-4 mysql
+ rmdir /docker-entrypoint-initdb.d
+ mkdir -p /data/mysql/auditlog
+ mkdir -p /data/mysql/binlog
+ mkdir -p /data/mysql/docker-entrypoint-initdb.d
+ ln -s /data/mysql/docker-entrypoint-initdb.d /docker-entrypoint-initdb.d
KB_MYSQL_N=4
KB_MYSQL_CLUSTER_UID=315ecb61-8f82-4ebb-84bc-49660b80398b
+ generate_cluster_info
+ local pod_name=mysql-cluster-mysql-4
+ local cluster_members=
+ export MYSQL_PORT=3306
+ MYSQL_PORT=3306
+ export MYSQL_CONSENSUS_PORT=13306
+ MYSQL_CONSENSUS_PORT=13306
+ export KB_MYSQL_VOLUME_DIR=/data/mysql/
+ KB_MYSQL_VOLUME_DIR=/data/mysql/
+ export KB_MYSQL_CONF_FILE=/opt/mysql/my.cnf
+ KB_MYSQL_CONF_FILE=/opt/mysql/my.cnf
+ '[' -z 4 ']'
+ echo KB_MYSQL_N=4
+ '[' -z 315ecb61-8f82-4ebb-84bc-49660b80398b ']'
+ echo KB_MYSQL_CLUSTER_UID=315ecb61-8f82-4ebb-84bc-49660b80398b
+ (( i = 0 ))
+ (( i < KB_REPLICA_COUNT ))
+ '[' 0 -gt 0 ']'
++ get_replica_host 0
++ local id=0
++ '[' -z '' ']'
++ host=KB_0_HOSTNAME
++ echo ''
+ host=
/scripts/setup.sh: line 50: host: missing member hostname

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.
@JashBook JashBook added the kind/bug Something isn't working label May 13, 2024
@JashBook JashBook added this to the Release 0.9.0 milestone May 13, 2024
@JashBook JashBook added the severity/major Great chance user will encounter the same problem label May 13, 2024
@JashBook JashBook changed the title [BUG] apecloud-mysql cluster hscale offline instance then stop and start, pod CrashLoopBackOff kind/bug [BUG] apecloud-mysql cluster hscale offline instance then stop and start, pod CrashLoopBackOff May 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@xuriwuyun xuriwuyun

Labels
kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Projects
None yet
Milestone
Release 0.9.0
Development

No branches or pull requests
2 participants
@xuriwuyun @JashBook