Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] tdengine cluster hscale offline instance then stop and start, pod CrashLoopBackOff #7320

Closed
JashBook opened this issue May 13, 2024 · 1 comment
Closed

[BUG] tdengine cluster hscale offline instance then stop and start, pod CrashLoopBackOff #7320

JashBook opened this issue May 13, 2024 · 1 comment
Assignees
@leon-inf
Labels
kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Milestone
Release 0.9.0

Comments

@JashBook
Copy link
Collaborator

Describe the bug

kbcli version
Kubernetes: v1.27.9
KubeBlocks: 0.9.0-beta.23
kbcli: 0.9.0-beta.9

To Reproduce
Steps to reproduce the behavior:

  1. create cluster
kbcli cluster create  tdengine-cluster --termination-policy=WipeOut --cluster-definition=tdengine --cluster-version=tdengine-3.0.5.0 --set cpu=500m,memory=1Gi,replicas=5,storage=5Gi
  1. hscale offline instance
kubectl create -f -<<EOF
apiVersion: apps.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  generateName: tdengine-cluster-hscaleoffinstance-
spec:
  clusterRef: tdengine-cluster
  horizontalScaling:
  - componentName: tdengine
    replicas: 4
    offlineInstances: ["tdengine-cluster-tdengine-0"]
  ttlSecondsAfterSucceed: 0
  type: HorizontalScaling
EOF
  1. stop -- start
kbcli cluster stop tdengine-cluster --auto-approve 
kbcli cluster start tdengine-cluster
  1. See error
kubectl get pod -l app.kubernetes.io/instance=tdengine-cluster
NAME                          READY   STATUS             RESTARTS      AGE
tdengine-cluster-tdengine-1   1/2     CrashLoopBackOff   4 (84s ago)   4m14s
tdengine-cluster-tdengine-2   1/2     CrashLoopBackOff   4 (62s ago)   3m54s
tdengine-cluster-tdengine-3   1/2     CrashLoopBackOff   4 (30s ago)   3m24s
tdengine-cluster-tdengine-4   0/2     CrashLoopBackOff   4 (19s ago)   2m54s

describe pod

➜  ~ kubectl describe pod tdengine-cluster-tdengine-1
Name:         tdengine-cluster-tdengine-1
Namespace:    default
Priority:     0
Node:         aks-largepool-26178435-vmss000000/10.224.0.5
Start Time:   Mon, 13 May 2024 17:12:26 +0800
Labels:       app.kubernetes.io/component=tdengine
              app.kubernetes.io/instance=tdengine-cluster
              app.kubernetes.io/managed-by=kubeblocks
              app.kubernetes.io/name=tdengine
              app.kubernetes.io/version=
              apps.kubeblocks.io/cluster-uid=41546a1a-ea02-4c87-a06e-f4ee1cf56903
              apps.kubeblocks.io/component-name=tdengine
              apps.kubeblocks.io/pod-name=tdengine-cluster-tdengine-1
              clusterdefinition.kubeblocks.io/name=tdengine
              clusterversion.kubeblocks.io/name=tdengine-3.0.5.0
              controller-revision-hash=76b6dc7575
              workloads.kubeblocks.io/instance=tdengine-cluster-tdengine
              workloads.kubeblocks.io/managed-by=InstanceSet
Annotations:  apps.kubeblocks.io/component-replicas: 4
Status:       Running
IP:           10.244.1.23
IPs:
  IP:           10.244.1.23
Controlled By:  InstanceSet/tdengine-cluster-tdengine
Containers:
  tdengine:
    Container ID:   containerd://7d70c93fbebd5f584991b4b39ae8ae9e744eb91f8bf5dd2ebdbdecb4b657194e
    Image:          docker.io/tdengine/tdengine:3.0.5.0
    Image ID:       docker.io/tdengine/tdengine@sha256:a7b599a377e088232204b9a996beba6fd78c7ee836a1992a61801c3331a42032
    Ports:          6030/TCP, 6041/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Mon, 13 May 2024 17:12:30 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  1Gi
    Requests:
      cpu:      500m
      memory:   1Gi
    Liveness:   tcp-socket :6030 delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:  tcp-socket :6030 delay=5s timeout=60s period=10s #success=1 #failure=3
    Environment Variables from:
      tdengine-cluster-tdengine-env      ConfigMap  Optional: false
      tdengine-cluster-tdengine-its-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:            tdengine-cluster-tdengine-1 (v1:metadata.name)
      KB_POD_UID:              (v1:metadata.uid)
      KB_NAMESPACE:           default (v1:metadata.namespace)
      KB_SA_NAME:              (v1:spec.serviceAccountName)
      KB_NODENAME:             (v1:spec.nodeName)
      KB_HOST_IP:              (v1:status.hostIP)
      KB_POD_IP:               (v1:status.podIP)
      KB_POD_IPS:              (v1:status.podIPs)
      KB_HOSTIP:               (v1:status.hostIP)
      KB_PODIP:                (v1:status.podIP)
      KB_PODIPS:               (v1:status.podIPs)
      KB_POD_FQDN:            $(KB_POD_NAME).tdengine-cluster-tdengine-headless.$(KB_NAMESPACE).svc
      SVC_PORT:               6030
      TAOS_ADAPTER_PORT:      6041
      TZ:                     Asia/Shanghai
      TAOS_FQDN:              $(KB_POD_FQDN).cluster.local
      TAOS_FIRST_EP:          $(KB_0_HOSTNAME).$(KB_NAMESPACE).svc.cluster.local:$(SVC_PORT)
      TAOS_ADAPTER_USERNAME:  <set to the key 'username' in secret 'tdengine-cluster-conn-credential'>  Optional: false
      TAOS_ADAPTER_PASSWORD:  <set to the key 'password' in secret 'tdengine-cluster-conn-credential'>  Optional: false
    Mounts:
      /etc/taos from taos-config (rw)
      /scripts from scripts (rw)
      /var/lib/taos from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jwgbt (ro)
  metrics:
    Container ID:  containerd://d036f61ea36c6488c973e40969af975c58088865f97477c892803d2cf5adbd6a
    Image:         infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/taoskeeper:0.0.5
    Image ID:      infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/taoskeeper@sha256:09e1536c94f208cb55dfba094be76107d750d6d57e79d8d5b421f9191e4fff5c
    Port:          6043/TCP
    Host Port:     0/TCP
    Command:
      taoskeeper
    Args:
      -c
      /etc/taos/taoskeeper.toml
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Mon, 13 May 2024 17:16:42 +0800
      Finished:     Mon, 13 May 2024 17:16:53 +0800
    Ready:          False
    Restart Count:  5
    Limits:
      cpu:     0
      memory:  0
    Requests:
      cpu:     0
      memory:  0
    Environment Variables from:
      tdengine-cluster-tdengine-env      ConfigMap  Optional: false
      tdengine-cluster-tdengine-its-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:                    tdengine-cluster-tdengine-1 (v1:metadata.name)
      KB_POD_UID:                      (v1:metadata.uid)
      KB_NAMESPACE:                   default (v1:metadata.namespace)
      KB_SA_NAME:                      (v1:spec.serviceAccountName)
      KB_NODENAME:                     (v1:spec.nodeName)
      KB_HOST_IP:                      (v1:status.hostIP)
      KB_POD_IP:                       (v1:status.podIP)
      KB_POD_IPS:                      (v1:status.podIPs)
      KB_HOSTIP:                       (v1:status.hostIP)
      KB_PODIP:                        (v1:status.podIP)
      KB_PODIPS:                       (v1:status.podIPs)
      KB_POD_FQDN:                    $(KB_POD_NAME).tdengine-cluster-tdengine-headless.$(KB_NAMESPACE).svc
      TAOS_KEEPER_TDENGINE_USERNAME:  <set to the key 'username' in secret 'tdengine-cluster-conn-credential'>  Optional: false
      TAOS_KEEPER_TDENGINE_PASSWORD:  <set to the key 'password' in secret 'tdengine-cluster-conn-credential'>  Optional: false
    Mounts:
      /etc/taos/ from metrics-configuration (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jwgbt (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  metrics-configuration:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      tdengine-cluster-tdengine-metrics-configuration
    Optional:  false
  scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      tdengine-cluster-tdengine-taos-adapter-scripts
    Optional:  false
  taos-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      tdengine-cluster-tdengine-tdengine-taos-config
    Optional:  false
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-tdengine-cluster-tdengine-1
    ReadOnly:   false
  kube-api-access-jwgbt:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 kb-data=true:NoSchedule
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  4m41s                 default-scheduler  Successfully assigned default/tdengine-cluster-tdengine-1 to aks-largepool-26178435-vmss000000
  Normal   Pulled     4m38s                 kubelet            Container image "docker.io/tdengine/tdengine:3.0.5.0" already present on machine
  Normal   Created    4m38s                 kubelet            Created container tdengine
  Normal   Started    4m38s                 kubelet            Started container tdengine
  Warning  BackOff    2m17s (x7 over 4m)    kubelet            Back-off restarting failed container metrics in pod tdengine-cluster-tdengine-1_default(5e102184-6659-45aa-84cb-378d25a2680f)
  Normal   Pulled     2m3s (x5 over 4m23s)  kubelet            Container image "infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/taoskeeper:0.0.5" already present on machine
  Normal   Created    2m3s (x5 over 4m23s)  kubelet            Created container metrics
  Normal   Started    2m3s (x5 over 4m23s)  kubelet            Started container metrics

logs pod

kubectl logs tdengine-cluster-tdengine-1 tdengine
sysctl: setting key "kernel.core_pattern": Read-only file system
05/13 17:12:30.818634 00000045 taos_ADAPTER info "start server:00000045" model=main
05/13 17:12:30.820682 00000045 taos_ADAPTER info "init plugin collectd/v1" model=plugin
05/13 17:12:30.822992 00000045 taos_ADAPTER info "collectd disabled" model=collectd
05/13 17:12:30.823018 00000045 taos_ADAPTER info "init plugin influxdb/v1" model=plugin
05/13 17:12:30.823395 00000045 taos_ADAPTER info "init plugin node_exporter/v1" model=plugin
05/13 17:12:30.828117 00000045 taos_ADAPTER info "node_exporter disabled" model=NodeExporter
05/13 17:12:30.828148 00000045 taos_ADAPTER info "init plugin opentsdb/v1" model=plugin
05/13 17:12:30.828511 00000045 taos_ADAPTER info "init plugin opentsdb_telnet/v1" model=plugin
05/13 17:12:30.884466 00000045 taos_ADAPTER info "opentsdb_telnet disabled" model=opentsdb_telnet
05/13 17:12:30.884491 00000045 taos_ADAPTER info "init plugin prometheus/v1" model=plugin
05/13 17:12:30.884821 00000045 taos_ADAPTER info "init plugin statsd/v1" model=plugin
05/13 17:12:30.890102 00000045 taos_ADAPTER info "statsd disabled" model=statsd
05/13 17:12:30.890369 00000045 taos_ADAPTER info "all plugin init finish" model=plugin
05/13 17:12:30.890381 00000045 taos_ADAPTER info "all plugin start finish" model=plugin
05/13 17:12:30.890489 00000045 taos_ADAPTER info "Running in terminal." model=main
05/13 17:12:30.891890 00000045 taos_ADAPTER info "server on : 6041" model=main
05/13 17:12:30.999022 00000070 UTL apoll url not load
05/13 17:12:30.999083 00000070 UTL load from cfg file /etc/taos//taos.cfg success
05/13 17:12:30.999098 00000070 UTL env file:.env not load
05/13 17:12:30.999119 00000070 UTL load from env variables cfg success
05/13 17:12:31.001144 00000070 DND startup path:/root args:taosd
05/13 17:12:31.002436 00000070 UTL apoll url not load
05/13 17:12:31.002489 00000070 UTL load from cfg file /etc/taos//taos.cfg success
05/13 17:12:31.002500 00000070 UTL env file:.env not load
05/13 17:12:31.002522 00000070 UTL load from env variables cfg success
05/13 17:12:31.002714 00000070 UTL                      global config
05/13 17:12:31.002735 00000070 UTL =================================================================
05/13 17:12:31.002738 00000070 UTL env_var      firstEp                  $(KB_0_HOSTNAME).default.svc.cluster.local:6030
05/13 17:12:31.002741 00000070 UTL default      secondEp                 tdengine-cluster-tdengine-1.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030
05/13 17:12:31.002744 00000070 UTL env_var      fqdn                     tdengine-cluster-tdengine-1.tdengine-cluster-tdengine-headless.default.svc.cluster.local
05/13 17:12:31.002747 00000070 UTL default      serverPort               6030
05/13 17:12:31.002750 00000070 UTL default      tempDir                  /tmp/
05/13 17:12:31.002752 00000070 UTL default      minimalTmpDirGB          1.00
05/13 17:12:31.002760 00000070 UTL default      shellActivityTimer       3
05/13 17:12:31.002762 00000070 UTL default      compressMsgSize          -1
05/13 17:12:31.002765 00000070 UTL default      compressColData          -1
05/13 17:12:31.002768 00000070 UTL default      queryPolicy              1
05/13 17:12:31.002770 00000070 UTL default      enableQueryHb            0
05/13 17:12:31.002773 00000070 UTL default      enableScience            0
05/13 17:12:31.002777 00000070 UTL default      querySmaOptimize         0
05/13 17:12:31.002780 00000070 UTL default      queryPlannerTrace        0
05/13 17:12:31.002784 00000070 UTL default      queryNodeChunkSize       32768
05/13 17:12:31.002789 00000070 UTL default      queryUseNodeAllocator    1
05/13 17:12:31.002792 00000070 UTL default      keepColumnName           0
05/13 17:12:31.002796 00000070 UTL default      smlChildTableName        
05/13 17:12:31.002800 00000070 UTL default      smlTagName               _tag_null
05/13 17:12:31.002804 00000070 UTL default      maxInsertBatchRows       1000000
05/13 17:12:31.002808 00000070 UTL default      maxRetryWaitTime         10000
05/13 17:12:31.002812 00000070 UTL default      useAdapter               0
05/13 17:12:31.002816 00000070 UTL default      crashReporting           1
05/13 17:12:31.002820 00000070 UTL default      queryMaxConcurrentTable  200
05/13 17:12:31.002824 00000070 UTL default      metaCacheMaxSize         -1
05/13 17:12:31.002829 00000070 UTL default      slowLogThreshold         3
05/13 17:12:31.002833 00000070 UTL default      slowLogScope             
05/13 17:12:31.002840 00000070 UTL default      numOfRpcThreads          4
05/13 17:12:31.002843 00000070 UTL default      numOfRpcSessions         10000
05/13 17:12:31.002848 00000070 UTL default      timeToGetAvailableConn   500000
05/13 17:12:31.002852 00000070 UTL default      numOfTaskQueueThreads    4
05/13 17:12:31.002857 00000070 UTL default      dataDir                  /var/lib/taos/
05/13 17:12:31.002861 00000070 UTL default      minimalDataDirGB         2.00
05/13 17:12:31.002865 00000070 UTL default      supportVnodes            16
05/13 17:12:31.002869 00000070 UTL default      maxShellConns            50000
05/13 17:12:31.002874 00000070 UTL default      statusInterval           1
05/13 17:12:31.002878 00000070 UTL default      minSlidingTime           1
05/13 17:12:31.002888 00000070 UTL default      minIntervalTime          1
05/13 17:12:31.002892 00000070 UTL default      maxNumOfDistinctRes      10000000
05/13 17:12:31.002894 00000070 UTL default      countAlwaysReturnValue   1
05/13 17:12:31.002897 00000070 UTL default      queryBufferSize          -1
05/13 17:12:31.002899 00000070 UTL default      printAuth                0
05/13 17:12:31.002902 00000070 UTL default      queryRspPolicy           0
05/13 17:12:31.002904 00000070 UTL default      numOfRpcThreads          4
05/13 17:12:31.002906 00000070 UTL default      numOfRpcSessions         10000
05/13 17:12:31.002909 00000070 UTL default      timeToGetAvailableConn   10000
05/13 17:12:31.002911 00000070 UTL default      numOfCommitThreads       4
05/13 17:12:31.002914 00000070 UTL default      numOfMnodeReadThreads    1
05/13 17:12:31.002916 00000070 UTL default      numOfVnodeQueryThreads   16
05/13 17:12:31.002919 00000070 UTL default      ratioOfVnodeStreamThrea  2.00
05/13 17:12:31.002921 00000070 UTL default      numOfVnodeFetchThreads   4
05/13 17:12:31.002924 00000070 UTL default      numOfVnodeRsmaThreads    8
05/13 17:12:31.002926 00000070 UTL default      numOfQnodeQueryThreads   16
05/13 17:12:31.002929 00000070 UTL default      numOfSnodeSharedThreads  2
05/13 17:12:31.002933 00000070 UTL default      numOfSnodeUniqueThreads  2
05/13 17:12:31.002937 00000070 UTL default      rpcQueueMemoryAllowed    3365278924
05/13 17:12:31.002941 00000070 UTL default      syncElectInterval        25000
05/13 17:12:31.002945 00000070 UTL default      syncHeartbeatInterval    1000
05/13 17:12:31.002950 00000070 UTL default      syncHeartbeatTimeout     20000
05/13 17:12:31.002957 00000070 UTL default      vndCommitMaxInterval     600000
05/13 17:12:31.002961 00000070 UTL default      mndSdbWriteDelta         200
05/13 17:12:31.002965 00000070 UTL default      mndLogRetention          2000
05/13 17:12:31.002970 00000070 UTL cfg_file     monitor                  1
05/13 17:12:31.002974 00000070 UTL default      monitorInterval          30
05/13 17:12:31.002978 00000070 UTL cfg_file     monitorFqdn              localhost
05/13 17:12:31.002982 00000070 UTL default      monitorPort              6043
05/13 17:12:31.002986 00000070 UTL default      monitorMaxLogs           100
05/13 17:12:31.002990 00000070 UTL default      monitorComp              0
05/13 17:12:31.002994 00000070 UTL default      crashReporting           1
05/13 17:12:31.002998 00000070 UTL default      telemetryReporting       1
05/13 17:12:31.003002 00000070 UTL default      telemetryInterval        43200
05/13 17:12:31.003006 00000070 UTL default      telemetryServer          telemetry.tdengine.com
05/13 17:12:31.003010 00000070 UTL default      telemetryPort            80
05/13 17:12:31.003014 00000070 UTL default      transPullupInterval      2
05/13 17:12:31.003017 00000070 UTL default      mqRebalanceInterval      2
05/13 17:12:31.003021 00000070 UTL default      ttlUnit                  86400
05/13 17:12:31.003025 00000070 UTL default      ttlPushInterval          3600
05/13 17:12:31.003029 00000070 UTL default      uptimeInterval           300
05/13 17:12:31.003032 00000070 UTL default      queryRsmaTolerance       1000
05/13 17:12:31.003035 00000070 UTL default      walFsyncDataSizeLimit    104857600
05/13 17:12:31.003039 00000070 UTL default      udf                      1
05/13 17:12:31.003043 00000070 UTL default      udfdResFuncs             
05/13 17:12:31.003047 00000070 UTL default      udfdLdLibPath            
05/13 17:12:31.003052 00000070 UTL default      disableStream            0
05/13 17:12:31.003056 00000070 UTL default      streamBufferSize         134217728
05/13 17:12:31.003060 00000070 UTL default      checkpointInterval       10800000
05/13 17:12:31.003064 00000070 UTL default      cacheLazyLoadThreshold   500
05/13 17:12:31.003068 00000070 UTL default      filterScalarMode         0
05/13 17:12:31.003072 00000070 UTL default      maxStreamBackendCache    128
05/13 17:12:31.003076 00000070 UTL default      configDir                /etc/taos/
05/13 17:12:31.003080 00000070 UTL default      scriptDir                /etc/taos/
05/13 17:12:31.003084 00000070 UTL default      logDir                   /var/log/taos/
05/13 17:12:31.003088 00000070 UTL default      minimalLogDirGB          1.00
05/13 17:12:31.003092 00000070 UTL default      numOfLogLines            10000000
05/13 17:12:31.003097 00000070 UTL default      asyncLog                 1
05/13 17:12:31.003101 00000070 UTL default      logKeepDays              0
05/13 17:12:31.003105 00000070 UTL default      debugFlag                0
05/13 17:12:31.003109 00000070 UTL default      simDebugFlag             143
05/13 17:12:31.003114 00000070 UTL default      tmrDebugFlag             131
05/13 17:12:31.003118 00000070 UTL default      uDebugFlag               131
05/13 17:12:31.003122 00000070 UTL default      rpcDebugFlag             131
05/13 17:12:31.003126 00000070 UTL default      jniDebugFlag             131
05/13 17:12:31.003131 00000070 UTL default      qDebugFlag               131
05/13 17:12:31.003134 00000070 UTL default      cDebugFlag               131
05/13 17:12:31.003138 00000070 UTL default      dDebugFlag               131
05/13 17:12:31.003142 00000070 UTL default      vDebugFlag               131
05/13 17:12:31.003146 00000070 UTL default      mDebugFlag               131
05/13 17:12:31.003150 00000070 UTL default      wDebugFlag               131
05/13 17:12:31.003154 00000070 UTL default      sDebugFlag               131
05/13 17:12:31.003158 00000070 UTL default      tsdbDebugFlag            131
05/13 17:12:31.003162 00000070 UTL default      tqDebugFlag              131
05/13 17:12:31.003166 00000070 UTL default      fsDebugFlag              131
05/13 17:12:31.003170 00000070 UTL default      udfDebugFlag             131
05/13 17:12:31.003175 00000070 UTL default      smaDebugFlag             131
05/13 17:12:31.003179 00000070 UTL default      idxDebugFlag             131
05/13 17:12:31.003184 00000070 UTL default      tdbDebugFlag             131
05/13 17:12:31.003189 00000070 UTL default      metaDebugFlag            131
05/13 17:12:31.003193 00000070 UTL default      timezone                 Asia/Shanghai (CST, +0800)
05/13 17:12:31.003197 00000070 UTL default      locale                   en_US.UTF-8
05/13 17:12:31.003201 00000070 UTL default      charset                  UTF-8
05/13 17:12:31.003206 00000070 UTL default      assert                   1
05/13 17:12:31.003210 00000070 UTL default      enableCoreFile           1
05/13 17:12:31.003214 00000070 UTL default      numOfCores               8.00
05/13 17:12:31.003220 00000070 UTL default      SSE42                    0
05/13 17:12:31.003224 00000070 UTL default      AVX                      0
05/13 17:12:31.003228 00000070 UTL default      AVX2                     0
05/13 17:12:31.003232 00000070 UTL default      FMA                      0
05/13 17:12:31.003236 00000070 UTL default      SIMD-builtins            0
05/13 17:12:31.003240 00000070 UTL default      tagFilterCache           0
05/13 17:12:31.003244 00000070 UTL default      openMax                  1048576
05/13 17:12:31.003249 00000070 UTL default      streamMax                16
05/13 17:12:31.003253 00000070 UTL default      pageSizeKB               4
05/13 17:12:31.003257 00000070 UTL default      totalMemoryKB            32864052
05/13 17:12:31.003260 00000070 UTL default      os sysname               Linux
05/13 17:12:31.003264 00000070 UTL default      os nodename              tdengine-cluster-tdengine-1
05/13 17:12:31.003268 00000070 UTL default      os release               5.15.0-1061-azure
05/13 17:12:31.003272 00000070 UTL default      os version               #70-Ubuntu SMP Wed Apr 3 02:05:58 UTC 2024
05/13 17:12:31.003277 00000070 UTL default      os machine               x86_64
05/13 17:12:31.003281 00000070 UTL default      version                  3.0.5.0
05/13 17:12:31.003284 00000070 UTL default      compatible_version       3.0.0.0
05/13 17:12:31.003288 00000070 UTL default      gitinfo                  e6ddaa7d4e7d0a7909ad16bf15f237bab25dd860
05/13 17:12:31.003292 00000070 UTL default      buildinfo                Built at 2023-06-07 20:22
05/13 17:12:31.003296 00000070 UTL =================================================================
05/13 17:12:31.009233 00000070 DND start to init dnode env
05/13 17:12:31.009667 00000070 DND succceed to read dnode file /var/lib/taos//dnode/dnode.json
05/13 17:12:31.010644 00000070 DND mnode file:/var/lib/taos//mnode/mnode.json not exist
05/13 17:12:31.010659 00000070 DND file:/var/lib/taos//qnode/qnode.json not exist
05/13 17:12:31.010667 00000070 DND file:/var/lib/taos//snode/snode.json not exist
05/13 17:12:31.012535 00000070 DND dnode env is initialized
05/13 17:12:31.012550 00000070 DND start to init service
05/13 17:12:31.012573 00000070 DND node:dnode, start to open
05/13 17:12:31.012586 00000070 UTL worker:dnode-mgmt is initialized, min:1 max:1
05/13 17:12:31.012663 00000070 UTL worker:dnode-mgmt:0 is launched, total:1
05/13 17:12:31.012671 00000070 UTL worker:dnode-mgmt, queue:0x7f9f462688c0 is allocated, ahandle:0x7f9f4ac65dc0
05/13 17:12:31.012681 00000081 UTL worker:dnode-mgmt:0 is running, thread:00000081
05/13 17:12:31.016538 00000082 UDF start to init udfd
05/13 17:12:31.016624 00000082 UDF udfd LD_LIBRARY_PATH: ::/usr/lib
05/13 17:12:31.041450 00000082 UDF udfd is initialized
05/13 17:12:31.041588 00000070 DND node:dnode, has been opened
05/13 17:12:31.041669 00000070 DND node:vnode, start to open
05/13 17:12:31.043141 00000070 WAL wal module is initialized, rsetId:4
05/13 17:12:31.046096 00000070 UTL worker:vnode-query is initialized, min:16 max:16
05/13 17:12:31.046137 00000070 UTL worker:vnode-stream is initialized as auto
05/13 17:12:31.046148 00000070 UTL worker:vnode-fetch is initialized, max:4
05/13 17:12:31.046158 00000070 UTL worker:vnode-mgmt is initialized, min:1 max:1
05/13 17:12:31.046286 00000070 UTL worker:vnode-mgmt:0 is launched, total:1
05/13 17:12:31.046306 00000070 UTL worker:vnode-mgmt, queue:0x7f9f46268960 is allocated, ahandle:0x7f9f4ac2f180
05/13 17:12:31.046355 00000091 UTL worker:vnode-mgmt:0 is running, thread:00000091
05/13 17:12:31.046629 00000070 DND vnode file:/var/lib/taos//vnode/vnodes.json not exist
05/13 17:12:31.046672 00000070 DND open 0 vnodes with 4 threads
05/13 17:12:31.046679 00000070 DND successfully opened 0 vnodes
05/13 17:12:31.046715 00000070 UDF get dnodeId:2 from env, pipe path:/var/lib/taos//.udfd.sock.2
05/13 17:12:31.048938 00000070 UDF udfc initialized
05/13 17:12:31.048964 00000070 DND node:vnode, has been opened
05/13 17:12:31.050349 00000070 DND restore 0 vnodes with 4 threads
05/13 17:12:31.050420 00000070 DND The daemon initialized successfully
05/13 17:12:42.498951 00000093 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:12:45.757447 00000045 taos_ADAPTER error "connect server error" model=restful req_id=0x760d37139a800001 error=[0x40a] Mnode not found
05/13 17:12:45.757733 00000045 taos_ADAPTER info "| 200 | 10.454790891s |             ::1 | POST | %2Frest%2Fsql " model=web req_id=0x760d37139a800001
05/13 17:12:53.119270 00000093 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:12:56.293695 00000045 taos_ADAPTER error "connect server error" req_id=0x760d37139d200002 error=[0x40a] Mnode not found model=restful
05/13 17:12:56.293854 00000045 taos_ADAPTER info "| 200 | 10.432222366s |       127.0.0.1 | POST | %2Frest%2Fsql " model=web req_id=0x760d37139d200002
05/13 17:13:03.758637 00000093 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:13:07.867641 00000045 taos_ADAPTER error "connect server error" model=restful req_id=0x760d37139ff00003 error=[0x40a] Mnode not found
05/13 17:13:07.867872 00000045 taos_ADAPTER info "| 200 | 10.440242884s |       127.0.0.1 | POST | %2Frest%2Fsql " model=web req_id=0x760d37139ff00003
05/13 17:13:14.396982 00000093 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:13:25.044886 00000093 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:13:33.375374 00000045 taos_ADAPTER error "connect server error" model=restful req_id=0x760d3713a6300004 error=[0x40a] Mnode not found
05/13 17:13:33.375541 00000045 taos_ADAPTER info "| 200 | 10.420128288s |       127.0.0.1 | POST | %2Frest%2Fsql " req_id=0x760d3713a6300004 model=web
05/13 17:13:35.692995 00000093 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:13:46.306228 00000093 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:13:56.922195 00000093 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:14:07.545647 00000093 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:14:14.400260 00000045 taos_ADAPTER error "connect server error" model=restful req_id=0x760d3713b0300005 error=[0x40a] Mnode not found
05/13 17:14:14.400427 00000045 taos_ADAPTER info "| 200 | 10.444104613s |       127.0.0.1 | POST | %2Frest%2Fsql " req_id=0x760d3713b0300005 model=web
05/13 17:14:18.158536 00000093 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:14:28.777394 00000093 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:14:39.418703 00000093 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:14:50.027879 00000093 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:15:00.649719 00000093 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:15:11.265092 00000093 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:15:16.387198 00000045 taos_ADAPTER error "connect server error" model=restful req_id=0x760d3713bf500006 error=[0x40a] Mnode not found
05/13 17:15:16.387450 00000045 taos_ADAPTER info "| 200 | 10.425020043s |       127.0.0.1 | POST | %2Frest%2Fsql " model=web req_id=0x760d3713bf500006
05/13 17:15:21.898411 00000093 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:15:32.518807 00000093 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:15:43.131721 00000093 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:15:53.760128 00000093 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:16:04.391550 00000093 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:16:15.021811 00000093 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:16:25.641178 00000093 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:16:36.271103 00000093 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:16:46.891663 00000093 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:16:53.370317 00000045 taos_ADAPTER error "connect server error" req_id=0x760d3713d7000007 error=[0x40a] Mnode not found model=restful
05/13 17:16:53.370476 00000045 taos_ADAPTER info "| 200 | 10.410127408s |       127.0.0.1 | POST | %2Frest%2Fsql " model=web req_id=0x760d3713d7000007
05/13 17:16:57.510662 00000093 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:17:08.135047 00000093 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:17:18.742383 00000093 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0

➜  ~ kubectl logs tdengine-cluster-tdengine-2 tdengine
sysctl: setting key "kernel.core_pattern": Read-only file system
05/13 17:13:00.426491 00000045 taos_ADAPTER info "start server:00000045" model=main
05/13 17:13:00.426692 00000045 taos_ADAPTER info "init plugin node_exporter/v1" model=plugin
05/13 17:13:00.428717 00000045 taos_ADAPTER info "node_exporter disabled" model=NodeExporter
05/13 17:13:00.428741 00000045 taos_ADAPTER info "init plugin opentsdb/v1" model=plugin
05/13 17:13:00.428912 00000045 taos_ADAPTER info "init plugin opentsdb_telnet/v1" model=plugin
05/13 17:13:00.430150 00000045 taos_ADAPTER info "opentsdb_telnet disabled" model=opentsdb_telnet
05/13 17:13:00.430163 00000045 taos_ADAPTER info "init plugin prometheus/v1" model=plugin
05/13 17:13:00.430297 00000045 taos_ADAPTER info "init plugin statsd/v1" model=plugin
05/13 17:13:00.432272 00000045 taos_ADAPTER info "statsd disabled" model=statsd
05/13 17:13:00.432287 00000045 taos_ADAPTER info "init plugin collectd/v1" model=plugin
05/13 17:13:00.433190 00000045 taos_ADAPTER info "collectd disabled" model=collectd
05/13 17:13:00.433202 00000045 taos_ADAPTER info "init plugin influxdb/v1" model=plugin
05/13 17:13:00.433616 00000045 taos_ADAPTER info "all plugin init finish" model=plugin
05/13 17:13:00.433629 00000045 taos_ADAPTER info "all plugin start finish" model=plugin
05/13 17:13:00.433736 00000045 taos_ADAPTER info "Running in terminal." model=main
05/13 17:13:00.485046 00000045 taos_ADAPTER info "server on : 6041" model=main
05/13 17:13:00.632190 00000069 UTL apoll url not load
05/13 17:13:00.632283 00000069 UTL load from cfg file /etc/taos//taos.cfg success
05/13 17:13:00.632300 00000069 UTL env file:.env not load
05/13 17:13:00.632352 00000069 UTL load from env variables cfg success
05/13 17:13:00.634366 00000069 DND startup path:/root args:taosd
05/13 17:13:00.635715 00000069 UTL apoll url not load
05/13 17:13:00.635770 00000069 UTL load from cfg file /etc/taos//taos.cfg success
05/13 17:13:00.635780 00000069 UTL env file:.env not load
05/13 17:13:00.635801 00000069 UTL load from env variables cfg success
05/13 17:13:00.635957 00000069 UTL                      global config
05/13 17:13:00.635965 00000069 UTL =================================================================
05/13 17:13:00.635968 00000069 UTL env_var      firstEp                  $(KB_0_HOSTNAME).default.svc.cluster.local:6030
05/13 17:13:00.635972 00000069 UTL default      secondEp                 tdengine-cluster-tdengine-2.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030
05/13 17:13:00.635975 00000069 UTL env_var      fqdn                     tdengine-cluster-tdengine-2.tdengine-cluster-tdengine-headless.default.svc.cluster.local
05/13 17:13:00.635977 00000069 UTL default      serverPort               6030
05/13 17:13:00.635980 00000069 UTL default      tempDir                  /tmp/
05/13 17:13:00.635983 00000069 UTL default      minimalTmpDirGB          1.00
05/13 17:13:00.635990 00000069 UTL default      shellActivityTimer       3
05/13 17:13:00.635993 00000069 UTL default      compressMsgSize          -1
05/13 17:13:00.635995 00000069 UTL default      compressColData          -1
05/13 17:13:00.635998 00000069 UTL default      queryPolicy              1
05/13 17:13:00.636001 00000069 UTL default      enableQueryHb            0
05/13 17:13:00.636004 00000069 UTL default      enableScience            0
05/13 17:13:00.636007 00000069 UTL default      querySmaOptimize         0
05/13 17:13:00.636010 00000069 UTL default      queryPlannerTrace        0
05/13 17:13:00.636012 00000069 UTL default      queryNodeChunkSize       32768
05/13 17:13:00.636015 00000069 UTL default      queryUseNodeAllocator    1
05/13 17:13:00.636017 00000069 UTL default      keepColumnName           0
05/13 17:13:00.636020 00000069 UTL default      smlChildTableName        
05/13 17:13:00.636023 00000069 UTL default      smlTagName               _tag_null
05/13 17:13:00.636025 00000069 UTL default      maxInsertBatchRows       1000000
05/13 17:13:00.636028 00000069 UTL default      maxRetryWaitTime         10000
05/13 17:13:00.636030 00000069 UTL default      useAdapter               0
05/13 17:13:00.636033 00000069 UTL default      crashReporting           1
05/13 17:13:00.636035 00000069 UTL default      queryMaxConcurrentTable  200
05/13 17:13:00.636038 00000069 UTL default      metaCacheMaxSize         -1
05/13 17:13:00.636040 00000069 UTL default      slowLogThreshold         3
05/13 17:13:00.636042 00000069 UTL default      slowLogScope             
05/13 17:13:00.636045 00000069 UTL default      numOfRpcThreads          4
05/13 17:13:00.636047 00000069 UTL default      numOfRpcSessions         10000
05/13 17:13:00.636050 00000069 UTL default      timeToGetAvailableConn   500000
05/13 17:13:00.636052 00000069 UTL default      numOfTaskQueueThreads    4
05/13 17:13:00.636055 00000069 UTL default      dataDir                  /var/lib/taos/
05/13 17:13:00.636057 00000069 UTL default      minimalDataDirGB         2.00
05/13 17:13:00.636060 00000069 UTL default      supportVnodes            16
05/13 17:13:00.636062 00000069 UTL default      maxShellConns            50000
05/13 17:13:00.636065 00000069 UTL default      statusInterval           1
05/13 17:13:00.636067 00000069 UTL default      minSlidingTime           1
05/13 17:13:00.636070 00000069 UTL default      minIntervalTime          1
05/13 17:13:00.636072 00000069 UTL default      maxNumOfDistinctRes      10000000
05/13 17:13:00.636074 00000069 UTL default      countAlwaysReturnValue   1
05/13 17:13:00.636077 00000069 UTL default      queryBufferSize          -1
05/13 17:13:00.636080 00000069 UTL default      printAuth                0
05/13 17:13:00.636082 00000069 UTL default      queryRspPolicy           0
05/13 17:13:00.636084 00000069 UTL default      numOfRpcThreads          4
05/13 17:13:00.636087 00000069 UTL default      numOfRpcSessions         10000
05/13 17:13:00.636089 00000069 UTL default      timeToGetAvailableConn   10000
05/13 17:13:00.636092 00000069 UTL default      numOfCommitThreads       4
05/13 17:13:00.636094 00000069 UTL default      numOfMnodeReadThreads    1
05/13 17:13:00.636097 00000069 UTL default      numOfVnodeQueryThreads   16
05/13 17:13:00.636100 00000069 UTL default      ratioOfVnodeStreamThrea  2.00
05/13 17:13:00.636102 00000069 UTL default      numOfVnodeFetchThreads   4
05/13 17:13:00.636105 00000069 UTL default      numOfVnodeRsmaThreads    8
05/13 17:13:00.636107 00000069 UTL default      numOfQnodeQueryThreads   16
05/13 17:13:00.636110 00000069 UTL default      numOfSnodeSharedThreads  2
05/13 17:13:00.636112 00000069 UTL default      numOfSnodeUniqueThreads  2
05/13 17:13:00.636114 00000069 UTL default      rpcQueueMemoryAllowed    3365278924
05/13 17:13:00.636117 00000069 UTL default      syncElectInterval        25000
05/13 17:13:00.636120 00000069 UTL default      syncHeartbeatInterval    1000
05/13 17:13:00.636122 00000069 UTL default      syncHeartbeatTimeout     20000
05/13 17:13:00.636125 00000069 UTL default      vndCommitMaxInterval     600000
05/13 17:13:00.636127 00000069 UTL default      mndSdbWriteDelta         200
05/13 17:13:00.636130 00000069 UTL default      mndLogRetention          2000
05/13 17:13:00.636132 00000069 UTL cfg_file     monitor                  1
05/13 17:13:00.636135 00000069 UTL default      monitorInterval          30
05/13 17:13:00.636138 00000069 UTL cfg_file     monitorFqdn              localhost
05/13 17:13:00.636140 00000069 UTL default      monitorPort              6043
05/13 17:13:00.636143 00000069 UTL default      monitorMaxLogs           100
05/13 17:13:00.636145 00000069 UTL default      monitorComp              0
05/13 17:13:00.636148 00000069 UTL default      crashReporting           1
05/13 17:13:00.636150 00000069 UTL default      telemetryReporting       1
05/13 17:13:00.636153 00000069 UTL default      telemetryInterval        43200
05/13 17:13:00.636156 00000069 UTL default      telemetryServer          telemetry.tdengine.com
05/13 17:13:00.636158 00000069 UTL default      telemetryPort            80
05/13 17:13:00.636161 00000069 UTL default      transPullupInterval      2
05/13 17:13:00.636163 00000069 UTL default      mqRebalanceInterval      2
05/13 17:13:00.636166 00000069 UTL default      ttlUnit                  86400
05/13 17:13:00.636169 00000069 UTL default      ttlPushInterval          3600
05/13 17:13:00.636172 00000069 UTL default      uptimeInterval           300
05/13 17:13:00.636174 00000069 UTL default      queryRsmaTolerance       1000
05/13 17:13:00.636177 00000069 UTL default      walFsyncDataSizeLimit    104857600
05/13 17:13:00.636180 00000069 UTL default      udf                      1
05/13 17:13:00.636182 00000069 UTL default      udfdResFuncs             
05/13 17:13:00.636185 00000069 UTL default      udfdLdLibPath            
05/13 17:13:00.636187 00000069 UTL default      disableStream            0
05/13 17:13:00.636190 00000069 UTL default      streamBufferSize         134217728
05/13 17:13:00.636193 00000069 UTL default      checkpointInterval       10800000
05/13 17:13:00.636195 00000069 UTL default      cacheLazyLoadThreshold   500
05/13 17:13:00.636198 00000069 UTL default      filterScalarMode         0
05/13 17:13:00.636200 00000069 UTL default      maxStreamBackendCache    128
05/13 17:13:00.636202 00000069 UTL default      configDir                /etc/taos/
05/13 17:13:00.636205 00000069 UTL default      scriptDir                /etc/taos/
05/13 17:13:00.636207 00000069 UTL default      logDir                   /var/log/taos/
05/13 17:13:00.636210 00000069 UTL default      minimalLogDirGB          1.00
05/13 17:13:00.636213 00000069 UTL default      numOfLogLines            10000000
05/13 17:13:00.636215 00000069 UTL default      asyncLog                 1
05/13 17:13:00.636218 00000069 UTL default      logKeepDays              0
05/13 17:13:00.636221 00000069 UTL default      debugFlag                0
05/13 17:13:00.636223 00000069 UTL default      simDebugFlag             143
05/13 17:13:00.636226 00000069 UTL default      tmrDebugFlag             131
05/13 17:13:00.636229 00000069 UTL default      uDebugFlag               131
05/13 17:13:00.636231 00000069 UTL default      rpcDebugFlag             131
05/13 17:13:00.636234 00000069 UTL default      jniDebugFlag             131
05/13 17:13:00.636236 00000069 UTL default      qDebugFlag               131
05/13 17:13:00.636239 00000069 UTL default      cDebugFlag               131
05/13 17:13:00.636241 00000069 UTL default      dDebugFlag               131
05/13 17:13:00.636243 00000069 UTL default      vDebugFlag               131
05/13 17:13:00.636246 00000069 UTL default      mDebugFlag               131
05/13 17:13:00.636249 00000069 UTL default      wDebugFlag               131
05/13 17:13:00.636251 00000069 UTL default      sDebugFlag               131
05/13 17:13:00.636254 00000069 UTL default      tsdbDebugFlag            131
05/13 17:13:00.636256 00000069 UTL default      tqDebugFlag              131
05/13 17:13:00.636259 00000069 UTL default      fsDebugFlag              131
05/13 17:13:00.636261 00000069 UTL default      udfDebugFlag             131
05/13 17:13:00.636263 00000069 UTL default      smaDebugFlag             131
05/13 17:13:00.636266 00000069 UTL default      idxDebugFlag             131
05/13 17:13:00.636268 00000069 UTL default      tdbDebugFlag             131
05/13 17:13:00.636270 00000069 UTL default      metaDebugFlag            131
05/13 17:13:00.636273 00000069 UTL default      timezone                 Asia/Shanghai (CST, +0800)
05/13 17:13:00.636276 00000069 UTL default      locale                   en_US.UTF-8
05/13 17:13:00.636278 00000069 UTL default      charset                  UTF-8
05/13 17:13:00.636280 00000069 UTL default      assert                   1
05/13 17:13:00.636283 00000069 UTL default      enableCoreFile           1
05/13 17:13:00.636285 00000069 UTL default      numOfCores               8.00
05/13 17:13:00.636289 00000069 UTL default      SSE42                    0
05/13 17:13:00.636291 00000069 UTL default      AVX                      0
05/13 17:13:00.636294 00000069 UTL default      AVX2                     0
05/13 17:13:00.636296 00000069 UTL default      FMA                      0
05/13 17:13:00.636299 00000069 UTL default      SIMD-builtins            0
05/13 17:13:00.636301 00000069 UTL default      tagFilterCache           0
05/13 17:13:00.636304 00000069 UTL default      openMax                  1048576
05/13 17:13:00.636306 00000069 UTL default      streamMax                16
05/13 17:13:00.636309 00000069 UTL default      pageSizeKB               4
05/13 17:13:00.636311 00000069 UTL default      totalMemoryKB            32864052
05/13 17:13:00.636314 00000069 UTL default      os sysname               Linux
05/13 17:13:00.636316 00000069 UTL default      os nodename              tdengine-cluster-tdengine-2
05/13 17:13:00.636319 00000069 UTL default      os release               5.15.0-1061-azure
05/13 17:13:00.636321 00000069 UTL default      os version               #70-Ubuntu SMP Wed Apr 3 02:05:58 UTC 2024
05/13 17:13:00.636324 00000069 UTL default      os machine               x86_64
05/13 17:13:00.636326 00000069 UTL default      version                  3.0.5.0
05/13 17:13:00.636329 00000069 UTL default      compatible_version       3.0.0.0
05/13 17:13:00.636331 00000069 UTL default      gitinfo                  e6ddaa7d4e7d0a7909ad16bf15f237bab25dd860
05/13 17:13:00.636334 00000069 UTL default      buildinfo                Built at 2023-06-07 20:22
05/13 17:13:00.636336 00000069 UTL =================================================================
05/13 17:13:00.642710 00000069 DND start to init dnode env
05/13 17:13:00.646140 00000069 DND succceed to read dnode file /var/lib/taos//dnode/dnode.json
05/13 17:13:00.647211 00000069 DND mnode file:/var/lib/taos//mnode/mnode.json not exist
05/13 17:13:00.647225 00000069 DND file:/var/lib/taos//qnode/qnode.json not exist
05/13 17:13:00.647234 00000069 DND file:/var/lib/taos//snode/snode.json not exist
05/13 17:13:00.649154 00000069 DND dnode env is initialized
05/13 17:13:00.649169 00000069 DND start to init service
05/13 17:13:00.649192 00000069 DND node:dnode, start to open
05/13 17:13:00.649203 00000069 UTL worker:dnode-mgmt is initialized, min:1 max:1
05/13 17:13:00.649255 00000069 UTL worker:dnode-mgmt:0 is launched, total:1
05/13 17:13:00.649269 00000069 UTL worker:dnode-mgmt, queue:0x7fb6a2a688c0 is allocated, ahandle:0x7fb6a7465dc0
05/13 17:13:00.649290 00000080 UTL worker:dnode-mgmt:0 is running, thread:00000080
05/13 17:13:00.652276 00000081 UDF start to init udfd
05/13 17:13:00.652326 00000081 UDF udfd LD_LIBRARY_PATH: ::/usr/lib
05/13 17:13:00.669584 00000081 UDF udfd is initialized
05/13 17:13:00.669749 00000069 DND node:dnode, has been opened
05/13 17:13:00.669807 00000069 DND node:vnode, start to open
05/13 17:13:00.671002 00000069 WAL wal module is initialized, rsetId:4
05/13 17:13:00.673325 00000069 UTL worker:vnode-query is initialized, min:16 max:16
05/13 17:13:00.673341 00000069 UTL worker:vnode-stream is initialized as auto
05/13 17:13:00.673346 00000069 UTL worker:vnode-fetch is initialized, max:4
05/13 17:13:00.673349 00000069 UTL worker:vnode-mgmt is initialized, min:1 max:1
05/13 17:13:00.673410 00000069 UTL worker:vnode-mgmt:0 is launched, total:1
05/13 17:13:00.673418 00000069 UTL worker:vnode-mgmt, queue:0x7fb6a2a68960 is allocated, ahandle:0x7fb6a742f180
05/13 17:13:00.673454 00000090 UTL worker:vnode-mgmt:0 is running, thread:00000090
05/13 17:13:00.677177 00000069 DND vnode file:/var/lib/taos//vnode/vnodes.json not exist
05/13 17:13:00.677216 00000069 DND open 0 vnodes with 4 threads
05/13 17:13:00.677222 00000069 DND successfully opened 0 vnodes
05/13 17:13:00.677248 00000069 UDF get dnodeId:3 from env, pipe path:/var/lib/taos//.udfd.sock.3
05/13 17:13:00.683510 00000069 UDF udfc initialized
05/13 17:13:00.683530 00000069 DND node:vnode, has been opened
05/13 17:13:00.684601 00000069 DND restore 0 vnodes with 4 threads
05/13 17:13:00.684652 00000069 DND The daemon initialized successfully
05/13 17:13:12.123355 00000095 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:13:15.349503 00000045 taos_ADAPTER error "connect server error" model=restful req_id=0x40fd3713a1c00001 error=[0x40a] Mnode not found
05/13 17:13:15.349757 00000045 taos_ADAPTER info "| 200 | 10.437754226s |             ::1 | POST | %2Frest%2Fsql " model=web req_id=0x40fd3713a1c00001
05/13 17:13:22.751219 00000095 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:13:25.902368 00000045 taos_ADAPTER error "connect server error" model=restful req_id=0x40fd3713a4500002 error=[0x40a] Mnode not found
05/13 17:13:25.902566 00000045 taos_ADAPTER info "| 200 | 10.438269204s |       127.0.0.1 | POST | %2Frest%2Fsql " model=web req_id=0x40fd3713a4500002
05/13 17:13:33.390732 00000095 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:13:36.962789 00000045 taos_ADAPTER error "connect server error" model=restful req_id=0x40fd3713a7100003 error=[0x40a] Mnode not found
05/13 17:13:36.962981 00000045 taos_ADAPTER info "| 200 | 10.418838262s |       127.0.0.1 | POST | %2Frest%2Fsql " model=web req_id=0x40fd3713a7100003
05/13 17:13:44.019332 00000095 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:13:54.648437 00000095 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:14:00.390004 00000045 taos_ADAPTER error "connect server error" error=[0x40a] Mnode not found model=restful req_id=0x40fd3713acc00004
05/13 17:14:00.390195 00000045 taos_ADAPTER info "| 200 | 10.429322846s |       127.0.0.1 | POST | %2Frest%2Fsql " model=web req_id=0x40fd3713acc00004
05/13 17:14:05.266989 00000095 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:14:15.902916 00000095 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:14:26.536139 00000095 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:14:37.151344 00000095 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:14:38.382219 00000045 taos_ADAPTER error "connect server error" model=restful error=[0x40a] Mnode not found req_id=0x40fd3713b6000005
05/13 17:14:38.382422 00000045 taos_ADAPTER info "| 200 | 10.420856448s |       127.0.0.1 | POST | %2Frest%2Fsql " model=web req_id=0x40fd3713b6000005
05/13 17:14:47.781768 00000095 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:14:58.423718 00000095 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:15:09.051827 00000095 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:15:19.682525 00000095 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:15:30.287185 00000095 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:15:38.385349 00000045 taos_ADAPTER error "connect server error" error=[0x40a] Mnode not found model=restful req_id=0x40fd3713c4b00006
05/13 17:15:38.385655 00000045 taos_ADAPTER info "| 200 | 10.427283768s |       127.0.0.1 | POST | %2Frest%2Fsql " model=web req_id=0x40fd3713c4b00006
05/13 17:15:40.922022 00000095 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:15:51.548337 00000095 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:16:02.178867 00000095 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:16:12.805925 00000095 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:16:23.414134 00000095 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:16:34.043978 00000095 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:16:44.691579 00000095 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:16:55.324463 00000095 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:17:05.948010 00000095 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:17:16.572298 00000095 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0
05/13 17:17:21.372073 00000045 taos_ADAPTER error "connect server error" model=restful req_id=0x40fd3713ddd00007 error=[0x40a] Mnode not found
05/13 17:17:21.372246 00000045 taos_ADAPTER info "| 200 | 10.405683564s |       127.0.0.1 | POST | %2Frest%2Fsql " model=web req_id=0x40fd3713ddd00007
05/13 17:17:27.196853 00000095 DND ERROR failed to send status req since some vnode/qnode/mnode(s) out of service, epSet:{tdengine-cluster-tdengine-0.tdengine-cluster-tdengine-headless.default.svc.cluster.local:6030}, inUse:0

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.
@JashBook JashBook added kind/bug Something isn't working severity/major Great chance user will encounter the same problem labels May 13, 2024
@JashBook JashBook added this to the Release 0.9.0 milestone May 13, 2024
@JashBook JashBook changed the title [BUG] [BUG] tdengine cluster hscale offline instance then stop and start, pod CrashLoopBackOff May 13, 2024
@leon-inf leon-inf mentioned this issue May 14, 2024
Merged
@leon-inf
Copy link
Contributor

tdengine doesn't support to offline the first instance currently.

@leon-inf leon-inf closed this as not planned Won't fix, can't repro, duplicate, stale May 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@leon-inf leon-inf

Labels
kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Projects
None yet
Milestone
Release 0.9.0
Development

No branches or pull requests
2 participants
@JashBook @leon-inf