Service Account failed to be authenticated on the secondly added edge node with requireAuthorization feature gate enabled

[IterableTrucks]

What happened:
Pods on the first added edge node run normally with requireAuthorization feature gate enabled. But after I add the second edge node with the same edgecore configuration, the pod running on the second edge node cannot request k8s api：
authentication.go:73] "Unable to authenticatethe request" err="serviceaccount ns1/sa1 not found". Meanwhile the pod with same manifest runs normally on the first edge node.

What you expected to happen:
Pods can request k8s api on every edge node.

How to reproduce it (as minimally and precisely as possible):

The configuration of edgecore:

apiVersion: edgecore.config.kubeedge.io/v1alpha2
database:
  aliasName: default
  dataSource: /var/lib/kubeedge/edgecore.db
  driverName: sqlite3
kind: EdgeCore
featureGates:
  requireAuthorization: true
modules:
  dbTest:
    enable: false
  deviceTwin:
    dmiSockPath: /etc/kubeedge/dmi.sock
    enable: true
  edgeHub:
    enable: true
    heartbeat: 15
    httpServer: https://192.168.3.45:10002
    messageBurst: 60
    messageQPS: 30
    projectID: e632aba927ea4ac2b575ec1603d56f10
    quic:
      enable: false
      handshakeTimeout: 30
      readDeadline: 15
      server: 192.168.3.45:10001
      writeDeadline: 15
    rotateCertificates: true
    tlsCaFile: /etc/kubeedge/ca/rootCA.crt
    tlsCertFile: /etc/kubeedge/certs/server.crt
    tlsPrivateKeyFile: /etc/kubeedge/certs/server.key
    token: ""
    websocket:
      enable: true
      handshakeTimeout: 30
      readDeadline: 15
      server: 192.168.3.45:10000
      writeDeadline: 15
  edgeStream:
    enable: true
    handshakeTimeout: 30
    readDeadline: 15
    server: 192.168.3.45:10004
    tlsTunnelCAFile: /etc/kubeedge/ca/rootCA.crt
    tlsTunnelCertFile: /etc/kubeedge/certs/server.crt
    tlsTunnelPrivateKeyFile: /etc/kubeedge/certs/server.key
    writeDeadline: 15
  edged:
    containerRuntime: remote
    enable: true
    hostnameOverride: nm178#(nm177  on the first node)
    masterServiceNamespace: default
    maxContainerCount: -1
    maxPerPodContainerCount: 1
    minimumGCAge: 0s
    podSandboxImage: kubeedge/pause:3.6
    registerNode: true
    registerNodeNamespace: default
    registerSchedulable: true
    remoteImageEndpoint: unix:///run/containerd/containerd.sock
    remoteRuntimeEndpoint: unix:///run/containerd/containerd.sock
    rootDirectory: /var/lib/edged
    tailoredKubeletConfig:
      address: 127.0.0.1
      cgroupDriver: systemd
      cgroupsPerQOS: true
      clusterDNS:
      - 169.254.96.16      
      clusterDomain: cluster.local
      configMapAndSecretChangeDetectionStrategy: Get
      containerLogMaxFiles: 5
      containerLogMaxSize: 10Mi
      containerRuntimeEndpoint: unix:///var/run/crio/crio.sock
      contentType: application/json
      cpuCFSQuota: true
      cpuCFSQuotaPeriod: 100ms
      cpuManagerPolicy: none
      cpuManagerReconcilePeriod: 10s
      enableControllerAttachDetach: true
      enableDebugFlagsHandler: true
      enableDebuggingHandlers: true
      enableProfilingHandler: true
      enableSystemLogHandler: true
      enforceNodeAllocatable:
      - pods
      eventBurst: 100
      eventRecordQPS: 50
      evictionHard:
        imagefs.available: 5%
        memory.available: 100Mi
        nodefs.available: 3%
        nodefs.inodesFree: 5%
      evictionPressureTransitionPeriod: 5m0s
      failSwapOn: false
      fileCheckFrequency: 20s
      hairpinMode: promiscuous-bridge
      imageGCHighThresholdPercent: 85
      imageGCLowThresholdPercent: 80
      imageMinimumGCAge: 2m0s
      imageServiceEndpoint: unix:///var/run/crio/crio.sock
      iptablesDropBit: 15
      iptablesMasqueradeBit: 14
      localStorageCapacityIsolation: true
      logging:
        flushFrequency: 5s
        format: text
        options:
          json:
            infoBufferSize: "0"
        verbosity: 0
      makeIPTablesUtilChains: true
      maxOpenFiles: 1000000
      maxPods: 110
      memoryManagerPolicy: None
      memorySwap: {}
      memoryThrottlingFactor: 0.9
      nodeLeaseDurationSeconds: 40
      nodeStatusMaxImages: 0
      nodeStatusReportFrequency: 5m0s
      nodeStatusUpdateFrequency: 10s
      oomScoreAdj: -999
      podPidsLimit: -1
      readOnlyPort: 10350
      registerNode: true
      registryBurst: 10
      registryPullQPS: 5
      resolvConf: /etc/resolv.conf
      runtimeRequestTimeout: 2m0s
      seccompDefault: false
      serializeImagePulls: true
      shutdownGracePeriod: 0s
      shutdownGracePeriodCriticalPods: 0s
      staticPodPath: /etc/kubeedge/manifests
      streamingConnectionIdleTimeout: 4h0m0s
      syncFrequency: 1m0s
      topologyManagerPolicy: none
      topologyManagerScope: container
      volumePluginDir: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
      volumeStatsAggPeriod: 1m0s
  eventBus:
    enable: true
    eventBusTLS:
      enable: false
      tlsMqttCAFile: /etc/kubeedge/ca/rootCA.crt
      tlsMqttCertFile: /etc/kubeedge/certs/server.crt
      tlsMqttPrivateKeyFile: /etc/kubeedge/certs/server.key
    mqttMode: 0
    mqttPassword: ""
    mqttPubClientID: ""
    mqttQOS: 0
    mqttRetain: false
    mqttServerExternal: tcp://127.0.0.1:1883
    mqttServerInternal: tcp://127.0.0.1:1884
    mqttSessionQueueSize: 100
    mqttSubClientID: ""
    mqttUsername: ""
  metaManager:
    contextSendGroup: hub
    contextSendModule: websocket
    enable: true
    metaServer:
      apiAudiences: null
      dummyServer: 169.254.30.10:10550
      enable: true
      server: 127.0.0.1:10550
      serviceAccountIssuers:
      - https://kubernetes.default.svc.cluster.local
      serviceAccountKeyFiles: null
      tlsCaFile: /etc/kubeedge/ca/rootCA.crt
      tlsCertFile: /etc/kubeedge/certs/server.crt
      tlsPrivateKeyFile: /etc/kubeedge/certs/server.key
    remoteQueryTimeout: 60
  serviceBus:
    enable: false
    port: 9060
    server: 127.0.0.1
    timeout: 60

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): v1.26.5

• KubeEdge version(e.g. cloudcore --version and edgecore --version): v1.17.0

• Cloud nodes Environment:

  • Hardware configuration (e.g. lscpu): x86_64 20cores 64GB RAM
  • OS (e.g. cat /etc/os-release): Ubuntu 20.04.6
  • Kernel (e.g. uname -a): 5.4.0-176-generic
  • Go version (e.g. go version):
  • Others:
• Edge nodes Environment:

  • edgecore version (e.g. edgecore --version): v1.17.0
  • Hardware configuration (e.g. lscpu): aarch64 8cores 6GB RAM
  • OS (e.g. cat /etc/os-release): Ubuntu 20.04.5
  • Kernel (e.g. uname -a): 4.14.48
  • Go version (e.g. go version):
  • Others:

[Shelley-BaoYue]

I will try to reproduce it in my own environment and it may take a while. If you have any new progress, please feel free to communicate here.

[zhuyaguang]

my edgecore logs also Appear logs

May 21 00:15:56 edge1 edgecore[6586]: E0521 00:15:56.484334 6586 authentication.go:73] "Unable to authenticate the request" err="tokenData not found when authenticating"

[Shelley-BaoYue]

my edgecore logs also Appear logs

May 21 00:15:56 edge1 edgecore[6586]: E0521 00:15:56.484334 6586 authentication.go:73] "Unable to authenticate the request" err="tokenData not found when authenticating"

Does the problem also occur when multiple edge nodes are connected?