Teams Bot App with SSO on K8S Experiences Multiple Responses with HPA Scaling Beyond Two Pods

[wiedu-gcws28]

Hi,

We have developed a bot App by referencing the example of the bot app with SSO Enabled from the teams toolkit sample and deployed it to our on-premise K8S, making it publicly accessible at the service endpoint (https://testdomain.com/service/api/messages). The manifest for this teams bot app has also been uploaded to our organization's apps.

Teams toolkit SSO Bot Sample

At the same time, we have also created an Azure Bot app resource in Azure and set the message endpoint to point to our service endpoint on K8s

Azure Bot app message endpoint

Currently, we have encountered an issue where if the HPA on K8s is set with more than two Pods, there is a situation where multiple responses are returned for a single input

multiple responses are returned for a single input

HPA Settings in deployment yaml file

When the HPA is set with only one Pod, it operates normally .

Is there a best practice for HPA settings(we need more than two pods) when deploying with the bot framework (to my knowledge, the teams toolkit bot is based on the bot framework) to avoid this issue?

[hund030]

Hi @wiedu-gcws28
I didn't reproduce your issue. I tried migrating my bot in k8s to horizontal autoscaling and it works fine even there are multiple pods.
There are two pods but only one pod get the request.


And I exactly receive only one response.

I guess there may be an issue in your deployment yaml file rather than in the bot framework. Could you provide more information? The complete yaml file may help.

Here is my deployment yaml file for your reference:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sso-bot
spec:
  selector:
    matchLabels:
      app: sso-bot
  template:
    metadata:
      labels:
        app: sso-bot
    spec:
      containers:
        - name: sso-bot
          image: <image>
          ports:
            - containerPort: 80
          imagePullPolicy: Always
          envFrom:
            - secretRef:
                name: dev-secrets
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: sso-bot
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: sso-bot
  minReplicas: 2
  maxReplicas: 4
  metrics:
    - type: Resource
      resource:
        name: cpu
        target: 
          type: Utilization
          averageUtilization: 50
---
apiVersion: v1
kind: Service
metadata:
  name: sso-bot
spec:
  type: ClusterIP
  ports:
    - port: 80
  selector:
    app: sso-bot
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt
    solvers:
    - http01:
        ingress:
          class: nginx
          podTemplate:
            spec:
              nodeSelector:
                "kubernetes.io/os": linux
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sso-bot-ingress
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - <host>
    secretName: tls-secret
  rules:
    - host: <host>
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: sso-bot
                port:
                  number: 80

[hund030]

This issue occurs when there are multiple client login and the bot may receive a signin/tokenExchange event from each client. The sample implements a dedup step to deduplicate the requests by storing activity id in memory.
However, when there are multiple pods in k8s, the dedup step may not work as expected because pods do not share memory. You will need a distributed storage. The bot framework supports Azure Blob Storage or Azure Cosmos DB. You can find how to use them in this document: Write directly to storage

I also draft a sample for your reference.
In teamsBot.ts

    // const memoryStorage = new MemoryStorage();
    // Create conversation and user state with in-memory storage provider.
    // this.conversationState = new ConversationState(memoryStorage);
    // this.userState = new UserState(memoryStorage);
    // this.dialog = new SSODialog(new MemoryStorage());

    const blobsStorage = new BlobsStorage(
      config.blobConnectionString,
      config.blobContainerName
    );
    // Create conversation and user state with Azure Blob Storage
    this.conversationState = new ConversationState(blobsStorage);
    this.userState = new UserState(blobsStorage);
    this.dialog = new SSODialog(blobStorage);

In ssoDialog.ts

import { ContainerClient, RestError } from "@azure/storage-blob";

  async onEndDialog(context: TurnContext) {
    const key = this.getStorageKey(context);
    try {
      await this.dedupStorage.delete([key]);
    } catch (err) {
      console.debug(`delete dedup storage error: ${err.toString?.()}}`);
    }
  }

  // If a user is signed into multiple Teams clients, the Bot might receive a "signin/tokenExchange" from each client.
  // Each token exchange request for a specific user login will have an identical activity.value.Id.
  // Only one of these token exchange requests should be processed by the bot.  For a distributed bot in production,
  // this requires a distributed storage to ensure only one token exchange is processed.
  async shouldDedup(context: TurnContext): Promise<boolean> {
    const key = encodeURIComponent(this.getStorageKey(context));
    try {
      const storeItems = JSON.stringify({ [key]: {} });
      // `botbuilder-azure-blobs` does not support `ifNoneMatch` condition, so I use azure blobs sdk directly.
      const containerClient = new ContainerClient(
        config.blobConnectionString,
        config.blobContainerName
      );
      await containerClient
        .getBlockBlobClient(key)
        .upload(storeItems, storeItems.length, {
          conditions: { ifNoneMatch: "*" },
          blobHTTPHeaders: { blobContentType: "application/json" },
        });
    } catch (error) {
      if (error instanceof RestError && error.code === "BlobAlreadyExists") {
        return true;
      }
    }

    return false;
  }