Teams Bot Integration Sample

[sandrist]

We've recently released a new sample application which shows how you can integrate \psi with the Teams bot architecture to develop intelligent bots that can participate in live meetings!

This sample was originally announced at the virtual Platform for Situated Intelligence workshop, and the recording of that session can be found here.

Please note that although it is hosted in the Microsoft Graph repository, you should post any issues or questions about the sample in this repo.

[irwinwilliams]

Hey, I'd like to integrate the OpenCV walkthrough with my own Teams Bot experiment. But I'm stuck on how to get the communication going between the teams stream and python. I'm attempting to write the code using zeroMQ, but not seeing the link.

[AshleyF]

Have you taken a look at the Python Interop Walkthrough? Perhaps you could begin with and build on that. Otherwise, could you give more specifics on problems you're hitting?

[irwinwilliams]

Yep, so with that walkthrough, it's using a MediaCapture object to get frames from a webcam.
var webcamFrames = mediaCapture.Out.Resize(640, 480);
which eventually is sampled to get byte arrays to post to NetMQ.
...

// create down-sized, down-sampled, sparse, compressed, stream of frames
var webcamBytes =
    webcamFrames
        .Sample(TimeSpan.FromMilliseconds(200))
        .EncodeJpeg(90, DeliveryPolicy.LatestMessage)
        .Select(jpg => jpg.Resource.GetBuffer());
webcamBytes.Write("WebcamBytes", store);

...

// send frames over ZeroMQ
var frameWriter = new NetMQWriter<byte[]>(
    pipeline,
    "frames",
    "tcp://127.0.0.1:30000",
    MessagePackFormat.Instance);
webcamBytes.PipeTo(frameWriter);

In the teams sample, video frames are aggregated like so:

var video = this.videoInConnector.Aggregate(
                new Dictionary<string, Shared<PsiImage>>(),
                (aggregate, frames) =>
                {
                    // aggregate dictionary of participant ID -> video frame
                    foreach (var frame in frames)
                    {
                        if (aggregate.TryGetValue(frame.Key, out Shared<PsiImage> old))
                        {
                            old.Dispose();
                            aggregate.Remove(frame.Key);
                        }

                        aggregate.Add(frame.Key, frame.Value.Item1.AddRef());
                    }

                    return aggregate;
                });

But I'm at a loss as to how to get this into the write shape to pipe bytes over to ZeroMQ.

[AshleyF]

I see what you're running up against. The video is a stream of Dictionary<string, Shared<Image>>. It turns out that this cannot be serialized by MessagePack. Let's try simplifying and reducing the type, both to save bytes over the wire and to give MessagePack a better shot at serializing. For example:

var encoder = new ImageToJpegStreamEncoder() { QualityLevel = 90 };
var simpleVideo = video.Select(dict =>
{
    return dict.Select(kv =>
    {
        var img = kv.Value.Resource.Encode(encoder).GetBuffer();
        return (kv.Key, img);
    }).ToList();
});

This converts video to a stream of List<(string, byte[])>. Note that the first Select is a \psi operator, while the second inner one is a LINQ operator over the Dictionary. Encoding the image saves space, but also Shared<Image> was the type that MessagePack couldn't handle (because of pointers within). Also notice the .ToList() to convert the lazy IEnumerable, which holds unserializable closures, to a concrete, reified collection.

The simple List of (string, byte[]) tuples should now be no problem.

var videoWriter = new NetMQWriter<List<(string, byte[])>>(
    pipeline,
    "frames",
    "tcp://127.0.0.1:30000",
    MessagePackFormat.Instance); ;
simpleVideo.PipeTo(videoWriter);

It unpacks on the Python side as a list of lists:

input = zmq.Context().socket(zmq.SUB)
input.setsockopt_string(zmq.SUBSCRIBE, u"frames")
input.connect("tcp://127.0.0.1:30000")

[topic, payload] = input.recv_multipart()
message = msgpack.unpackb(payload, raw=True)
for x in message[b"message"]:
    key = x[0]
    frame = x[1]
    # do whatever with them

I hope this helps!

[irwinwilliams]

Thank you, @AshleyF , I couldn't get Python to work on windows - did on bash. Had to install the MediaPack on the win11 OS.
Now, with Python up, I'm getting this exception:

System.InvalidOperationException
Message=Receiver cannot subscribe to an emitter from a different pipeline. Use a Connector if you need to connect emitters and receivers from different pipelines.

(Stacktrace omitted).

[AshleyF]

This error (cross-pipeline subscription) indicates that an emitter in one pipeline is being piped to a receiver in another pipeline. \psi restricts this from happening accidentally but does allow such a structure by way of a Connector between. A common cause is when creating composite components as subpipelines and accidentally exposing inner component emitters/receivers directly on the outer component. If you could share your code or a description of the component structure, I'd be happy to take a look.

Also, here's a bit on the wiki talking about composite components and connectors: https://github.com/microsoft/psi/wiki/Writing-Components#4-composite-components

[richw-kai]

Hi there, I've been playing around with the PSI Bot for Teams and have managed to get a local version up and running and working well. I was wondering if it would be possible for some deployment information to be posted around the sample? I was hoping to get this up and running on an Azure VM so that I could let others access it for proving the concept without relying on my local network connection and ngrok. I couldn't find anything related to this in the docs

I've managed to build a full package and a running package but I'm new to the world of .net and don't really know what the best scenario would be to open ports up that can speak directly to the application.

I'm assuming my generated executable should be managed through IIS or a similar service manager to ensure it stays up but I wasn't sure where to start.

I noticed that Kestrel is a requirement instead of IIS for local development, but looking for deploying via kestrel suggests this is only ever used for .net core applications.

Any help would be much appreciated.

Thanks,
Rich

[sandrist]

Hi Rich, we haven't tried deploying the bot in an Azure VM ourselves, so I don't have too much insight I can provide, unfortunately.

However, I am aware of a project in which someone else used the Teams bot integration sample to build their own psi bot for Teams, and they seem to have figured out how to deploy to the cloud. Maybe this doc will give you some ideas?
https://github.com/AI4Bharat/INCLUDE-MS-Teams-Integration/blob/main/docs/deploy.md

[irwinwilliams]

Hi there,
When I was exploring how to use PSI with this sample, it was all done via a VM in Azure. In my proof of concept using this, I was able to get it to render in a teams meeting - after some trials, here: https://www.youtube.com/watch?v=wjQmDisBg5U&t=9491s
I did use things like ngrok to tunnel into the VM, and it worked fine.
As it turns out, my unfamiliarity with some of the streams in PSI led to some confusion (as you can see in the above call), but that got eventually sorted out.
I don't think you should have too many problems in PoC context. And as long as you get more experience with how everything fits together, you should find alternative approaches to getting things deployed.