POC of queue to store events from MSTeams

[jespino]

This is a POC in draft of storing things in a queue inside mattermost.

The idea is, each event get stored in the databases, if the even gets processed right away, we delete it from the queue.

In the future I will add a pooling mechanism that is going to try again up to 5 times each element in the queue that has more than 0 tries, and we start trying to deliver everything that is in the queue.

This would allow us to provide a more resilient event handling. My main concern here is the performance impact, I think we have space to reduce the database load, but I'm not sure how much.

Probably would be cool to have a load test with this code to figure out the degree of impact on the database.

[enahum]

I have run a load test with this PR.

1. There are no crashes with high load which is good
2. There are no metrics on the three new store functions as there are not present in the timelayer (will try again after generating)
3. a previous run was able to support 13953 users and with this PR it went down to 13776 users
4. Database latency seems to have increased as well (see graphs)

Previous	This PR

I'll run it again after executing make generate and report back

[enahum]

a second run reports a total of 13629 users which is a bit lower than the previous run, so I can say that probably we are looking at supporting 13.5K users. DB Latency seems to reach a max of 500ms.

The way we have constructed the test so far it only includes calls for EnqueueActivity and DequeueActivity.

Here some graphs

Queue	Everything

[lieut-data]

@jespino, propose we pause on these changes while we define next steps for the project.