Replace the polling in the webui with Websocket events

[togir2]

Is your feature request related to a problem?

The webui is using polling to update the data its showing.
E.g. the /Schedule/get-current-playlist/ endpoint is called ~5 sec. to check for changed playout information.

The Library and "Scheduled Shows" are also polling but on a much longer interval, which can lead to inconsistent views when multiple people (or tabs) are using Libretime simultaneously.
E.g. one is adding content to a show via the "Calendar" and the other one (not noticing, in his view the show is still empty) via the "Dashboard". This will result in duplicate content.

Describe the solution you'd like

I would like to add Websoket support to the Django-Api via channels.
I would like to have two sockets one "public" to which external clients can connect to get "now-playing" and "schedule-updates", and one "internal" for the webui to replace the polling.

I do not want to replace the existing REST-Api. I do think the Websockets only make sense for update notifications, not the initial loading/updating data. Also I think the REST-Api is easier to integrate to external tools.

I took a look through the webui and worked out the following events:

• live-info-changed
  Metadata from the playout
• calendar-changed
  When a show in the calendar gets updated/ deleted
• library-changed
  When a Song/ Smartblock / Playlist / Track is added, changed or deleted
• scheduled-content-feed-changed
  When the "Scheduled Shows" tab on the "Dashboard" should update

The events should contain the changed data for the webui to check if it should update.

The clients should be able to choose which events they want to receive.

Describe alternatives you've considered

An alternative to the Django-Channels would be to use RabbitMQ-websockets but then we would need to write cusom code to send the messages. Also I do not feel well exposing the RabbitMQ to the internet.

Also a Graphql endpoint could combine the "REST-Api" with an subscription feature for updates.
But as there is already a REST-Api I don't think we should have an additional api with "similar" features.
Also for the "subscription" the Django-Channels are needed to.

Additional context

The Websockets can only work for the items that are handled in the Django-Api so this should be added incrementally as the api gets migrated.

As a POC I would like to add the Websocket updates for the "live-info-changed" event as I think this is the easiest to implement in Django and webui.

I would appreciate to have some feedback for this idea :)

[jooola]

This idea has been in my head for a long time, great that you want to tackle it.

I would like to add Websoket support to the Django-Api via channels.

I agree, I wonder what backend we want to use though, redis seem to be the obvious answer, but maybe we can leverage the existing services (rabbitmq/postgresl) instead of adding a new one. Or maybe we should simply add redis as it might be useful in the future anyway.

I would like to have two sockets one "public" to which external clients can connect to get "now-playing" and "schedule-updates", and one "internal" for the webui to replace the polling.

Sounds like a good idea.

One problem we need to solve before doing this, is to properly handle authentication and authorization in django.

Right now, we only allow read only access on some resources for the public, read only with the api key for internal communication and the read/write for the admin, but maybe @paddatrapper can correct me on this one, as we might support a bit more.

In addition, we might have to authenticate twice (legacy + api) for any user using the UI, or setup a shared session store and only log in once. I would love to only log using the api and use a custom shared php session store for legacy #1788. Maybe we also use token based auth instead.

[togir2]

I agree, I wonder what backend we want to use though, redis seem to be the obvious answer, but maybe we can leverage the existing services (rabbitmq/postgresl) instead of adding a new one. Or maybe we should simply add redis as it might be useful in the future anyway.

I do not think we need redis, there is a backend for rabbitmq https://github.com/CJWorkbench/channels_rabbitmq

[paddatrapper]

Right now, we only allow read only access on some resources for the public, read only with the api key for internal communication and the read/write for the admin, but maybe @paddatrapper can correct me on this one, as we might support a bit more.

The permission structure is based on the same groups as the UI - Guest, Host, Programme Manager and Admin. Specific permission handling isn't done for Admin, as they designated Django super admin access. The rest are defined in https://github.com/libretime/libretime/blob/main/api/libretime_api/permission_constants.py and https://github.com/libretime/libretime/blob/main/api/libretime_api/permissions.py

[togir2]

I did some thinking and have a proposal on how to intgrate and couple channels into the django-app

The rough idea idea is:

• Every model should be subscriptable via a channel.
• There can be custom "groups" where databatase models do not fit well
• There will be one channel endpoint /ws/events
• The endpoint is read-only

The channel will transport messages that are encoded as json.
All messages have a top level "type" field to distinguish them.

Client side

After connection client can send two events:

{"type":"subscribe", "group":"groupname1" }
{"type":"sunubscribe", "group":"groupname1"}

For troubleshooting the client will receive a response to either of these events:

{"type": "subscription_status": "subscriptions":["groupname1", "groupname2"]}

Server side

Add client to a group

When the server receives a "subscribe" message it will check if the
user (if there is a user) has permission to view the resource.
If the user has the permission the client will be added to the "group" of the
requested model.

Send notifications to a group

We can overwrite the save/delete methods of the Django models.
The methods will then notify the "group" model.__name__ about the changed data .
The message a client will receive could look like this:

// action could be add, update, delete
{"type": "data", "action": "delete",, "group": "group_name" "data": json(model_values)}

We can either do this individually in every model, provide a class decorator or use a wrapper class that overrides the methods.
I prefer the wrapper class.

Custom groups

There could be custom groups that do not map to a django model.
There should be one file which list all the custom groups.
E.g. a "live_info" group that pushes combined data about the current show and track.
They should start with a prefix (that cannot be a class-name in python e.g "event-*") to avoid having a name collision with a model class later. As custom group permissions can also not checked via the Django permission convention it must provide a permission.

Permission checking

The permission for a group is of form "view_modelname" for models or the specified one for the custom groups.
We can check with user.has_permission() if a user can be added to a group.

Documentation and type generation

There should be a page in the docs that list all available groups and the data-format.
The data should be generated from the models classes.
The custom endpoints would need to provide there own definition class.

I do not think they can be integrated into the openApi schema.yml
OAI/OpenAPI-Specification#55 (comment)

There is https://www.asyncapi.com/ which seems it can do a similar thing for web-sockets.
We could then (in future) generating Datetypes for the client to ease on the development there.