How does broadcast-channel work?

[hongbo-miao]

Hi OPAL team, I reading https://docs.opal.ac/overview/architecture#opals-architecture-potentially-uses-two-pubsub-channels-

While the lightweight channel requires no additional infrastructure, and can suffice for the can we are running only a single OPAL-server. If we wish to scale-out OPAL-servers, we achieve this using a backbone Pub/Sub (such as Redis, Kafka, or Postgres Listen/Notify) to sync all the servers (So a client connecting to one server, receive notifications of updates that are triggered by another server) The backbone Pub/Sub is connected to the lightweight Pub/SUb through the Broadcaster module.

I am trying to understand this. I did an experiment by changing opal-server replicas to 3, and update my policy repo, based on the logs, seems all 3 opal-server synced from the repo directly.
The broadcast-channel (PostgreSQL) didn’t do anything special based on the log.
I am wondering does broadcast-channel only start to work when those 3 opal-server are not sync, for example, when one opal-server is failed to connect the policy repo directly?
Thanks!

[asafc]

As @orweis mentioned on slack, the broadcast channel has a different job.

Consider OPAL's network architecture:

1. there can be many OPAL servers (server replicas)
2. there can be many OPAL clients (can be replicas or on different pods/machines)
3. each OPAL client is connected to one OPAL server.
4. one OPAL server can serve many clients.
5. all messages must be addressed with a topic, and must pass through a server. In other words, the only way to publish an update is through the OPAL server API. Even if you published the update with OPAL's cli - behind the scenes the cli uses the OPAL server REST API to publish the update.
6. Each OPAL server is responsible to pass messages to its own clients (i.e: keep the clients' state in-sync), but - each OPAL server has different clients.

The broadcast-channel is used between two different OPAL servers, in other words:

• The only possible communication channel between two OPAL servers is the broadcast channel.
• The only possible communication channel between an OPAL client and its OPAL server is the lightweight pub/sub channel (over websockets).

Let's demonstrate with an example:

Example 1

• Let's say we have two OPAL servers: s1 and s2.
• client c1 is a subscriber of s1 and is subscribed on the topics: topicA and topicB.
• client c2 is a subscriber of s2 and is subscribed on the topics: topicB and topicC.

Let's say that one of your services published an update on topic TopicC via server s1.

• The only client that should get this message is c1.
• s1 publishes the message via its pub/sub channel to c1.
• However, s1 cannot know if other OPAL servers also have clients that are subscribed on TopicC.
• Therefore s1 will also publish the message to the broadcast channel.
• All servers always listen to the broadcast channel (and they are smart enough to ignore messages that they sent themselves).
• Server s2 will receive the message, but since it does not have any clients subscribed on topicC, it ignores the message.

The paths of the message were:

• originator -> s1 -> s1 direct subscribers (clients) on TopicC -> pub/sub channel -> c1
• originator -> s1 -> broadcast channel -> all other servers -> s2 -> no subcribers on topicC -> s2 ignores the message.

Example 2

Same example, but now the update was published on topic TopicB via server s1.
Only difference will be that c2 will also get the message via s2 (who will get the message from s1 via the broadcast channel).

The paths of the message are:

• originator -> s1 -> s1 direct subscribers (clients) on TopicB -> pub/sub channel -> c1
• originator -> s1 -> broadcast channel -> all other servers -> s2 -> s2 subscribers on TopicB -> pub/sub channel -> c2.

The broadcast channel helps us scale the OPAL cluster, by having many servers - each can have its own clients - and still make sure that all the clients will get the messages directed at them, no matter what server that are connected to.

Hope that is clear enough, please let me know if you need more clarification :)

[hongbo-miao]

Wow, this is so detailed and clear. Thank you so much @asafc !

[loan75]

Hello, i have the feeling that example 1 is wrong.
It said: "Server s2 will receive the message, but since it does not have any clients subscribed on topicC, it ignores the message." but in the example we got "client c2 is a subscriber of s2 and is subscribed on the topics: topicB and topicC.".
But server s2 have a client that is subscribed on the topicC, so it should fw the message to him.

The good path then should be:
originator -> s1 -> broadcast channel -> all other servers -> s2 -> s2 subscribers on TopicC -> pub/sub channel -> c2

Or we change the example 1 an say:
Let's say that one of your services published an update on topic TopicE via server s1.

Am i right ?

[orweis]

HI @loan75, yes you are right -seems @asafc had some confusion or typo here.

In the example:

• client c1 is a subscriber of s1 and is subscribed on the topics: topicA and topicB.
• client c2 is a subscriber of s2 and is subscribed on the topics: topicB and topicC.

Assuming a broadcast channel between the servers, If we publish:

• topicC to s1 -> client c2 will get it through the broadcaster. [example 1]
• topicC to s2 -> client c2 will get it directly from s2
• topicB to s2 -> client c2 will get it directly from s2, and client c1 through the broadcaster
• topicB to s1 -> client c1 will get it directly from s1, and client c2 through the broadcaster
• topicA to s1 -> client c1 will get it directly from s1
• topicA to s2 -> client c1 will get it through the broadcaster

Thanks for catching this error :)