Running components on different hosts

[ve-i-uj]

Hello.

I've tried deploying KBEngine components in Docker so that each component runs in a separate container. But I came across the fact that the Machine component registers other components only if they are running on the same host as the Machine. I run components on physical computers instead of Docker containers and the result was the same.

I did an experiment. I ran two computers on the same LAN. On the first computer I launched Machine, on the other computer I launched Logger. Under the debugger, I found that the Machine received the Machine::onBroadcastInterface message on the broadcast from the Logger, because the components are on the same local network. But Machine didn't register Logger, because Logger sent a message from diffrent host.

Here in the code there is a condition due to which the Logger is not registered
https://github.com/kbengine/kbengine/blob/master/kbe/src/server/machine/machine.cpp#L110

...

    void Machine::onBroadcastInterface(Network::Channel* pChannel, int32 uid, std::string& username,

        ...

        // 只记录本机启动的进程
        if(this->networkInterface().intTcpAddr().ip == intaddr ||
                    this->networkInterface().extTcpAddr().ip == intaddr)
        {
            pinfos = Components::getSingleton().findComponent((COMPONENT_TYPE)componentType, uid, componentID);

        ...

This condition discards all components that do not have the same address as the Machine, i.e. all components running on other hosts.

Since the components cannot register with the Machine, they cannot find each other and the cluster does not start. The first errors in the log entries appear when DBMgr cannot find the Logger. Other components also can not find each other.

An example error from DBMgr.

  INFO root [2023-08-30 10:45:47,821] - Components::findComponents: find logger(1)...
 ERROR root [2023-08-30 10:45:49,323] - BundleBroadcast::receive: failed! It can be caused by the firewall, the broadcastaddr, etc.Maybe broadcastaddr is not a LAN ADDR, or the Machine process is not running.
 ERROR root [2023-08-30 10:45:49,323] - Components::findComponents: receive error!
  WARN root [2023-08-30 10:45:49,323] - Components::findComponents: not found logger!

But the Machine is running and under the debugger you can see that the Machine is receiving messages from the broadcast. But due to checking than the host is the same like Machine host, no registration occurs.

Do I understand correctly that a cluster can only work if all components are running on the same host / computer?

[Wandar]

可能是因为网卡没开启广播,用/sbin/ip route add broadcast 255.255.255.255 dev eth1试试

[ve-i-uj]

Thanks for your reply.

Broadcasting is working fine. The Machine and Logger components were on different PCs, and the Machine received the "Machine::onBroadcastInterface" message sent by the Logger over the broadcast channel. The problem in the "if" statement: all "Machine::onBroadcastInterface" messages from not Machine host are rejected.

...

    void Machine::onBroadcastInterface(Network::Channel* pChannel, int32 uid, std::string& username,

        ...

        // 只记录本机启动的进程
        if(this->networkInterface().intTcpAddr().ip == intaddr ||
                    this->networkInterface().extTcpAddr().ip == intaddr)
        {
            pinfos = Components::getSingleton().findComponent((COMPONENT_TYPE)componentType, uid, componentID);

        ...

Could you please explain to me how to run components on different hosts?

[Wandar]

the "internalInterface" label of each component must be filled with internal IP on each machine's xml file,the "addresses" under "machine" must also be filled with all the internal IPs.

[ve-i-uj]

Thanks, it works.

The Machine component must be running on each node of the KBEngne cluster. And the addresses of all nodes must be specified in the root/machine/addresses/item elements in the kbengine.xml file on each node. The internalInterface elements were empty.