Romana couldn't support configuring network topology at runtime?

[mdianjun]

Once romana daemon is started, /romana/ipam/data is create in etcd。 While, if I want to change network topology（e.g. add hosts, add groups）, but this is not allowed。

[chrismarino]

Hi @mdianjun Updating the network topology on a running system is not well documented right now. Hosts can be added easily, new networks can be added easily, but changing the topology for a network that has already been defined is kind of tricky. Because prefixes are defined for a network before an IP address is allocated, a change to running network requires that the IP addresses and prefixes can be merged into the new topology.

There is an upcoming PR to allow topology merging, but is has not been comited/released.

But adding hosts to existing groups is possible today. And adding more networks (with non-overlapping CIDRs) is also possible.

Additional hosts will be automatically added to 'groups' in the network topology and get an IP with the proper prefix, etc.

As for new networks, see this example topology with 2 networks. http://docs.romana.io/Content/advanced.html#using-multiple-networks

You should be able to add the additional networks onto a running cluster. Although @pritesh and/or @cgilmour will have to provide the details on how it is done.

[chrismarino]

When new nodes are added to a Kubernetes cluster, Romana will add each new node to a 'group[]' in the topology (in a round robin fashion) and pods launched on that node will get addresses with that group prefix. Romana has a 'listener' that watches for changes to the node list. If the topology specifies more details such as 'rack-1', 'rack-2' etc, Romana will look for tags with these lables to assign the new node to that group.

More detail here: http://docs.romana.io/Content/networking.html#topology

[chrismarino]

@mdianjun One more comment about updating network topologies on running clusters. A common change to a network topology would be to add racks. It is also likely that with the new rack, additional addresses would become available on the larger cluster. This kind of change to a network topology would is easily supported with an updated topology file that includes the new rack and network.

So, for example, this multi-rack topology could be updated with rack-5 through rack-8 that used

    "networks": [
        {
            "name"      : "new-racks",
            "cidr"      : "10.222.0.0/16",
            "block_mask": 29
        }

These kind of topology changes are possible today, as long as the network CIDRs do not overlap.

More complicated changes to a network topology will require the upcoming PR to merge the IPs and ranges, etc.

[mdianjun]

How to reload topology configuration?

1. change configmap of romana-daemon's --initial-network, but romana-daemon reports:

...
IPAM exists at /ipam/data: true
Ignoring initial topology file /etc/romana/network/custom-network.json as IPAM already exists
...

2. use romana cli, but the tool doesn't implement the interfaces of "host add, host delete, etc."

[root@A02-R12-I186-226 kube-script]# romana host add

Unimplemented: Add host/s.

[chrismarino]

Hi @mdianjun for item #1, I believe these are just warnings. It goes to my earlier point that when the topology changes, the IP addresses in use on running clusters, and the routes to them, must survive. This means that an updated network topology has to 'merge' with the new one. There is a PR for this, but it is not yet committed.

If the updated topology does not conflict in any way with the old one, everything should be OK. However, this requires some advance planning.

As for item #2, the Romana CLI should not be needed for any operations with Kubernetes. Adding nodes, etc. is done with Kubernetes tools (i.e. kubeadm, Kubespray, kops, Ansible, etc.) and Romana watches the changes and configures the network accordingly. And even if you wanted to use the CLI, has not been updated for v2.0. See this note.