Make IPAM topology aware

[jbrendel]

Scenario: Someone wishes to deploy a Romana cluster across multiple L2 domains. For example: Multiple AWS VPCs, or multiple L2 domains in a datacenter (maybe multiple racks with a ToR on top).

• IPAM/topology need to know that there are different L2 domains. As a config item, the number of L2 domains and the max number of hosts in each L2 domain can be specified. For starters, we can assume that the max number of hosts in each L2 domain is the same.
• IPAM needs to calculate which Romana address range is to be used for each L2 domain.
• Via command line and API it should be possible for the operator to see the configured L2 domains and the Romana address ranges, which have been assigned to them. This is useful for the operator who needs to take this information in order to manually configure some routes.
• It should be possible to add L2 domains to IPAM's config ahead of time, even way before any nodes in a given L2 actually want to join the cluster. But at the same time, it should be possible to add a new L2 domain after the fact. This may possibly be solved by saying: With this many hosts per L2 domain, we can have at most X L2 domains. So, we always go ahead and assume that there will be X L2 domains. Need to discuss this a little.
• IPAM needs to learn to keep track of IP addresses on a per L2 domain basis.
• When a new node joins the cluster, there should be some useful information in the join-request, which can tell us what L2 domain that host is in. This could be a number of things and may be different based on the environment. For example, it could be the node IP address, or some tag that is gleaned from an environment variable or config file when the request is made. We need to be a little flexible here. The tag could be retrieved via a startup script and environment variable in AWS. In other environments... not sure yet.
• IPAM needs to be configured with some advanced knowledge to use the correct information contained in the request in order to determine which L2 domain the new node is located in. Maybe we can just say: Either source address or some tag in the join request.
• The agents need to be modified to only add host routes to other hosts in its own L2 domain. Everything else is done via the default route.

[chrismarino]

No mention here of how the external infrastructure gets configured with routes. For AWS it would be manually, but for L3 ToRs Romana needs to announce it's container network.

[jbrendel]

That's right. Configuring external hardware would be a separate task.

[ymmt2005]

I'd like to know whether this issue has already been addressed with the current romana or not.

Maybe we can already auto-select topology with k8s node labels as described here?

Or this requires something more to dynamically assign range of IP addresses in a subnet?

Thanks in advance!

[chrismarino]

Hi @ymmt2005 This issue is quite old. It should have been closed long ago. Topology aware IPAM is an essential aspect of Romana v2.0 which has been available for quite some time. The details on how to create your own custom topology are described here.

You can also use this topology map wizard as well.

[chrismarino]

@ymmt2005 As for your question about dynamically assigning IP addresses in a subnet, there are several aspects to this. To start, you can create a topology that uses only one network range, and then later update the topology to include a new (i.e. second), no-overlapping network. Or you can define both networks up front.

Either way, once they are defined, you can tag your pod spec with the network name and Romana IPAM will pull an IP from the proper network. Send an email to info@romana.io if you want to join our Slack channel to get more help on getting this to work.

[ymmt2005]

@chrismarino Thank you for the clarifications! I got it.