Support creating a Cluster from a DC/OS url (i.e., Cluster.from_url).

[jieyu]

When creating a Cluster for an existing DC/OS cluster, the most convenient way is to create it from a single DC/OS url, instead of asking users to manually specifying master/agents node information which is tedious. Technically, the library can use DC/OS APIs to get all the master/agents, and populate the necessary information in Cluster object.

[adamtheturtle]

Thanks @jieyu !

I imagine that this would look like Cluster.from_url(url=url, ...).
The ... is where it gets interesting!

With Cluster.from_nodes we take Node objects for masters, agents and public_agents.

On an implementation level - Cluster.from_url would likely create Node objects and then return a Cluster.from_nodes from those objects.

Node.__init__ takes the following:

public_ip_address: IPv4Address,
private_ip_address: IPv4Address,
default_user: str,
ssh_key_path: Path,
default_transport: Transport,

We can construct a Node from a URL if we can know that information for each node, either from the user or from the API.
I'm not familiar yet with what the DC/OS API gives us, but I'm hoping that at best it will give us the public IP addresses and the private IP addresses of all nodes, and the node role (master, agent or public agent).

With that, we have to also provide the default user, SSH key path and default transport, and those must be the same for all nodes.
We can provide sensible defaults for (some of) these.

I imagine therefore that the interface may look something like:

Cluster.from_url(url, default_user, ssh_key_path, default_transport)

I do not want to think too deeply about this before I know the specifics of the DC/OS API endpoints you are referring to - please could you like me to those?

Assuming that these API endpoints are not open to the world, we would likely run the curl command on the host which the given URL resolves to.

I guess the method would do something like:

def from_url

    one_master_public_ip = socket.gethostbyname(url)
    one_master_node = Node(
        public_ip_address=one_master_public_ip,
        private_ip_address=one_master_public_ip,
        default_user=default_user,
        ssh_key_path=ssh_key_path,
        default_transport=default_transport,
    )
    
    path = url + '/some/dcos/api/endpoint'
    endpoint_json = one_master_node.run(args=['curl', path]).stdout
    ...
    
    masters = agents = public_agents = {}
    for item in loaded_endpoint_json['masters']:
        node = Node(public_ip=...)
        masters.add(node)

    ...
    
    cluster = Cluster.from_nodes(
        masters=masters,
        agents=agents,
        public_agents=public_agents,
    )
    
    return cluster

[jieyu]

@adamtheturtle the dcos-test-utils package has some helpers to get master/agent ips
https://dcos-test-utils.readthedocs.io/en/latest/dcos_test_utils.html#module-dcos_test_utils.dcos_api

Maybe you can directly use those?

[adamtheturtle]

Looking under the hood at how those are retrieved if they are not set by the user we can see set_node_lists_if_unset.

set_node_lists_if_unset makes HTTP requests to /exhibitor/exhibitor/v1/cluster/list and /mesos/slaves.

To keep in mind when considering this:

• To create a DcosApiSession which can make these requests we need credentials, further expanding the interface for Cluster.from_url.
• Making HTTP requests requires the VPN on macOS, unless we special-case Docker and make sure that localhost is used. So far the Cluster class is backend agnostic, but I'm sure we could work something out.

I think better is to use Node.run and then curl those API endpoints from a master node. This does not need any login credentials and we can use the given transport, such as docker-exec.

[adamtheturtle]

Just a thought to consider later - maybe this is better as Cluster.from_master_node and the user has to construct a Node object.