Nutanix Cluster on Equinix Metal

This Terraform module will deploy a proof-of-concept demonstrative Nutanix Cluster in Layer 2 isolation on Equinix Metal. DNS, DHCP, and Cluster internet access is managed by an Ubuntu 22.04 bastion/gateway node.

Acronyms and Terms

• AOS: Acropolis Operating System
• NOS: Nutanix Operating System (Used interchangeably with AOS)
• AHV: AOS Hypervisor
• Phoenix: The AOS/NOS Installer
• CVM: Cluster Virtual Machine
• Prism: AOS Cluster Web UI

Nutanix Installation in a Nutshell

For those who are unfamiliar with Nutanix. Nutanix is HCI (Hyperconverged Infrastructure) software. See https://www.nutanix.com/products/nutanix-cloud-infrastructure for more details from Nutanix.

Nutanix AOS is typically deployed in a private network without public IPs assigned directly to the host. This experience differs from what many cloud users would expect in an OS deployment.

This POC Terraform module is inspired by the Deploying a multi-node Nutanix cluster on Metal guide which goes into detail about how to deploy Nutanix and the required networking configuration on Equinix Metal. Follow that guide for step-by-step instructions that you can customize along the way.

By deploying this POC Terraform module, you will get an automated and opinionated minimal Nutanix Cluster that will help provide a quick introduction to the platform's capabilities.

This project is NOT intended to demonstrate best practices or Day-2 operations, including security, scale, monitoring, and disaster recovery.

To accommodate deployment requirements, this module will create:

• 1x m3.small.x86 node running Ubuntu 22.04 in a hybrid-bonded networking mode

  This "bastion" node will act as a router and jump box. DHCP, DNS, and NAT (internet access) functionality will be provided by dnsmasq.

• 3x m3.large.x86 nodes running Nutanix LTS 6.5 in layer2-bonded networking mode

  Workload Optimized hardware reservations are preferred and required for capacity and to ensure hardware compatibility with Nutanix. On-demand instances will be deployed by default, see "On-Demand Instances" notes below for more details.

• 1x VLAN and Metal Gateway with VRF

  The VRF will route a /22 IP range within the VLAN, providing ample IP space for POC purposes.

  The bastion node will attach to this VLAN, and the Nutanix nodes will passively access this as their Native VLAN with DHCP addresses from the VRF space assigned by the bastion node.

• 1x SSH Key configured to access the bastion node

  Terraform will create an SSH key scoped to this deployment. The key will be stored in the Terraform workspace.

• 1x Metal Project

  Optionally deploy a new project to test the POC in isolation or deploy it within an existing project.

Terraform installation

You'll need Terraform installed and an Equinix Metal account with an API key.

If you have the Metal CLI configured, the following will setup your authentication and project settings in an OSX or Linux shell environment.

eval $(metal env -o terraform --export) #
export TF_VAR_metal_metro=sl # Deploy to Seoul

Otherwise, copy terraform.tfvars.example to terraform.tfvars and edit the input values before continuing.

Run the following from your console terminal:

terraform init
terraform apply

When complete, after roughly 45m, you'll see something like the following:

Outputs:

bastion_public_ip = "ipv4-address"
nutanix_sos_hostname = [
  "uuid1@sos.sl1.platformequinix.com",
  "uuid2@sos.sl1.platformequinix.com",
  "uuid3@sos.sl1.platformequinix.com",
]
ssh_private_key = "/terraform/workspace/ssh-key-abc123"
ssh_forward_command = "ssh -L 9440:1.2.3.4:9440 -i /terraform/workspace/ssh-key-abc123 root@ipv4-address"

See "Known Problems" if your terraform apply does not complete successfully.

Next Steps

You have several ways to access the bastion node, Nutanix nodes, and the cluster.

Login to Prism GUI

• First create an SSH port forward session with the bastion host:

  Mac or Linux

  $(terraform output -raw ssh_forward_command)

  Windows

  invoke-expression $(terraform output -raw ssh_forward_command)

• Then open a browser and navigate to https://localhost:9440 (the certificate will not match the domain)

• See Logging Into Prism Central for more details (including default credentials)

Access the Bastion host over SSH

For access to the bastion node, for troubleshooting the installation or to for network access to the Nutanix nodes, you can SSH into the bastion host using:

ssh -i $(terraform output -raw ssh_private_key) root@$(terraform output -raw bastion_public_ip)

Access the Nutanix nodes over SSH

You can open a direct SSH session to the Nutanix nodes using the bastion host as a jumpbox. Debug details from the Cluster install can be found within /home/nutanix/data/logs.

ssh -i $(terraform output -raw ssh_private_key) -j root@$(terraform output -raw bastion_public_ip) nutanix@$(terraform output -raw cvim_ip_address)

Access the Nutanix nodes out-of-band

You can access use the SOS (Serial-Over-SSH) interface for out-of-bands access using the default credentials for Nutanix nodes.

ssh -i $(terraform output -raw ssh_private_key) $(terraform output -raw nutanix_sos_hostname[0]) # access the first node
ssh -i $(terraform output -raw ssh_private_key) $(terraform output -raw nutanix_sos_hostname[1]) # access the second node
ssh -i $(terraform output -raw ssh_private_key) $(terraform output -raw nutanix_sos_hostname[2]) # access the third node

Known Problems

On-Demand Instances

This POC allocates a m3.small.x86 node for the Bastion host by default, you can change this to another instance type of your choosing by setting the metal_bastion_plan variable.

This POC allocates m3.large.x86 instances for the Nutanix nodes. Not all on-demand m3.large.x86 nodes will work. At the time of writing, we recommend the SL or AM Metros for deployment. If a Nutanix node fails to provision, please try to terraform apply again. A node that fails to provision with the Nutanix AOS will be automatically removed from your project. Terraform will subsequently attempt to replace those servers.

Production deployments should use qualified Workload Optimized instances for Nutanix nodes. Create a hardware reservation or contact Equinix Metal to obtain validated Nutanix compatible servers. You can also convert a successfully deployed on-demand instance to a hardware reservation. Hardware Reservations will ensure that you get the correct hardware for your Nutanix deployments.

SSH failures while running on macOS

The Nutanix devices have sshd configured with MaxSessions 1. In most cases this is not a problem, but in our testing on macOS we observed frequent SSH connection errors. These connection errors can be resolved by turning off the SSH agent in your terminal before running terraform apply. To turn off your SSH agent in a macOS terminal, run unset SSH_AUTH_SOCK.

Other Timeouts and Connection issues

This POC project has not ironed out all potential networking and provisioning timing hiccups that can occur. In many situations, running terraform apply again will progress the deployment to the next step. If you do not see progress after 3 attempts, open an issue on GitHub: https://github.com/equinix-labs/terraform-equinix-metal-nutanix-cluster/issues/new.

Examples

To view examples for how you can leverage this module, please see the examples directory.

Requirements

Name	Version
terraform	>= 1.0
equinix	>= 1.30
local	>= 2.5
null	>= 3
random	>= 3

Providers

Name	Version
equinix	>= 1.30
local	>= 2.5
null	>= 3
random	>= 3
terraform	n/a

Modules

Name	Source	Version
ssh	./modules/ssh/	n/a

Resources

Name	Type
equinix_metal_device.bastion	resource
equinix_metal_device.nutanix	resource
equinix_metal_gateway.gateway	resource
equinix_metal_port.bastion_bond0	resource
equinix_metal_port.nutanix	resource
equinix_metal_project.nutanix	resource
equinix_metal_reserved_ip_block.nutanix	resource
equinix_metal_vlan.nutanix	resource
equinix_metal_vrf.nutanix	resource
null_resource.finalize_cluster	resource
null_resource.get_cvm_ip	resource
null_resource.reboot_nutanix	resource
null_resource.wait_for_dhcp	resource
null_resource.wait_for_firstboot	resource
random_string.vrf_name_suffix	resource
terraform_data.input_validation	resource
equinix_metal_project.nutanix	data source
equinix_metal_vlan.nutanix	data source
local_file.cvm_ip_address	data source

Inputs

Name	Description	Type	Default	Required
metal_auth_token	Equinix Metal API token.	string	n/a	yes
metal_metro	The metro to create the cluster in.	string	n/a	yes
create_project	(Optional) to use an existing project matching metal_project_name, set this to false.	bool	true	no
create_vlan	Whether to create a new VLAN for this project.	bool	true	no
metal_bastion_plan	Which plan to use for the bastion host.	string	"m3.small.x86"	no
metal_nutanix_os	Which OS to use for the Nutanix nodes.	string	"nutanix_lts_6_5"	no
metal_nutanix_plan	Which plan to use for the Nutanix nodes (must be Nutanix compatible, see https://deploy.equinix.com/developers/os-compatibility/)	string	"m3.large.x86"	no
metal_organization_id	The ID of the Metal organization in which to create the project if create_project is true.	string	null	no
metal_project_id	The ID of the Metal project in which to deploy to cluster. If create_project is false and you do not specify a project name, the project will be looked up by ID. One (and only one) of metal_project_name or metal_project_id is required or metal_project_id must be set.	string	""	no
metal_project_name	The name of the Metal project in which to deploy the cluster. If create_project is false and you do not specify a project ID, the project will be looked up by name. One (and only one) of metal_project_name or metal_project_id is required or metal_project_id must be set. Required if create_project is true.	string	""	no
metal_vlan_description	Description to add to created VLAN.	string	"ntnx-demo"	no
metal_vlan_id	ID of the VLAN you wish to use.	number	null	no
nutanix_node_count	The number of Nutanix nodes to create.	number	3	no
nutanix_reservation_ids	Hardware reservation IDs to use for the Nutanix nodes. If specified, the length of this list must be the same as nutanix_node_count. Each item can be a reservation UUID or next-available. If you use reservation UUIDs, make sure that they are in the same metro specified in metal_metro.	list(string)	[]	no
skip_cluster_creation	Skip the creation of the Nutanix cluster.	bool	false	no

Outputs

Name	Description
bastion_public_ip	The public IP address of the bastion host
cvim_ip_address	The IP address of the CVM
iscsi_data_services_ip	Reserved IP for cluster ISCSI Data Services IP
nutanix_sos_hostname	The SOS address to the nutanix machine.
prism_central_ip_address	Reserved IP for Prism Central VM
ssh_forward_command	SSH port forward command to use to connect to the Prism GUI
ssh_private_key	The private key for the SSH keypair
virtual_ip_address	Reserved IP for cluster virtal IP

Contributing

If you would like to contribute to this module, see CONTRIBUTING page.

License

Apache License, Version 2.0. See LICENSE.