Disconnected OpenShift Testbed

A terraform module for creating a testbed for Disconnected OpenShift clusters. It configures resources on Amazon to emulate a disconnected environment, providing the means for a user to practice or demonstrate "sneakernet" use-cases for getting OpenShift content into a disconnected environment on AWS, while still allowing for API connectivity. This is similar enough to other IPI use cases (Metal3, vSphere) that are very common on other infrastructures, as well as closely related to the experience on "disconnected" AWS overlays, to be a meaningful test bench.

For an example of this Terraform module's intended use cases, including more robust configuration post-provisioning please see the oc-mirror E2E testing framework repository.

Basic Information

Environment Prerequisites

• Terraform >= 1.4.2
• An AWS profile defined in ~/.aws/credentials or IAM credentials (access key id and secret is fine) exported
  • This AWS profile needs to have a significant amount of privilege
  • You should have a Route53 Hosted Zone available on the account

What do I get?

This terraform module creates a VPC with 7 subnets across 3 availability zones:

• Three of those subnets (one in each zone) are on a traditional AWS IGW that enables outbound internet access
  • These are the "connected" subnets
• One of those subnets is the designated NAT subnet, with a unique CIDR, access to the IGW, and source/dest check disabled on AWS
• Three of them (one in each zone) are on an AWS subnet with a route set to the instance in the NAT subnet
  • These are the "isolated" subnets

Three instances are stood up:

• One in a "connected" subnet, designated to be a registry
  • TCP ports 443 and 80 are allowed to this instance
  • An elastic IP and Route53 record are created for this instance
  • You need to configure this as a registry yourself
• One in the NAT subnet, designated to be the proxy for isolated subnets
  • An elastic IP and Route53 record are created for this instance
  • You need to configure this as a proxy yourself
• One in an "isolated" subnet, designated to be the disconnected bastion
  • You need to ensure that proxy configuration meets your expectations for this instance

An S3 bucket and IAM credentials to use it (read/write) are created:

• These are designed for use by the registry

And finally, a set of IAM credentials which are adequate to install OpenShift are created.

Example usage

Reference the module from your own terraform, with an initialized AWS provider. An example terraform file might look like this:

module "testbed" {
  source = "github.com/jharmison-redhat/disconnected-openshift-testbed"

  public_key     = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIC5Da2XARZmB8KsjASv6MQoAS6sAXrw0yE5Y8ANJ5yTG"
  cluster_name   = "disco"
  cluster_domain = "redhat4govaws.io"
}

provider "aws" {
  region = "us-west-2"
  default_tags {
    tags = {
      Name    = "disco.redhat4govaws.io"
      Project = "disconnected-openshift-testbed"
    }
  }
}

Some more robust examples, including parametrized ones with outputs, are available in the examples directory. Terraform outputs are expected to be required for consumption by follow-on automation.

Module Documentation

Requirements

Name	Version
terraform	>= 1.4.2
aws	4.58.0

Providers

Name	Version
aws	4.58.0

Modules

Name	Source	Version
ocp_installer	./modules/ocp_installer	n/a
registry	./modules/registry	n/a
vpc	./modules/vpc	n/a

Resources

Name	Type
aws_key_pair.ec2_key	resource
aws_ami.rhel	data source
aws_availability_zones.available	data source
aws_region.current	data source
aws_route53_zone.public	data source

Inputs

Name	Description	Type	Default	Required
ami_type	The AMI type to use, Access2 or Hourly. Availability may depend on your AWS account being linked with Red Hat Cloud Access.	string	"Hourly"	no
bastion_disk_gb	The size of the disk, in GB, for the bastion instance. Expected to be large, to support sneakernetting of content.	number	500	no
cluster_domain	The name of the domain under which your OpenShift cluster will reside (Note that this needs to be a Hosted Zone managed in Route53).	string	n/a	yes
cluster_name	The name you will be giving your OpenShift cluster in metadata.name in install-config.yaml. A new private HostedZone will be created for this as a subdomain of the cluster_domain.	string	n/a	yes
instance_password	The password to set for the ec2-user on created instances.	string	""	no
large_flavor	The AWS flavor to use for the larger instance (registry, bastion).	string	"t3.large"	no
proxy_disk_gb	The size of the disk, in GB, for the proxy instance.	number	20	no
public_key	The SSH public key string to use for the instances.	string	n/a	yes
registry_disk_gb	The size of the disk, in GB, for the registry instance. Since the registry instance is expected to use S3 storage, can be small.	number	20	no
rhel_version	The major version of RHEL to use for the AMI selection.	string	"8"	no
small_flavor	The AWS flavor to use for smaller instance (proxy).	string	"t3.small"	no

Outputs

Name	Description
bastion_instance	Information about the bastion instance.
ocp_installer	The IAM Access Key ID and Secret for the OpenShift installation user.
private_dns_zone	The DNS name of the new private DNS zone, internal to the VPC.
proxy_instance	Information about the proxy instance.
registry_bucket	The AWS S3 bucket for the registry and IAM credentials required to access it.
registry_instance	Information about the registry instance.
vpc	Information about the provisioned VPC and its networks.

VPC Submodule Documentation

Requirements

Name	Version
terraform	>= 1.4.2
aws	4.58.0

Providers

Name	Version
aws	4.58.0

Modules

No modules.

Resources

Name	Type
aws_default_route_table.public	resource
aws_default_security_group.default	resource
aws_eip.proxy	resource
aws_instance.bastion	resource
aws_instance.proxy	resource
aws_internet_gateway.default	resource
aws_route53_record.bastion_private	resource
aws_route53_record.proxy	resource
aws_route53_record.proxy_private	resource
aws_route53_zone.private	resource
aws_route_table.private	resource
aws_route_table_association.private	resource
aws_subnet.nat	resource
aws_subnet.private	resource
aws_subnet.public	resource
aws_vpc.vpc	resource

Inputs

Name	Description	Type	Default	Required
ami_id	The AMI to use for the proxy and bastion instances.	string	n/a	yes
availability_zones	The availability zones to create subnets for.	list(string)	n/a	yes
bastion_disk_gb	The size of the disk, in GB, for the bastion instance. Expected to be large, to support sneakernetting of content.	number	500	no
bastion_flavor	The instance type to use for the isolated bastion host.	string	"t3.small"	no
bastion_hostname	The hostname to use when building the bastion instance.	string	"bastion"	no
cluster_domain	The name of the domain under which your OpenShift cluster will reside (Note that this needs to be a Hosted Zone managed in Route53).	string	n/a	yes
cluster_name	The name you will be giving your OpenShift cluster in metadata.name in install-config.yaml. A new private HostedZone will be created for this as a subdomain of the cluster_domain.	string	n/a	yes
hosted_zone	The Route53 Hosted Zone ID for the the cluster_domain, for creating public proxy records.	string	n/a	yes
instance_password	The password to set for the ec2-user on the proxy and bastion instances.	string	""	no
proxy_disk_gb	The size of the disk, in GB, for the proxy instance.	number	20	no
proxy_flavor	The instance type to use for the proxy instance.	string	"t3.small"	no
proxy_hostname	The hostname to use when building the proxy instance and creating Route 53 records for it.	string	"proxy"	no
ssh_key	The SSH public key to use for the proxy and bastion instances - must already exist as an aws_key_pair!	string	n/a	yes
vpc_cidr	The CIDR-notation subnet for the entire VPC.	string	"10.0.0.0/16"	no

Outputs

Name	Description
bastion_instance	Information about the bastion instance.
private_subnets	Details about the subnets that are isolated by routing through the proxy.
private_zone	The private Hosted Zone created for the VPC.
proxy_instance	Information about the proxy instance.
public_subnets	Details about the subnets that route through the IGW to the public internet.

Registry Submodule Documentation

Requirements

Name	Version
terraform	>= 1.4.2
aws	4.58.0

Providers

Name	Version
aws	4.58.0

Modules

No modules.

Resources

Name	Type
aws_eip.registry	resource
aws_iam_access_key.registry	resource
aws_iam_policy.registry	resource
aws_iam_user.registry	resource
aws_iam_user_policy_attachment.registry	resource
aws_instance.registry	resource
aws_kms_key.registry	resource
aws_route53_record.registry	resource
aws_route53_record.registry_private	resource
aws_s3_bucket.registry	resource
aws_s3_bucket_public_access_block.registry	resource
aws_security_group.registry	resource
aws_subnet.registry	data source
aws_vpc.disco	data source

Inputs

Name	Description	Type	Default	Required
ami_id	The ID of the AMI that should be used for the registry.	string	n/a	yes
availability_zone	The availability zone into which the registry instance should be placed - should align with the subnet's zone.	string	n/a	yes
cluster_domain	The name of the domain under which your OpenShift cluster will reside. Will be used in the construction of DNS records for the registry in the public and private zones.	string	n/a	yes
cluster_name	The name you will be giving your OpenShift cluster in metadata.name in install-config.yaml. Will be used in the construction of DNS records for the registry in the public and private zones.	string	n/a	yes
disk_gb	The size of the disk, in GB, for the registry instance. Since the registry instance is expected to use S3 storage, can be small.	number	20	no
flavor	The instance type to use for the registry instance.	string	"t3.large"	no
hostname	The hostname to use when building the instance and creating Route 53 records for it.	string	"registry"	no
instance_password	The password to set for the ec2-user on the registry instance.	string	""	no
private_zone_id	The Route53 Hosted Zone ID which contains the domain for creating private registry records.	string	n/a	yes
private_zone_name	The Route53 Hosted Zone name which contains the domain for creating private registry records.	string	n/a	yes
public_zone	The Route53 Hosted Zone ID which contains the domain for creating public registry records.	string	n/a	yes
ssh_key_name	The SSH public key to use for the proxy instance - must already exist as an aws_key_pair!	string	n/a	yes
subnet_id	The ID of the existing VPC subnet into which the instance should associate its default interface.	string	n/a	yes

Outputs

Name	Description
registry_instance	Information about the registry instance.
s3_bucket	The AWS S3 bucket and IAM credentials required to access it.

OCP Installer Submodule Documentation

Requirements

Name	Version
terraform	>= 1.4.2
aws	4.58.0

Providers

Name	Version
aws	4.58.0

Modules

No modules.

Resources

Name	Type
aws_iam_access_key.ocp_installer	resource
aws_iam_policy.ocp_installer	resource
aws_iam_user.ocp_installer	resource
aws_iam_user_policy_attachment.ocp_installer	resource

Inputs

Name	Description	Type	Default	Required
domain	The full name of the domain which will be used to uniquely identify the IAM user created for OpenShift installation.	string	n/a	yes

Outputs

Name	Description
ocp_installer	The IAM Access Key ID and Secret for the OpenShift installation user.