Kubernetes exposes the concept of a Cloud Provider which interfaces with the IAAS to provision TCP Load Balancers, Nodes, Networking Routes, and Persistent Volumes.
CFCR can configure Kubernetes with a Cloud Provider through the following methods. In each example it is assumed that you already have access to a BOSH Director.
-
Create a service account and IAM profiles for your master and worker nodes. This Terraform script can be used as a reference guide.
Note: The service accounts should be used per CFCR deployment (NOT per director)
-
Save the service account email addresses and the loadbalancer backend service into a vars file that will be used to create the cloud-config.
If your loadbalancer uses target pools instead, you will need to customize the vars file and the ops-file in the next step.
$ export deployment_name="your deployment name" $ cat << EOF > ${deployment_name}-cc-vars.yml cfcr_master_service_account_address: <master-service-account-email> cfcr_worker_service_account_address: <worker-service-account-email> deployment_name: <deployment-name> cfcr_backend_service: <backend-service> EOF
-
Add a cloud config for the deployment with BOSH generic configs
$ export KD="path to kubo-deployment repo" $ bosh update-config --name ${deployment_name} \ ${KD}/manifests/cloud-config/iaas/gcp/use-vm-extensions.yml \ --type cloud \ --vars-file ${deployment_name}-cc-vars.yml
-
Deploy CFCR
$ bosh deploy -d ${deployment_name} \ ${KD}/manifests/cfcr.yml \ -o ${KD}/manifests/ops-files/iaas/gcp/cloud-provider.yml \ -o ${KD}/manifests/ops-files/use-vm-extensions.yml \ -o ${KD}/manifests/ops-files/rename.yml \ -v deployment_name=${deployment_name}
-
To test that the cloud provider has been configured correctly, create a simple nginx deployment with an external load balancer
$ kubectl apply -f https://github.com/cloudfoundry-incubator/kubo-ci/raw/master/specs/nginx-lb.yml # wait for the nginx service to get an external-ip $ kubectl get services $ export external_ip=$(kubectl get service/nginx -o jsonpath={.status.loadBalancer.ingress[0].ip}) $ curl http://${external_ip}:80
-
Create IAM profiles for your master and worker nodes. This Terraform script can be used as a reference guide.
Note: The profiles should be used per CFCR deployment (NOT per director)
-
Save the profile into a vars file that will be used to create the cloud-config
$ export deployment_name="your deployment name" $ cat << EOF > ${deployment_name}-cc-vars.yml master_iam_instance_profile: <master-iam-profile-name> worker_iam_instance_profile: <worker-iam-profile-name> cfcr_master_target_pool: <list-of-elbs-for-master> kubernetes_cluster_tag: <tag-for-k8s-cluster-components> deployment_name: <deployment-name> EOF
-
Add a cloud config for the deployment with BOSH generic configs
$ export KD="path to kubo-deployment repo" $ bosh update-config --name ${deployment_name} \ ${KD}/manifests/cloud-config/iaas/aws/use-vm-extensions.yml \ --type cloud \ --vars-file ${deployment_name}-cc-vars.yml
-
Deploy CFCR
$ bosh deploy -d ${deployment_name} \ ${KD}/manifests/cfcr.yml \ -o ${KD}/manifests/ops-files/iaas/aws/cloud-provider.yml \ -o ${KD}/manifests/ops-files/use-vm-extensions.yml \ -o ${KD}/manifests/ops-files/iaas/aws/lb.yml \ -o ${KD}/manifests/ops-files/rename.yml \ -v deployment_name=${deployment_name}
-
To test that the cloud provider has been configured correctly, create a simple nginx deployment with an external load balancer
$ kubectl apply -f https://github.com/cloudfoundry-incubator/kubo-ci/raw/master/specs/nginx-lb.yml # wait for the nginx service to get an external-ip $ kubectl get services $ export external_ip=$(kubectl get service/nginx -o jsonpath={.status.loadBalancer.ingress[0].hostname}) $ curl http://${external_ip}:80
-
Create creds that will be used by the cloud-manager-controller in master nodes to talk to the vSphere api. Check vSphere Cloud Provider documentation for list of privileges.
Note: vSphere workers do not need any credentials.
-
Save the information to connect to the vCenter into deployment vars file.
$ export deployment_name="your deployment name" $ cat << EOF > ${deployment_name}-vars.yml vcenter_master_user: <user> vcenter_master_password: <password> vcenter_ip: <vsphere.server> vcenter_dc: <vsphere.datacenter> vcenter_ds: <vsphere.datastore> vcenter_vms: <vsphere.vm_folder> director_uuid: <BOSH director UUID> deployment_name: <deployment-name> EOF
See for mode details at spec and at vSphere Cloud Provider documentation You can get the BOSH director UUID from the output of
bosh envcommand. -
Add (if does not exist) a vSphere specific cloud config with BOSH generic configs
$ bosh update-config --name cfcr-cc-vm_extension-vsphere \ ${KD}/manifests/cloud-config/iaas/vsphere/use-vm-extensions.yml \ --type cloud -
Deploy CFCR
$ export KD="path to kubo-deployment repo" $ bosh deploy -d ${deployment_name} \ ${KD}/manifests/cfcr.yml \ -o ${KD}/manifests/ops-files/iaas/vsphere/cloud-provider.yml \ -o ${KD}/manifests/ops-files/iaas/vsphere/use-vm-extensions.yml \ -o ${KD}/manifests/ops-files/rename.yml \ -v deployment_name=${deployment_name} \ --vars-file ${deployment_name}-vars.yml
NOTE: If the vSphere api is behind a proxy create the following ops file and add it when deploying with
-o add-proxy.ymlcat << EOF > add-proxy.yml - type: replace path: /instance_groups/name=master/jobs/name=kube-controller-manager/properties/http_proxy? value: ((http_proxy)) - type: replace path: /instance_groups/name=master/jobs/name=kube-controller-manager/properties/https_proxy? value: ((https_proxy)) - type: replace path: /instance_groups/name=master/jobs/name=kube-controller-manager/properties/no_proxy? value: ((no_proxy)) EOF
NOTE: If everything is behind a proxy add the following ops file when deploying with
-o {KD}/kubo-deployment/manifests/ops-files/add-proxy.yml -
To test that the cloud provider has been configured correctly, create a simple a workload with persistence volume
$ kubectl apply -f https://github.com/cloudfoundry-incubator/kubo-ci/raw/master/specs/storage-class-vsphere.yml $ kubectl apply -f https://github.com/cloudfoundry-incubator/kubo-ci/raw/master/specs/persistent-volume-claim.yml # wait for the volume to be attached $ kubectl describe pvc ci-claim # Type Reason Age From Message # ---- ------ ---- ---- ------- # Normal ProvisioningSucceeded 31s persistentvolume-controller Successfully provisioned volume ... #
NOTE: vSphere cloud-provider does not support service of type LoadBalancer
-
Save the information to connect to the OpenStack into deployment vars file.
$ export deployment_name="your deployment name" $ cat << EOF > ${deployment_name}-vars.yml auth_url: < authentication URL > openstack_domain: < domain > openstack_project_id: < tenant id > region: < region > openstack_username: < admin username > openstack_password: < admin user password > EOF
See for mode details at spec.
-
Deploy CFCR
$ export KD="path to kubo-deployment repo" $ bosh deploy -d ${deployment_name} \ ${KD}/manifests/cfcr.yml \ -o ${KD}/manifests/ops-files/iaas/openstack/cloud-provider.yml \ -o ${KD}/manifests/ops-files/rename.yml \ -v deployment_name=${deployment_name} \ --vars-file ${deployment_name}-vars.yml
-
To test that the cloud provider has been configured correctly, create a simple a workload with persistence volume
$ kubectl apply -f https://github.com/cloudfoundry-incubator/kubo-ci/raw/master/specs/storage-class-openstack.yml $ kubectl apply -f https://github.com/cloudfoundry-incubator/kubo-ci/raw/master/specs/persistent-volume-claim.yml # wait for the volume to be attached $ kubectl describe pvc ci-claim # Type Reason Age From Message # ---- ------ ---- ---- ------- # Normal ProvisioningSucceeded 31s persistentvolume-controller Successfully provisioned volume ... #