Deploying a UPI environment for OpenShift 4.1 on vSphere
NOTE: This process is not supported by Red Hat and is provided “as-is”. However, OpenShift 4.1 is fully supported on a user-provided infrastructure vSphere environment.
Please refer to the OpenShift docs for more information regarding supported configurations.
This process will utilize terraform to deploy the infrastructure used by the OpenShift Installer.
Please install the appropriate version for your operating system:
https://www.terraform.io/downloads.html
This article was created utilizing Terraform v0.11.12.
Download and import the Red Hat CoreOS image for vSphere
Download the latest version of govc and use it to import the OVA:
$ cd /tmp/
$ curl -L https://github.com/vmware/govmomi/releases/download/v0.20.0/govc_linux_amd64.gz | gunzip > /usr/local/bin/govc
$ curl -O https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.1/latest/rhcos-4.1.0-x86_64-vmware.ova
$ chmod +x /usr/local/bin/govc
$ export GOVC_URL='vcsa.example.com'
$ export GOVC_USERNAME='administrator@vsphere.local'
$ export GOVC_PASSWORD='password'
$ export GOVC_NETWORK='VM Network'
$ export GOVC_DATASTORE='vmware-datastore'
$ export GOVC_INSECURE=1 # If the host above uses a self-signed cert
$ govc import.ova -name=rhcos-latest ./rhcos-410.8.20190516.0-vmware.ova
This will provide a working Red Hat CoreOS image to use for deploying the user provided infrastructure environment (UPI).
Create the applicable DNS records for the OpenShift deployment
Take note of the SRV records for etcd. Also, examine the bootstrap-0 DNS records: host, api and api-int.
$ORIGIN apps.upi.example.com.
* A 10.x.y.38
* A 10.x.y.39
* A 10.x.y.40
$ORIGIN upi.example.com.
_etcd-server-ssl._tcp SRV 0 10 2380 etcd-0
_etcd-server-ssl._tcp SRV 0 10 2380 etcd-1
_etcd-server-ssl._tcp SRV 0 10 2380 etcd-2
bootstrap-0 A 10.x.y.34
control-plane-0 A 10.x.y.35
control-plane-1 A 10.x.y.36
control-plane-2 A 10.x.y.37
api A 10.x.y.34
api A 10.x.y.35
api A 10.x.y.36
api A 10.x.y.37
api-int A 10.x.y.34
api-int A 10.x.y.35
api-int A 10.x.y.36
api-int A 10.x.y.37
etcd-0 A 10.x.y.35
etcd-1 A 10.x.y.36
etcd-2 A 10.x.y.37
compute-0 A 10.x.y.38
compute-1 A 10.x.y.39
compute-2 A 10.x.y.40
To learn more about required DNS the OpenShift docs discuss further.
Create a working install-config file for the installer to consume
Download the latest 4.1 installer and client from the repository.
$ cd /tmp/
$ curl -O https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/openshift-install-linux-4.1.7.tar.gz
$ curl -O https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/openshift-client-linux-4.1.7.tar.gz
$ tar xzvf openshift-install-linux-4.1.7.tar.gz
$ tar xzvf openshift-client-linux-4.1.7.tar.gz
$ mv oc kubectl openshift-install /usr/local/bin/
$ mkdir ~/vsphere/
$ vi ~/vsphere/install-config.yaml
apiVersion: v1
baseDomain: example.com
metadata:
name: upi
networking:
machineCIDR: "10.x.y.0/23"
platform:
vsphere:
vCenter: vcsa67.example.com
username: "administrator@vsphere.local"
password: "password"
datacenter: dc
defaultDatastore: vmware-datastore
pullsecret: ‘omitted’
sshKey: ‘omitted’
$ openshift-install --dir ~/vsphere create ignition-configs
This will generate ignition configuration files that will need to be added to the terraform tfvars in the next section. Note, that the bootstrap ignition config is too large to be added via a virtual appliance property and will need to be stored somewhere external. The example listed below leverages a github gist.
Download the UPI terraform included in the OpenShift installer repository and update provisioning variables
Edit terraform.tfvars with the appropriate variables including the ignition configs from the previous steps:
$ git clone -b release-4.1 https://github.com/openshift/installer
$ vi openshift/installer/upi/vsphere/terraform.tfvars
bootstrap_ip = "10.x.y.34"
control_plane_ips = ["10.x.y.35","10.x.y.36","10.x.y.37"]
compute_ips = ["10.x.y.38","10.x.y.39","10.x.y.40"]
machine_cidr = "10.x.y.0/23"
cluster_id = "upi"
cluster_domain = "upi.example.com"
base_domain = "example.com"
vsphere_server = "vcsa67.example.com"
vsphere_user = "administrator@vsphere.local"
vsphere_password = "password"
vsphere_cluster = "cluster"
vsphere_datacenter = "dc"
vsphere_datastore = "vmware-datastore”
vm_template = "rhcos-latest"// Due to size limitations for the OVA use a URL for the bootstrap ignition config file
bootstrap_ignition_url = "https://gist.githubusercontent.com/test/12345/raw/12345/bootstrap.ign"// Ignition config for the control plane machines. Copy the contents of the master.ign generated by the installer.
control_plane_ignition = <<END_OF_MASTER_IGNITION
{"ignition":{"config":{"append":[{"source":"https://api-int.upi.example.com:22623/config/master","verification":{}}]},"security":{"tls":{"certificateAuthori
..omitted..
END_OF_MASTER_IGNITION
// Ignition config for the compute machines. Copy the contents of the worker.ign generated by the installer.
compute_ignition = <<END_OF_WORKER_IGNITION
{"ignition":{"config":{"append":[{"source":"https://api-int.upi.example.com:22623/config/worker","verification":{}}]},"security":{"tls":{"certificateAuthori
..omitted..
END_OF_WORKER_IGNITION
Edit the ignition.tf to update the gateway and DNS server to use for VM deployment:
$ vi openshift/installer/upi/vsphere/machine/ignition.tf
gw = "10.x.y.254"
$ vi openshift/installer/upi/vsphere/machine/ignition.tf
DNS1=10.x.y.2
Remove the route53 section since DNS has been created above.
$ vi openshift/installer/upi/vsphere/main.tf
/*
module "dns" {
source = "./route53"
base_domain = "${var.base_domain}"
cluster_domain = "${var.cluster_domain}"
bootstrap_count = "${var.bootstrap_complete ? 0 : 1}"
bootstrap_ips = ["${module.bootstrap.ip_addresses}"]
control_plane_count = "${var.control_plane_count}"
control_plane_ips = ["${module.control_plane.ip_addresses}"]
compute_count = "${var.compute_count}"
compute_ips = ["${module.compute.ip_addresses}"]
}
*/
Run terraform to initialize and provision the environment
$ cd openshift/installer/upi/vsphere/
$ terraform init
Initializing modules...
- module.folder
Getting source "./folder"
- module.resource_pool
- module.bootstrap
- module.control_plane
- module.compute
Initializing provider plugins...
..omitted..
$ terraform apply -auto-approve
..omitted..
module.control_plane.vsphere_virtual_machine.vm[0]: Creation complete after 40s (ID: 4203f331-984c-8103-83a9-20d760fbf605)
module.bootstrap.vsphere_virtual_machine.vm: Creation complete after 40s (ID: 4203d5ab-d5e1-4ca7-eece-66aeb4a52b7d)
Apply complete! Resources: 9 added, 0 changed, 0 destroyed.
The environment should now be deployed in vCenter:
Inform the installer that the bootstrap process has completed and delete the bootstrap VM
$ openshift-install --dir ~/vsphere wait-for bootstrap-complete
INFO Waiting up to 30m0s for the Kubernetes API at https://api.upi.example.com:6443...
INFO API v1.14.0+ef1f86e up
INFO Waiting up to 30m0s for bootstrapping to complete...
INFO It is now safe to remove the bootstrap resources
$ terraform apply -auto-approve -var 'bootstrap_complete=true'
..omitted..
module.bootstrap.vsphere_virtual_machine.vm: Destruction complete after 7s
Apply complete! Resources: 0 added, 0 changed, 1 destroyed.
The Cluster Image Registry does not pick a storage backend for vSphere platform. Therefore, the cluster operator will be stuck in progressing because it is waiting for an administrator to configure a storage backend for the image-registry. See the following bugzilla for details.
Set EmptyDir for non-production clusters by issuing the following commands:
$ cp ~/vsphere/auth/kubeconfig ~/.kube/config
$ oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"storage":{"emptyDir":{}}}}'
Delete the bootstrap's DNS records and inform the installer to proceed
Note, the remaining api and api-int records should continue pointing towards the control-plane servers.
$ openshift-install --dir ~/vsphere wait-for install-complete
INFO Waiting up to 30m0s for the cluster at https://api.upi.example.com:6443 to initialize...
INFO Waiting up to 10m0s for the openshift-console route to be created...
INFO Install complete!
INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/root/go/src/github.com/openshift/installer/bin/auth/kubeconfig'
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.upi.e2e.bos.redhat.com
INFO Login to the console with user: kubeadmin, password: my-kube-password
Enjoy the OpenShift Container Platform 4.1 cluster on a VMware software-defined datacenter!
Cleaning up
To remove the virtual machines from vCenter, terraform can be run with the destroy option:
$ terraform destroy -auto-approve
