How to install OpenShift on VMware with Terraform and static IP addresses

2020년 5월 13일Alejandro Guadarrama6분 읽기
Hybrid cloud Automation and management Virtualization

The following procedure is intended to create VM's from an OVA template booting with static IP's when the DHCP server can not reserve the IP addresses.

The problem

OCP requires that all DNS configurations be in place. VMWare requires that the DHCP assign the correct IPs to the VM. Since many real installations require the coordination with different teams in an organization, many times we don't have control of DNS, DHCP or Loadbalancer configurations.

Sometimes we need to do a "bare metal" installation over vmware to set the network configuration with kernel parameters at boot (ip, gateway, nameserver, etc.).

The coreos documentation explain how to create configurations using ignition files. I created a python script to put the network configuration using the ignition files created by the openshift-install program.

How does it work?

When the VM boots, it will take the first IP provided by the DHCP server (probably will not be the IP set by the script). (...dhclient[836]: bound to 10.56.240.99 -- renewal in 18253 seconds.)

Temporary IP

At the end of the boot process the VM will take the IP provided by the ignition file.

Final IP

There are two parts:

  1. Create the base64 encoding strings for each VM that includes the network configuration and hostname.
  2. Deploy the VM using terraform.

  • What do the modified ignition files contain?

Basically the ignition files are a json file; there are 3:

  • bootstrap.ign: It contains all the cluster configurations to create the OCP cluster.
  • master.ign: It contains the URL where the master node will pull the configuration, and the certificate to connect with the REST API of OCP.
  • worker.ign: It contains the same content as the master but pointing to the worker URL.

Example append-bootstrap.ign from the documentation.

{
  "ignition": {
    "config": {
      "append": [
        {
          "source": "http://10.56.241.10/bootstrap.ign",
          "verification": {}
        }
      ]
    },
    "timeouts": {},
    "version": "2.1.0"
  },
  "networkd": {},
  "passwd": {},
  "storage": {},
  "systemd": {}
}

Modified append-bootstrap.ign injecting network configurations. The Python script injects 2 values in the storage section /etc/hostname and /etc/sysconfig/network-scripts/ifcfg-ens192. In the networkd the script creates the unit 00-ens192.network. We still need to pull the bootstrap.ign from the webserver (due to the size of the file).

{
  "ignition": {
    "config": {
      "append": [
        {
          "source": "http://10.56.241.10/bootstrap.ign",
          "verification": {}
        }
      ]
    },
    "timeouts": {},
    "version": "2.1.0"
  },
  "networkd": {
    "units": [
      {
        "name": "00-ens192.network",
        "contents": "[Match]\nName=ens192\n\n[Network]\nAddress=.10.56.241.12/23\nGateway=10.56.240.254\nDNS=10.56.240.1\nDomains=example.com\n"
      }
    ]
  },
  "passwd": {},
  "storage": {
    "files": [
      {
        "path": "/etc/hostname",
        "mode": 420,
        "contents": {
          "source": "data:text/plain;charset=utf-8;base64,Ym9vdHN0cmFwLm9jcDQuZXhhbXBsZS5jb20=",
          "verification": {}
        },
        "filesystem": "root"
      },
      {
        "path": "/etc/sysconfig/network-scripts/ifcfg-ens192",
        "mode": 420,
        "contents": {
          "source": "data:text/plain;charset=utf-8;base64,IyBHZW5lcmF0ZWQgYnkgY3JlYXRlX2Jhc2U2NF9maWxlcwpUWVBFPUV0aGVybmV0Ck5BTUU9ImVuczE5MiIKREVWSUNFPSJlbnMxOTIiCk9OQk9PVD15ZXMKTkVUQk9PVD15ZXMKQk9PVFBST1RPPW5vbmUKSVBBRERSPSIxMC41Ni4yNDEuMTIiCk5FVE1BU0s9IjI1NS4yNTUuMjU0LjAiCkdBVEVXQVk9IjEwLjU2LjI0MC4yNTQiCkROUzE9IjEwLjU2LjI0MC4xIgo=",
          "verification": {}
        },
        "filesystem": "root"
      }
    ]
  },
  "systemd": {}
}

The Python script does the same for master.ign and worker.ign.

Pre-requirements

  • Python installed. (I use 3.8.2, I love to use pyenv)
  • Terraform latest version (v0.12.24) you can download here
  • Have the ignitions files already created (master.ign, worker.ign)
  • Update the values for your corresponding user, passwords, URL's, etc.

Procedure

  1. The OCP installation dir in my case is deploy/. Run the python script. This will create the file terraform/vars/machine_data.tfvars. We will need this file in the terraform procedure.
(vmware) alex@:/ocp4-vmware-terraform $ ./create_base64_files
(vmware) alex@:/ocp4-vmware-terraform $

  1. Modify the file terraform/vars/common.tfvars with the information of your vmware server.

  2. Change to the terraform folder. Unfortunately the terraform provider can't deploy an OVF/OVA template, seems it has a bug. We will use govc to upload the ova file and create the template.

(vmware) alex@:/ocp4-vmware-terraform $ cd terraform
alex@:/../terraform $ govc import.ova -dc="datacenter" -ds="datastore1" -pool="sni/Resources" -name="rhcos-4.4.3-x86_64-vmware.x86_64.ova" "https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/latest/latest/rhcos-4.4.3-x86_64-vmware.x86_64.ova"
alex@:/../terraform $ govc snapshot.create -dc="datacenter" -vm="rhcos-4.4.3-x86_64-vmware.x86_64.ova" clone-root
alex@:/../terraform $ govc vm.markastemplate -dc="datacenter" "rhcos-4.4.3-x86_64-vmware.x86_64.ova"

OVA template

  1. Create a plan in terraform. I divided it into 4 modules: bootstrap, master, worker and infra. We can create a plan to create 8 VM's (3 masters, 2 workers, 2 infras, and the bootstrap); but I recommend creating one plan at time, starting with the bootstrap.
alex@:/../terraform $ ./plan -target=module.bootstrap
Initializing modules...
Initializing the backend...
Initializing provider plugins...
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create
 <= read (data resources)
Terraform will perform the following actions:
....
Plan: 2 to add, 0 to change, 0 to destroy.
.
Warning: Resource targeting is in effect
.
You are creating a plan with the -target option, which means that the result
of this plan may not represent all of the changes requested by the current
configuration.
.
The -target option is not for routine use, and is provided only for
exceptional situations such as recovering from errors or mistakes, or when
Terraform specifically suggests to use it as part of an error message.
.
.
This plan was saved to: /mnt/z/Work/RedHat/ocp4-vmware-terraform/terraform/outputs/tf.output
.
To perform exactly these actions, run the following command to apply:
    terraform apply "/mnt/z/Work/RedHat/ocp4-vmware-terraform/terraform/outputs/tf.output"
.
.
  1. Apply the plan created.
alex@:/../terraform $ ./apply
Initializing modules...
Initializing the backend...
Initializing provider plugins...
.
.
.
data.vsphere_datacenter.dc: Refreshing state...
data.vsphere_datastore.datastore: Refreshing state...
data.vsphere_resource_pool.sni: Refreshing state...
data.vsphere_virtual_machine.rhcos_template: Refreshing state...
data.vsphere_network.network: Refreshing state...
data.vsphere_host.esxi67: Refreshing state...
vsphere_folder.cluster: Creating...
vsphere_folder.cluster: Creation complete after 1s [id=group-v185]
module.bootstrap.vsphere_virtual_machine.clone[0]: Creating...
module.bootstrap.vsphere_virtual_machine.clone[0]: Still creating... [10s elapsed]
module.bootstrap.vsphere_virtual_machine.clone[0]: Still creating... [20s elapsed]
module.bootstrap.vsphere_virtual_machine.clone[0]: Still creating... [30s elapsed]
module.bootstrap.vsphere_virtual_machine.clone[0]: Still creating... [40s elapsed]
module.bootstrap.vsphere_virtual_machine.clone[0]: Still creating... [50s elapsed]
module.bootstrap.vsphere_virtual_machine.clone[0]: Still creating... [1m0s elapsed]
.
.
.
module.bootstrap.vsphere_virtual_machine.clone[0]: Still creating... [3m10s elapsed]
module.bootstrap.vsphere_virtual_machine.clone[0]: Still creating... [3m20s elapsed]
module.bootstrap.vsphere_virtual_machine.clone[0]: Creation complete after 3m23s [id=423147d7-b7ff-11d2-1762-a2e10dd37584]
.
Warning: Applied changes may be incomplete
.
The plan was created with the -target option in effect, so some changes
requested in the configuration may have been ignored and the output values may
not be fully updated. Run the following command to verify that no other
changes are pending:
    terraform plan
.
Note that the -target option is not suitable for routine use, and is provided
only for exceptional situations such as recovering from errors or mistakes, or
when Terraform specifically suggests to use it as part of an error message.
.
Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
.
The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.
.
State path: terraform.tfstate

The output above shows 2 resources added, 1 is the folder ocp4 and the 2nd is the VM bootstrap.

  Boostrap created

  1. Repeat the the plan/apply for the masters, workers and infra (in case that you have).
alex@:/../terraform $ ./plan -target=module.master
.
Plan: 3 to add, 0 to change, 0 to destroy.
.
alex@:/../terraform $ ./apply
.
.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
module.master.vsphere_virtual_machine.clone[2]: Creating...
module.master.vsphere_virtual_machine.clone[0]: Creating...
module.master.vsphere_virtual_machine.clone[1]: Creating...
.
.
.
alex@:/../terraform $ ./plan -target=module.worker
.
alex@:/../terraform $ ./apply
.
.
.
alex@:/../terraform $ ./plan -target=module.infra
.
alex@:/../terraform $ ./apply
.
.

Master completed

Working manually with vmware

If you want to create the VM machines manually you can copy/paste the base64 strings to the manual procedure in vmware. Go to terraform/vars/machine_data.tfvars you can see the values for each server in base64 encoding.  Manual procedire

Destroy

You can easily destroy the bootstrap once you don't need it anymore.

alex@:/../terraform $ ./destroy --target=module.bootstrap
data.vsphere_datacenter.dc: Refreshing state...
data.vsphere_host.esxi67: Refreshing state...
Plan: 0 to add, 0 to change, 1 to destroy.
.
Warning: Resource targeting is in effect
.
Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.
.
  Enter a value: yes
.  
module.bootstrap.vsphere_virtual_machine.clone[0]: Destroying... [id=423147d7-b7ff-11d2-1762-a2e10dd37584]
module.bootstrap.vsphere_virtual_machine.clone[0]: Destruction complete after 9s
.
Destroy complete! Resources: 1 destroyed.

Or you can destroy everything

alex@:/../terraform $ ./destroy
Plan: 0 to add, 0 to change, 4 to destroy.
.
Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.
.
  Enter a value: yes
  module.master.vsphere_virtual_machine.clone[1]: Destroying... [id=4231f92a-08c9-6afe-7f0c-b17105e7991a]
  module.master.vsphere_virtual_machine.clone[0]: Destroying... [id=4231da99-3018-40dd-2532-157990aea86f]
  module.master.vsphere_virtual_machine.clone[2]: Destroying... [id=423188fc-571c-f243-367c-c6edaf8906b2]
  module.master.vsphere_virtual_machine.clone[2]: Destruction complete after 4s
  module.master.vsphere_virtual_machine.clone[1]: Destruction complete after 4s
  module.master.vsphere_virtual_machine.clone[0]: Destruction complete after 5s
  vsphere_folder.cluster: Destroying... [id=group-v185]
  vsphere_folder.cluster: Destruction complete after 0s
.  
  Destroy complete! Resources: 4 destroyed.

References

저자 소개

Alejandro Guadarrama

Alejandro Guadarrama

UI_Icon-Red_Hat-Close-A-Black-RGB

유사한 검색 결과

Blog post

Slash VM provisioning time on Red Hat Openshift Virtualization using Red Hat Ansible Automation Platform

Blog post

Red Hat Ansible Automation Platform: Measuring Business Impact with Dashboard and Analytics

오리지널 팟캐스트

Technically Speaking | Taming AI agents with observability

오리지널 팟캐스트

Ready To Launch | Compiler

채널별 검색

모든 채널 탐색
automation icon

오토메이션

기술, 팀, 인프라를 위한 IT 자동화 최신 동향
AI icon

인공지능

고객이 어디서나 AI 워크로드를 실행할 수 있도록 지원하는 플랫폼 업데이트
open hybrid cloud icon

오픈 하이브리드 클라우드

하이브리드 클라우드로 더욱 유연한 미래를 구축하는 방법을 알아보세요
security icon

보안

환경과 기술 전반에 걸쳐 리스크를 감소하는 방법에 대한 최신 정보
edge icon

엣지 컴퓨팅

엣지에서의 운영을 단순화하는 플랫폼 업데이트
Infrastructure icon

인프라

세계적으로 인정받은 기업용 Linux 플랫폼에 대한 최신 정보
application development icon

애플리케이션

복잡한 애플리케이션에 대한 솔루션 더 보기
Virtualization icon

가상화

온프레미스와 클라우드 환경에서 워크로드를 유연하게 운영하기 위한 엔터프라이즈 가상화의 미래