Automate your SAP HANA Scale-Up System Replication Deployment with High Availability

This post is the extension of the previous post, which talks about how to automate SAP HANA Scale-Up System Replication deployments on Red Hat Enterprise Linux (RHEL). This time, we will further automate the process using RHEL System Roles for SAP and Image Builder, and introduce the Red Hat High Availability solutions for managing SAP HANA System Replication.  

RHEL System Roles for SAP as part of RHEL for SAP Solutions is fully supported. It consists of three system roles, which are included in the RHEL for SAP Solutions repositories. 

The system role sap-preconfigure configures a system for SAP to be installed.  It includes adding packages, configuring DNS, configuring SELinux, and more. The system role sap-hana-preconfigure can apply changes to prepare the system prior to SAP HANA being installed. It is intended to be run after the SAP pre-configure role and includes elements such as setting the sap-hana tuned profile. And, system role sap-netweaver-preconfigure configures a system for SAP Netweaver to be installed.  Includes validating the machine and configuring the sap-netweaver tuned profile. 

RHEL High Availability

One of the key things customers look for when deploying SAP workloads on premise or public cloud is having high availability (HA) set up for their business/mission-critical SAP applications. Red Hat provides SAP certified high availability solutions for various SAP components. In this blog post, we discuss how to utilize the sap_hana_ha_pacemaker Ansible role to set up the Red Hat HA solution for managing SAP HANA Scale-Up System Replication on Azure. 

The RHEL for SAP Solutions subscription includes the Red Hat Enterprise Linux High Availability Add-on (RHEL HA Add-On), which  provides all the necessary packages for configuring a pacemaker-based cluster that provides reliability, scalability, and availability to critical production services. See the Automated SAP HANA System Replication in Scale-Up in pacemaker cluster and Deploying Highly Available SAP NetWeaver-based Servers Using Red Hat Enterprise Linux HA add-on with Pacemaker knowledge base articles for guidelines. 

Image Builder

RHEL 8 ships a new tool, called Image Builder, that allows you to create custom RHEL system images in a variety of formats. These include compatibility with major cloud providers and virtualization technologies available in the market. As a result, it enables you to quickly spin up new Red Hat Enterprise Linux (RHEL) systems in different platforms, according to your requirements.

In the recent RHEL updates, os-build replaces lorax-composer has been added as the new back-end for Image Builder. This updated back-end will allow for additional features to be delivered, the first of which is Push to Cloud (see the below picture).  Additional new features will be added to os-build, such as building RHEL versions different from the local system or different architectures, which is expected in future RHEL releases.

In this blog post, we will discuss how to set up a bootstrap server for SAP HANA deployment using Image Builder and libvirt tool. 

Automate your SAP HANA deployment

In the following demo, we will walk you through how to build a bootstrap server image for SAP HANA with Image Builder and libvirt tools, then discuss how to run the image on Azure Cloud, and finally discuss using the bootstrap server to set up SAP HANA Scale-Up System Replication on Azure Cloud.  

Customize OS image with Image Builder and libvirt

First, let’s set up Image Builder. You can refer to the section “Installing Image Builder” from the Red Hat Enterprise Linux 8 Image Builder: Building custom system images article. Once it is ready, you can build an OS image using Image Builder. And, you will first need to enable the required repositories on the server where the image will be created. Because you can choose the  packages which are available on the server. To do that, run the following command lines. 

Here are the repositories for RHEL for SAP Solutions: 

# subscription-manager repos \
--enable="rhel-8-for-$(uname -m)-baseos-e4s-rpms" \
--enable="rhel-8-for-$(uname -m)-appstream-e4s-rpms" \
--enable="rhel-8-for-$(uname -m)-sap-solutions-e4s-rpms" \
--enable="rhel-8-for-$(uname -m)-sap-netweaver-e4s-rpms" \
--enable="rhel-8-for-$(uname -m)-highavailability-e4s-rpms"

Enable Ansible Engine repository:

# subscription-manager repos \
--enable=ansible-2.8-for-rhel-8-$(uname -m)-rpms

Set up repositories for Azure CLI:

# rpm --import https://packages.microsoft.com/keys/microsoft.asc
# sh -c 'echo -e "[azure-cli]\nname=Azure CLI\nbaseurl=https://packages.microsoft.com/yumrepos/azure-cli\nenabled=1\ngpgcheck=1\ngpgkey=https://packages.microsoft.com/keys/microsoft.asc" > /etc/yum.repos.d/azure-cli.repo'

Once the repositories are set, log in to the Web Console of the server, and look for Image Builder on the left sidebar, which will show you all the blueprints. Create your own blueprint by clicking the “Create Blueprint” button on the top right corner, and you will see the following picture. 

After inputting the Name and Description, click "Create," which will take you to the Edit Packages page. You can pick and choose which packages you want to include in your OS images. To build a bootstrap server for SAP HANA, you will need the following packages at least: 

• ansible

• azure-cli

• gcc 

• libffi-devel 

• platform-python-devel 

• openssl-devel

• python3-pip 

• python3-wheel

• rhel-system-roles-sap

• git

• nfs-utils

Add the above packages to the blueprint, and click "Commit" twice to save the changes to your blueprint:

Once above step is complete, click "Create Image" at the top right corner:

In the next page, select “Azure Disk Image (.vhd)” as Image Type since we will boot up the image on Azure Cloud.

Then, click the “Create” button to create an image. Go to the Images page, where you can see that the image build is in progress.

Wait for the image build to complete, then you can download the image to your working directory and customize the image with libvirt tools.

Customize the image with libvirt tools

Before we start this section, you will need to set up libvirt tools by installing the packages:

# yum install qemu-kvm libvirt libvirt-python libguestfs-tools virt-install

To automate the setup of the bootstrap server, we would like to have Ansible roles and modules and playbooks to be installed and downloaded when the image is up and running, so here we use the first boot script (i.e., run_on_bootstrap.sh). 

#!/usr/bin/bash
# Download the required yml files and files
git clone https://github.com/EdwardJ1n/sap-hana-hsr-with-ha-on-azure-example.git >> /root/first_run.sh.log

# Install the required packages for the Azure Python SDK modules. 
pip3 install --upgrade pip >> /root/first_run.sh.log
pip install 'ansible[azure]' --use-feature=2020-resolver >> /root/first_run.sh.log

# Download Ansible roles for SAP HANA deployments
ansible-galaxy install -r /root/saphana_deployment/roles/requirements.yml >> /root/first_run.sh.log

Use libvirt tool to embed your first boot script using option --firstboot:

# virt-customize -a /<DIR>/bootstrap.vhd \
    --firstboot /root/run_on_bootstrap.sh \
    --root-password "password:redhat" \
    --hostname bootstrap \
    --ssh-inject root:file:./.ssh/id_rsa.pub \
    --selinux-relabel

With the above command, you also set up the ssh key to perform passwordless login. 

Now,  you have a bootstrap server image for SAP HANA and can upload the image to Azure. On Azure WebUI, you will first need to create a storage account (e.g., storageaccounthana) and a container (images) within the storage account. Change the access level of the container as follows: 

And then, set up the Shared access signature: 

Note: the steps shown in the above two pictures are needed for the playbook deploy_bootstrap.yml mentioned in the next section. 

After that, generate a SAS and connection string by clicking the button “Generate SAS and connection string”:

Upload the image using the az storage blob upload command. It can take a hour or more to complete depending on your network bandwidth to Azure:

# sas='<SAS token>'
# az storage blob upload --account-name storageaccounthana \
     --container-name images --name bootstrap.vhd \
     -f /ssd_vms/bootstrap.vhd --sas-token $sas

Once the upload is complete, you can run the playbook deploy_bootstrap.yml to boot up the bootstrap server.

---
- hosts: localhost
  connection: local

  vars:
    resource_group: ResourceGroupHana
    storage_account: storageaccounthana
    storage_container: images
    vm_image: bootstrapImage
    image_name: bootstrap.vhd
    virtual_network: ResourceGroupHanaVnet
    virtual_subnet: ResourceGroupHanaSubnet
    virtual_machine: bootstrap
    public_ip: bootstrapPublicIP
    nic: bootstrapVMNic
    security_group: bootstrapSecgroup
    image_URI: "https://{{ storage_account }}.blob.core.windows.net/{{ storage_container }}/{{ image_name }}"

  tasks:
    - name: Create a resource group
      azure_rm_resourcegroup:
        name: "{{ resource_group }}"
        location: eastus

    - name: Create an Azure Image from the image uploaded
      azure_rm_image:
        resource_group: "{{ resource_group }}"
        name: "{{ vm_image }}"

        source: "{{ image_URI }}"
        os_type: Linux

    - name: Create a virtual network
      azure_rm_virtualnetwork:
        resource_group: "{{ resource_group }}"
        name: "{{ virtual_network }}"
        address_prefixes: "10.0.0.0/16"

    - name: Create a subnet
      azure_rm_subnet:
        resource_group: "{{ resource_group }}"
        name: "{{ virtual_subnet }}"
        address_prefix: "10.0.1.0/24"
        virtual_network: "{{ virtual_network }}"

    - name: Create a public ip
      azure_rm_publicipaddress:
        resource_group: "{{ resource_group }}"
        allocation_method: Static
        name: "{{ public_ip }}"

    - name: Create a security group that allows HTTP
      azure_rm_securitygroup:
        resource_group: "{{ resource_group }}"
        name: "{{ security_group }}"
        rules:
          - name: 'AllowSSH'
            protocol: Tcp
            destination_port_range: 22
            access: Allow
            priority: 101
            direction: Inbound

    - name: Create a network interface
      azure_rm_networkinterface:
        resource_group: "{{ resource_group }}"
        name: "{{ nic }}"
        virtual_network: "{{ virtual_network }}"
        subnet: "{{ virtual_subnet }}"
        security_group: "{{ security_group }}"
        ip_configurations:
          - name: ipconfig1
            public_ip_address_name: "{{ public_ip }}"
            primary: True

    - name: Create a bootstrap server
      azure_rm_virtualmachine:
        resource_group: "{{ resource_group }}"
        name: "{{ virtual_machine }}"
        vm_size: Standard_DS1_v2
        admin_username: adminUser
        admin_password: "adminUser#1"
        ssh_public_keys:
          - path: /home/adminUser/.ssh/authorized_keys
            key_data: <Insert publish ssh key>
        network_interfaces: "{{ nic }}"
        image:
          name: "{{ vm_image }}"
          resource_group: "{{ resource_group }}"
        managed_disk_type: Standard_LRS
        data_disks:
          - lun: 0
            disk_size_gb: 40
            managed_disk_type: Standard_LRS
      register: result

    - name: Show the information of the VM
      debug: 
        var=result
...

Deploy SAP HANA Scale-Up System Replication

Now let’s take a look at the playbooks to deploy SAP HANA Scale-Up System Replication. Yaml file site.yml is the main playbook, which consists of four playbooks respectively to set up Azure credentials (setup_azure_credentials.yml), and provision Azure VMs as SAP HANA nodes (provision_azure_vms.yml) with the credentials, and then set up the Azure VMs (configure_azure_vms.yml) before SAP HANA deployment, finally deploy SAP HANA Scale-Up System Replication (deploy_sap_hana_hsr.yml).

Next, you will need to run setup_azure_credentials.yml to set up the authentication with Azure for Azure Resource Manager modules:  

- name: Set up Azure credentials
  hosts: localhost
      command: >  connection: local
  tasks:
    - name: Include Azure credentials
      include_vars: files/azure_cred.yml

    - name: Log into Azure

                az login -u "{{ azure_account }}" -p "{{ azure_secret }}"

    - name: Create a service principal
      command: >
                az ad sp create-for-rbac --name ServicePrincipalHana

      register: ad_info
    - set_fact:
        ad_vars: "{{ ad_info.stdout | from_json }}"

    - name: Get Azure account information
      command: >
                az account show
      register: account_info
    - set_fact:
        account_vars: "{{ account_info.stdout | from_json }}"

    - name: Insert Azure credentials configuration in /root/.azure/credentials
      blockinfile:
        path: /root/.azure/credentials
        create: yes
        block: |
          [default]
          subscription_id={{ account_vars.id }}
          client_id={{ ad_vars.appId }}
          secret={{ ad_vars.password }}
          tenant={{ ad_vars.tenant }}

    - name: Verify the Azure credentials by listing resource groups
      azure_rm_resourcegroup_facts:
      register: output
    - debug: var=output

# cat << EOF > ~/.azure_vault
<vault password>
EOF
# ansible-vault encrypt_string --vault-id dev@~/.azure_vault '<string to encrypt>' --name 'azure_secret'

Now, let’s take a look at provision_azure_vms.yml. It takes care of Azure VM provisioning. The difference from the previous blog is that the role azure_rm_availabilityset is added to create Azure Availability sets, which provide VM redundancy and availability.

---
- hosts: localhost
  connection: local
  vars:
    resource_group: ResourceGroupHana
    virtual_network: ResourceGroupHanaVnet
    virtual_subnet: ResourceGroupHanaSubnet
    virtual_machine1: saphana01
    virtual_machine2: saphana02
    nic1: NicHana01
    nic2: NicHana02
    security_group: SecurityGroupHana
    location: eastus
    avaiability_set: AvailabilitySetHana

  tasks:
    - name: Ensure a security group that allows SSH
      azure_rm_securitygroup:
        resource_group: "{{ resource_group }}"
        name: "{{ security_group }}"
        rules:
          - name: 'AllowSSH'
            protocol: Tcp
            destination_port_range: 22
            access: Allow
            priority: 101
            direction: Inbound

    - name: Ensure an availability set with advanced options
      azure_rm_availabilityset:
        name: "{{ avaiability_set }}"
        location: "{{ location }}"
        resource_group: "{{ resource_group }}"
        platform_update_domain_count: 5
        platform_fault_domain_count: 3
        sku: Aligned

    - name: Ensure a network interface for VM saphana01 
      azure_rm_networkinterface:
        resource_group: "{{ resource_group }}"
        name: "{{ nic1 }}"
        virtual_network: "{{ virtual_network }}"
        subnet_name: "{{ virtual_subnet }}"
        security_group: "{{ security_group }}"
        ip_configurations:
          - name: ipconfig1
            primary: True

    - name: Ensure a VM saphana01 to run SAP HANA
      azure_rm_virtualmachine:
        resource_group: "{{ resource_group }}"
        name: "{{ virtual_machine1 }}"
        vm_size: Standard_E8s_v3
        admin_username: adminUser
        admin_password: "adminUser#1"
        availability_set: "{{ avaiability_set }}"
        ssh_public_keys:
          - path: /home/adminUser/.ssh/authorized_keys
            key_data: <Insert public ssh key of bootstrap server>
        network_interfaces: "{{ nic1 }}"
        image:
          offer: "RHEL"
          publisher: "RedHat"
          sku: "8.1"
          version: "8.1.2020082711"
        managed_disk_type: Standard_LRS
        data_disks:
            - lun: 0
              disk_size_gb: 128
              managed_disk_type: Premium_LRS

    - name: Ensure a network interface for VM saphana02
      azure_rm_networkinterface:
        resource_group: "{{ resource_group }}"
        name: "{{ nic2 }}"
        virtual_network: "{{ virtual_network }}"
        subnet_name: "{{ virtual_subnet }}"
        security_group: "{{ security_group }}"
        ip_configurations:
          - name: ipconfig2
            primary: True

    - name: Ensure a VM saphana02 to run SAP HANA
      azure_rm_virtualmachine:
        resource_group: "{{ resource_group }}"
        name: "{{ virtual_machine2 }}"
        vm_size: Standard_E8s_v3
        admin_username: adminUser
        admin_password: "adminUser#1"
        availability_set: "{{ avaiability_set }}"
        ssh_public_keys:
          - path: /home/adminUser/.ssh/authorized_keys
            key_data: <Insert public ssh key of bootstrap server>
        network_interfaces: "{{ nic2 }}"
        image:
          offer: "RHEL"
          publisher: "RedHat"
          sku: "8.1"
          version: "8.1.2020082711"
        managed_disk_type: Standard_LRS
        data_disks:
            - lun: 0
              disk_size_gb: 128
              managed_disk_type: Premium_LRS
...

Once Azure VMs are provisioned, playbook site.yml will trigger configure_azure_vms.yml to configure the VMs and the bootstrap server in the OS level.

---
- hosts: localhost
  connection: local
  tasks:
  - name: Get facts for saphana01's network interface
    azure_rm_networkinterface_facts:
      resource_group: ResourceGroupHana
      name: NicHana01
    register: result
  - set_fact: 
      saphana01_ipaddr: "{{ result.networkinterfaces.0.ip_configurations.0.private_ip_address }}"

  - name: Get facts for saphana02's network interface
    azure_rm_networkinterface_facts:
      resource_group: ResourceGroupHana
      name: NicHana02
    register: result
  - set_fact: 
      saphana02_ipaddr: "{{ result.networkinterfaces.0.ip_configurations.0.private_ip_address }}"

  - name: Setup /etc/hosts
    blockinfile: 
      path: /etc/hosts
      block: |
        {{ saphana01_ipaddr }} saphana01 saphana01.tamlab.com
        {{ saphana02_ipaddr }} saphana02 saphana02.tamlab.com

- hosts: hana
  tasks:
  - name: Ensure that copy local /etc/hosts to saphana01 and saphana02
    copy:
      src: /etc/hosts
      dest: /etc/hosts
      owner: root
      group: root
      mode: '0644'
      backup: yes

  - name: Disable RHUI repos by renaming repo file /etc/yum.repos.d/rh-cloud.repo
    command: mv /etc/yum.repos.d/rh-cloud.repo /etc/yum.repos.d/rh-cloud.repo.bk
    ignore_errors: yes

  - name: Extend root filesystem "/" of VMs to 10GB, the default to 2GB
    lvol:
      vg: rootvg
      lv: rootlv
      size: 10240
      resizefs: yes

  - name: Resize root filesystem "/"
    command: >
              xfs_growfs /
...

As the last piece, deploy_sap_hana_hsr.yml is used to prepare SAP HANA nodes, install SAP HANA, configure SAP HANA System Replication, and set up pacemaker:

---
- hosts: hana
  roles:
          - { role: redhat_sap.sap_rhsm }
          - { role: redhat_sap.sap_hana_mediacheck }
          - { role: redhat_sap.sap_hostagent }
          - { role: linux-system-roles.timesync }
          - { role: linux-system-roles.storage }
          - { role: sap-preconfigure }
          - { role: sap-hana-preconfigure }
          - { role: redhat_sap.sap_hana_deployment }
          - { role: redhat_sap.sap_hana_hsr }
          - { role: redhat_sap.sap_hana_ha_pacemaker }
...

To set up pacemaker, the following lines are added to specify the variables for SAP HANA HA cluster in group_vars/hana.yml:

# Variables required for 'sap_hana_ha_pacemaker' role
sap_hana_ha_pacemaker_hana_sid: RHT
sap_hana_ha_pacemaker_hana_instance_number: "01"
sap_hana_ha_pacemaker_vip: 10.0.1.100
sap_hana_ha_pacemaker_hacluster_password: "MyPassw0rd!"
sap_hana_ha_pacemaker_node1_fqdn: saphana01.tamlab.com
sap_hana_ha_pacemaker_node2_fqdn: saphana02.tamlab.com
sap_hana_ha_pacemaker_node1_ip: 10.0.1.6
sap_hana_ha_pacemaker_node2_ip: 10.0.1.7

Now, it is time to kick off the playbook:

# ansible-playbook --vault-password-file ~/.azure_vault site.yml

Note: While running the playbook make sure that your console will not be disconnected due to SSH connection timeout. Otherwise, you may need to clean up the VMs and start over again. 

Here are some screenshots while the playbook is running.

Once the playbook is complete, you will see the collected HA cluster information stored in the form of node attributes that can be viewed by running the command pcs status --full:

Congratulations! Your SAP HANA Scale-Up System Replication with a High Availability cluster is up and running. Now, you can check out the Red Hat Insights application-focuses view for SAP HANA deployments, which provides automatic discovery, health and security assessment for SAP HANA on RHEL. 

Summary

As you can see System Roles for SAP and Ansible modules can speed up your SAP HANA deployment along with Image Builder. These help to manage complex processes in the automated way, which is a perfect solution to use in infrastructure workflows. You can easily integrate them with other building tools for continuous integration and deployment of systems to work efficiently and enhance your productivity.