Dynamic iSCSI Provisioning with Target

Introduction

Often in the early stages of implementing OpenShift with our customers, we find they don't have high quality distributed storage available to OpenShift. It isn't because there isn't distributed storage in the organization, most of the time it's just that the storage team has not been involved (yet) in the OpenShift initiative.

The OpenShift Ansible installer takes into account this scenario and allows you to create an NFS server as part of the installation. This option creates a non-HA NFS server which is good enough for non-production environments and provides distributed file storage.

The registry can use this storage, but logging and metrics (and potentially some customer's applications) require block storage.

When reasoning on this problem, a colleague of mine (Clark Hale) and myself noticed that it would be relatively straightforward to create an iSCSI server using target.

This would enable a non-HA block storage solution with a similar purpose as the NFS server installed by the Ansible Playbook to be used in non-production environment until an HA storage solution is implemented.

We also noticed that the target exposes a dynamic provisioning API via the targetd RPM package. So we built an out-of-tree iscsi/targetd dynamic provisioner.

We merged our initial project with the external-storage section of the kubernetes-incubator repo. The remainder of the article explains how to setup the iscsi server and configure dynamic provisioning.

For more details on the below steps, refer to the project readme section on the Ansible Playbooks.

Creating an iSCSI server

Clone the repo:

git clone https://github.com/kubernetes-incubator/external-storage

cd external-storage/iscsi/targetd

Update your inventory file to look as follows:

[OSEv3:children]

masters

nodes

etcd

targetd
[OSEv3:vars]

targetd_lvm_volume_group=vg-targetd
 targetd_lvm_physical_volume=/dev/vdb
 targetd_password=ciao
 targetd_user=admin
 targetd_iscsi_target=iqn.2003-01.org.example.mach1:1234
 iscsi_provisioner_pullspec=raffaelespazzoli/iscsi-controller:0.0.1
 iscsi_provisioner_default_storage_class=true
[targetd]
 targetd.cscc
[nodes]

ose-master1.cscc openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_schedulable=true

iscsi_initiator_name=iqn.2003-03.net.deadvax:ose-master1

ose-master2.cscc openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_schedulable=true

iscsi_initiator_name=iqn.2003-03.net.deadvax:ose-master2

ose-master3.cscc openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_schedulable=true

iscsi_initiator_name=iqn.2003-03.net.deadvax:ose-master3

ose-node1.cscc openshift_node_labels="{'region': 'primary', 'zone': 'default'}"

iscsi_initiator_name=iqn.2003-03.net.deadvax:ose-node1

ose-node2.cscc openshift_node_labels="{'region': 'primary', 'zone': 'default'}"

iscsi_initiator_name=iqn.2003-03.net.deadvax:ose-node2

In bold you can see the necessary changes.

To initialize your iSCSI server, run the following:

ansible-playbook -i <your inventory file> ansible/provisioner-playbook.yaml

This will setup the iSCSI server (target and targetd). It assumes that there is a block device available to hold the storage (/dev/vdb in the example).

Configuring OpenShift to Use the iSCSI Server

Nodes needs to be initialized in order to use iSCSI, the next Playbook does that:

ansible-playbook -i <your inventory file> ansible/initiator-playbook.yaml

Notice that this Playbook is very generic and could be used also with non-targetd iSCSI implementations.

Configuring Dynamic Provisioning

The last Playbook is optional and needed only if you want to set up dynamic provisioning:

ansible-playbook -i <your inventory file> ansible/provisioner-playbook.yaml

Pointing Logging and Metrics to an iSCSI Volume

As we said in the introduction, OpenShift logging and metrics require block storage.

Assuming you have installed logging and metrics with the OpenShift Ansible Playbook, the following instructions will refactor them to use iSCSI (notice that by running them you will lose the current content of the Cassandra and Elasticsearch database).

For logging execute the following:

oc set volume dc/<name-of-your-elasticsearch-dc> --add --name=elasticsearch-storage -t pvc --claim-size=10G --claim-name=elasticsearch-storage --claim-mode='ReadWriteOnce' --overwrite

For metrics execute the following:

oc scale --replicas=0 rc hawkular-cassandra-1

oc set volume rc/hawkular-cassandra-1 --add --name=cassandra-data -t pvc --claim-size=10G --claim-name=cassandra-data --claim-mode='ReadWriteOnce' --overwrite

oc scale --replicas=1 rc hawkular-cassandra-1

Conclusions

The Ansible Playbook and the dynamic provisioner for iSCSI target are meant to facilitate the implementation and configuration of iSCSI block storage when a more enterprise alternative is not available. Normally the iSCSI target is not configured to be HA (and I doubt it can be configured to be so) therefore I don’t recommend using it in production environments.