Introduction
Often in the early stages of implementing OpenShift with our customers, we find they don't have high quality distributed storage available to OpenShift. It isn't because there isn't distributed storage in the organization, most of the time it's just that the storage team has not been involved (yet) in the OpenShift initiative.
The OpenShift Ansible installer takes into account this scenario and allows you to create an NFS server as part of the installation. This option creates a non-HA NFS server which is good enough for non-production environments and provides distributed file storage.
The registry can use this storage, but logging and metrics (and potentially some customer's applications) require block storage.
When reasoning on this problem, a colleague of mine (Clark Hale) and myself noticed that it would be relatively straightforward to create an iSCSI server using target.
This would enable a non-HA block storage solution with a similar purpose as the NFS server installed by the Ansible Playbook to be used in non-production environment until an HA storage solution is implemented.
We also noticed that the target exposes a dynamic provisioning API via the targetd RPM package. So we built an out-of-tree iscsi/targetd dynamic provisioner.
We merged our initial project with the external-storage section of the kubernetes-incubator repo. The remainder of the article explains how to setup the iscsi server and configure dynamic provisioning.
For more details on the below steps, refer to the project readme section on the Ansible Playbooks.
Creating an iSCSI server
Clone the repo:
git clone https://github.com/kubernetes-incubator/external-storage
cd external-storage/iscsi/targetd
Update your inventory file to look as follows:
[OSEv3:children]
masters
nodes
etcd
targetd[OSEv3:vars]
targetd_lvm_volume_group=vg-targetd
targetd_lvm_physical_volume=/dev/vdb
targetd_password=ciao
targetd_user=admin
targetd_iscsi_target=iqn.2003-01.org.example.mach1:1234
iscsi_provisioner_pullspec=raffaelespazzoli/iscsi-controller:0.0.1
iscsi_provisioner_default_storage_class=true[targetd]
targetd.cscc[nodes]
ose-master1.cscc openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_schedulable=true
iscsi_initiator_name=iqn.2003-03.net.deadvax:ose-master1
ose-master2.cscc openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_schedulable=true
iscsi_initiator_name=iqn.2003-03.net.deadvax:ose-master2
ose-master3.cscc openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_schedulable=true
iscsi_initiator_name=iqn.2003-03.net.deadvax:ose-master3
ose-node1.cscc openshift_node_labels="{'region': 'primary', 'zone': 'default'}"
iscsi_initiator_name=iqn.2003-03.net.deadvax:ose-node1
ose-node2.cscc openshift_node_labels="{'region': 'primary', 'zone': 'default'}"
iscsi_initiator_name=iqn.2003-03.net.deadvax:ose-node2
In bold you can see the necessary changes.
To initialize your iSCSI server, run the following:
ansible-playbook -i <your inventory file> ansible/provisioner-playbook.yaml
This will setup the iSCSI server (target and targetd). It assumes that there is a block device available to hold the storage (/dev/vdb
in the example).
Configuring OpenShift to Use the iSCSI Server
Nodes needs to be initialized in order to use iSCSI, the next Playbook does that:
ansible-playbook -i <your inventory file> ansible/initiator-playbook.yaml
Notice that this Playbook is very generic and could be used also with non-targetd iSCSI implementations.
Configuring Dynamic Provisioning
The last Playbook is optional and needed only if you want to set up dynamic provisioning:
ansible-playbook -i <your inventory file> ansible/provisioner-playbook.yaml
Pointing Logging and Metrics to an iSCSI Volume
As we said in the introduction, OpenShift logging and metrics require block storage.
Assuming you have installed logging and metrics with the OpenShift Ansible Playbook, the following instructions will refactor them to use iSCSI (notice that by running them you will lose the current content of the Cassandra and Elasticsearch database).
For logging execute the following:
oc set volume dc/<name-of-your-elasticsearch-dc> --add --name=elasticsearch-storage -t pvc --claim-size=10G --claim-name=elasticsearch-storage --claim-mode='ReadWriteOnce' --overwrite
For metrics execute the following:
oc scale --replicas=0 rc hawkular-cassandra-1
oc set volume rc/hawkular-cassandra-1 --add --name=cassandra-data -t pvc --claim-size=10G --claim-name=cassandra-data --claim-mode='ReadWriteOnce' --overwrite
oc scale --replicas=1 rc hawkular-cassandra-1
Conclusions
The Ansible Playbook and the dynamic provisioner for iSCSI target are meant to facilitate the implementation and configuration of iSCSI block storage when a more enterprise alternative is not available. Normally the iSCSI target is not configured to be HA (and I doubt it can be configured to be so) therefore I don’t recommend using it in production environments.
关于作者
Raffaele is a full-stack enterprise architect with 20+ years of experience. Raffaele started his career in Italy as a Java Architect then gradually moved to Integration Architect and then Enterprise Architect. Later he moved to the United States to eventually become an OpenShift Architect for Red Hat consulting services, acquiring, in the process, knowledge of the infrastructure side of IT.
Currently Raffaele covers a consulting position of cross-portfolio application architect with a focus on OpenShift. Most of his career Raffaele worked with large financial institutions allowing him to acquire an understanding of enterprise processes and security and compliance requirements of large enterprise customers.
Raffaele has become part of the CNCF TAG Storage and contributed to the Cloud Native Disaster Recovery whitepaper.
Recently Raffaele has been focusing on how to improve the developer experience by implementing internal development platforms (IDP).
产品
工具
试用购买与出售
沟通
关于红帽
我们是世界领先的企业开源解决方案供应商,提供包括 Linux、云、容器和 Kubernetes。我们致力于提供经过安全强化的解决方案,从核心数据中心到网络边缘,让企业能够更轻松地跨平台和环境运营。