订阅内容

Introduction

Often in the early stages of implementing OpenShift with our customers, we find they don't have high quality distributed storage available to OpenShift. It isn't because there isn't distributed storage in the organization, most of the time it's just that the storage team has not been involved (yet) in the OpenShift initiative.

The OpenShift Ansible installer takes into account this scenario and allows you to create an NFS server as part of the installation. This option creates a non-HA NFS server which is good enough for non-production environments and provides distributed file storage.

The registry can use this storage, but logging and metrics (and potentially some customer's applications) require block storage.

When reasoning on this problem, a colleague of mine (Clark Hale) and myself noticed that it would be relatively straightforward to create an iSCSI server using target.

This would enable a non-HA block storage solution with a similar purpose as the NFS server installed by the Ansible Playbook to be used in non-production environment until an HA storage solution is implemented.

We also noticed that the target exposes a dynamic provisioning API via the targetd RPM package. So we built an out-of-tree iscsi/targetd dynamic provisioner.

We merged our initial project with the external-storage section of the kubernetes-incubator repo. The remainder of the article explains how to setup the iscsi server and configure dynamic provisioning.

For more details on the below steps, refer to the project readme section on the Ansible Playbooks.

Creating an iSCSI server

Clone the repo:

git clone https://github.com/kubernetes-incubator/external-storage
cd external-storage/iscsi/targetd

Update your inventory file to look as follows:

[OSEv3:children]
masters
nodes
etcd
targetd

[OSEv3:vars]
targetd_lvm_volume_group=vg-targetd
targetd_lvm_physical_volume=/dev/vdb
targetd_password=ciao
targetd_user=admin
targetd_iscsi_target=iqn.2003-01.org.example.mach1:1234
iscsi_provisioner_pullspec=raffaelespazzoli/iscsi-controller:0.0.1
iscsi_provisioner_default_storage_class=true

[targetd]
targetd.cscc

[nodes]
ose-master1.cscc openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_schedulable=true
iscsi_initiator_name=iqn.2003-03.net.deadvax:ose-master1
ose-master2.cscc openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_schedulable=true
iscsi_initiator_name=iqn.2003-03.net.deadvax:ose-master2
ose-master3.cscc openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_schedulable=true
iscsi_initiator_name=iqn.2003-03.net.deadvax:ose-master3
ose-node1.cscc openshift_node_labels="{'region': 'primary', 'zone': 'default'}"
iscsi_initiator_name=iqn.2003-03.net.deadvax:ose-node1
ose-node2.cscc openshift_node_labels="{'region': 'primary', 'zone': 'default'}"
iscsi_initiator_name=iqn.2003-03.net.deadvax:ose-node2

In bold you can see the necessary changes.

To initialize your iSCSI server, run the following:

ansible-playbook -i <your inventory file> ansible/provisioner-playbook.yaml

This will setup the iSCSI server (target and targetd). It assumes that there is a block device available to hold the storage (/dev/vdb in the example).

Configuring OpenShift to Use the iSCSI Server

Nodes needs to be initialized in order to use iSCSI, the next Playbook does that:

ansible-playbook -i <your inventory file> ansible/initiator-playbook.yaml

Notice that this Playbook is very generic and could be used also with non-targetd iSCSI implementations.

Configuring Dynamic Provisioning

The last Playbook is optional and needed only if you want to set up dynamic provisioning:

ansible-playbook -i <your inventory file> ansible/provisioner-playbook.yaml

Pointing Logging and Metrics to an iSCSI Volume

As we said in the introduction, OpenShift logging and metrics require block storage.

Assuming you have installed logging and metrics with the OpenShift Ansible Playbook, the following instructions will refactor them to use iSCSI (notice that by running them you will lose the current content of the Cassandra and Elasticsearch database).

For logging execute the following:

oc set volume dc/<name-of-your-elasticsearch-dc> --add --name=elasticsearch-storage -t pvc --claim-size=10G --claim-name=elasticsearch-storage --claim-mode='ReadWriteOnce' --overwrite

For metrics execute the following:

oc scale --replicas=0 rc hawkular-cassandra-1
oc set volume rc/hawkular-cassandra-1 --add --name=cassandra-data -t pvc --claim-size=10G --claim-name=cassandra-data --claim-mode='ReadWriteOnce' --overwrite
oc scale --replicas=1 rc hawkular-cassandra-1

Conclusions

The Ansible Playbook and the dynamic provisioner for iSCSI target are meant to facilitate the implementation and configuration of iSCSI block storage when a more enterprise alternative is not available. Normally the iSCSI target is not configured to be HA (and I doubt it can be configured to be so) therefore I don’t recommend using it in production environments.


关于作者

Raffaele is a full-stack enterprise architect with 20+ years of experience. Raffaele started his career in Italy as a Java Architect then gradually moved to Integration Architect and then Enterprise Architect. Later he moved to the United States to eventually become an OpenShift Architect for Red Hat consulting services, acquiring, in the process, knowledge of the infrastructure side of IT.

Currently Raffaele covers a consulting position of cross-portfolio application architect with a focus on OpenShift. Most of his career Raffaele worked with large financial institutions allowing him to acquire an understanding of enterprise processes and security and compliance requirements of large enterprise customers.

Raffaele has become part of the CNCF TAG Storage and contributed to the Cloud Native Disaster Recovery whitepaper.

Recently Raffaele has been focusing on how to improve the developer experience by implementing internal development platforms (IDP).

Read full bio
UI_Icon-Red_Hat-Close-A-Black-RGB

按频道浏览

automation icon

自动化

有关技术、团队和环境 IT 自动化的最新信息

AI icon

人工智能

平台更新使客户可以在任何地方运行人工智能工作负载

open hybrid cloud icon

开放混合云

了解我们如何利用混合云构建更灵活的未来

security icon

安全防护

有关我们如何跨环境和技术减少风险的最新信息

edge icon

边缘计算

简化边缘运维的平台更新

Infrastructure icon

基础架构

全球领先企业 Linux 平台的最新动态

application development icon

应用领域

我们针对最严峻的应用挑战的解决方案

Original series icon

原创节目

关于企业技术领域的创客和领导者们有趣的故事