订阅我们的博客

Introduction

In a prior blog post, we illustrated some best practices on which metrics to use when monitoring applications. We used the same metrics to construct dashboards and predict when something might break. We also discussed how, in certain cases, is it possible to preemptively intervene and prevent the failure from occurring (preventive maintenance).

It is a responsibility of the SRE team to automate, as much as possible, these preventive maintenance operations. Typically, the metric that best lends itself to be used for this purpose is saturation (of a given resource).

When it comes to storage, saturation can be calculated as used volume space versus the total size of the volume in question.

Preventive maintenance, in this case, would be to increase the size of the volume when the saturation reaches a certain threshold.

If using persistent storage, it is recommended to create a Prometheus alert like the following:

    - alert: "Storage Saturation"
     labels:
       severity: critical
     annotations:
       summary: Storage usage over 75%
       persistentvolumeclaim: {{` "{{ $labels.persistentvolumeclaim }}" `}}
       namespace: {{` "{{ $labels.namespace }}" `}}
     expr: kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes > 0.75
     for: 1m

This alert will notify an SRE when any volume crosses the 75% utilization threshold (this threshold is, of course, adjustable to your needs). When notified, an SRE has the chance to expand the volume.

Expanding the size of persistent storage volumes is a relatively new feature in Kubernetes that must be supported by all CSI storage drivers and is also supported by some in-tree storage drivers.

By combining the ability to monitor persistent volumes and the availability to expand them, we can build preventive maintenance automation that will expand volumes when they are reaching capacity. For the remainder of this article, an operator-based approach will be introduced to automate this use case of preventive maintenance of volumes.

The volume-expander-operator

The volume-expander-operator is an operator that monitors volumes, and when the saturation of a volume crosses a given threshold, it expands that volume. Expanding volumes, in some situations, requires that attached pods also be restarted. This functionality is also integrated into the operator.

For a volume to be monitored by the volume-expander-operator, one has to add the following annotation to the PersistentVolumeClaim:

volume-expander-operator.redhat-cop.io/autoexpand: “true”.

Once enabled, the volume-expander-operator will start polling the platform Prometheus instance that is part of OpenShift Monitoring and calculating the ratio between these two metrics: kubelet_volume_stats_used_bytes and kubelet_volume_stats_capacity_bytes.

This ratio represents the volume saturation, and the following annotations placed on the PersistentVolumeClaim determine when the volume should be expanded:

 

Annotation

Description

Default Value

volume-expander-operator.redhat-cop.io/expand-threshold-percent

Saturation point that will trigger the expansion of the volume

80

volume-expander-operator.redhat-cop.io/expand-by-percent

Percentage based on the size of the existing volume it will be expanded

25

volume-expander-operator.redhat-cop.io/expand-up-to

Upper bound of the volume

 

volume-expander-operator.redhat-cop.io/polling-frequency

Frequency by which the operator will query Prometheus for metrics

30s

 

When the saturation crosses the threshold, the volume will be expanded until it reaches the upper bound value.

The following example of is a PVC configured for auto expansion:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
 annotations:
   volume-expander-operator.redhat-cop.io/autoexpand: 'true'
   volume-expander-operator.redhat-cop.io/expand-threshold-percent: "85"
   volume-expander-operator.redhat-cop.io/expand-by-percent: "20"
   volume-expander-operator.redhat-cop.io/polling-frequency: "1m"
   volume-expander-operator.redhat-cop.io/expand-up-to: "20Gi"
 name: to-be-expanded
spec:
 accessModes:
   - ReadWriteOnce
 resources:
   requests:
     storage: "1Gi"

Installation

The volume-expander-operator can be installed via OperatorHub (look for the icon shown below) or via a Helm Chart.

 

Instructions using either of the supported installation methods can be found here.

Conclusion

The purpose of the volume-expander-operator is to relieve human operators from the toil of monitoring volumes and expanding them when needed. It also provides some protection against run-away processes that would indefinitely increase the consumption of storage space. However, it is always recommended to set a quota on the relative storage class to prevent this situation from occurring.

It is our hope that this operator will increase the adoption rate of stateful workloads on OpenShift.


关于作者

Raffaele is a full-stack enterprise architect with 20+ years of experience. Raffaele started his career in Italy as a Java Architect then gradually moved to Integration Architect and then Enterprise Architect. Later he moved to the United States to eventually become an OpenShift Architect for Red Hat consulting services, acquiring, in the process, knowledge of the infrastructure side of IT.

Currently Raffaele covers a consulting position of cross-portfolio application architect with a focus on OpenShift. Most of his career Raffaele worked with large financial institutions allowing him to acquire an understanding of enterprise processes and security and compliance requirements of large enterprise customers.

Raffaele has become part of the CNCF TAG Storage and contributed to the Cloud Native Disaster Recovery whitepaper.

Recently Raffaele has been focusing on how to improve the developer experience by implementing internal development platforms (IDP).

Read full bio

按频道浏览

automation icon

自动化

有关技术、团队和环境 IT 自动化的最新信息

AI icon

人工智能

平台更新使客户可以在任何地方运行人工智能工作负载

open hybrid cloud icon

开放混合云

了解我们如何利用混合云构建更灵活的未来

security icon

安全防护

有关我们如何跨环境和技术减少风险的最新信息

edge icon

边缘计算

简化边缘运维的平台更新

Infrastructure icon

基础架构

全球领先企业 Linux 平台的最新动态

application development icon

应用领域

我们针对最严峻的应用挑战的解决方案

Original series icon

原创节目

关于企业技术领域的创客和领导者们有趣的故事