Introduction

In a prior blog post, we illustrated some best practices on which metrics to use when monitoring applications. We used the same metrics to construct dashboards and predict when something might break. We also discussed how, in certain cases, is it possible to preemptively intervene and prevent the failure from occurring (preventive maintenance).

It is a responsibility of the SRE team to automate, as much as possible, these preventive maintenance operations. Typically, the metric that best lends itself to be used for this purpose is saturation (of a given resource).

When it comes to storage, saturation can be calculated as used volume space versus the total size of the volume in question.

Preventive maintenance, in this case, would be to increase the size of the volume when the saturation reaches a certain threshold.

If using persistent storage, it is recommended to create a Prometheus alert like the following:

    - alert: "Storage Saturation"
      labels:
        severity: critical
      annotations:
        summary: Storage usage over 75%
        persistentvolumeclaim: {{` "{{ $labels.persistentvolumeclaim }}" `}}
        namespace: {{` "{{ $labels.namespace }}" `}}
      expr: kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes > 0.75
      for: 1m

This alert will notify an SRE when any volume crosses the 75% utilization threshold (this threshold is, of course, adjustable to your needs). When notified, an SRE has the chance to expand the volume.

Expanding the size of persistent storage volumes is a relatively new feature in Kubernetes that must be supported by all CSI storage drivers and is also supported by some in-tree storage drivers.

By combining the ability to monitor persistent volumes and the availability to expand them, we can build preventive maintenance automation that will expand volumes when they are reaching capacity. For the remainder of this article, an operator-based approach will be introduced to automate this use case of preventive maintenance of volumes.

The volume-expander-operator

The volume-expander-operator is an operator that monitors volumes, and when the saturation of a volume crosses a given threshold, it expands that volume. Expanding volumes, in some situations, requires that attached pods also be restarted. This functionality is also integrated into the operator.

For a volume to be monitored by the volume-expander-operator, one has to add the following annotation to the PersistentVolumeClaim:

volume-expander-operator.redhat-cop.io/autoexpand: “true”.

Once enabled, the volume-expander-operator will start polling the platform Prometheus instance that is part of OpenShift Monitoring and calculating the ratio between these two metrics: kubelet_volume_stats_used_bytes and kubelet_volume_stats_capacity_bytes.

This ratio represents the volume saturation, and the following annotations placed on the PersistentVolumeClaim determine when the volume should be expanded:

When the saturation crosses the threshold, the volume will be expanded until it reaches the upper bound value.

The following example of is a PVC configured for auto expansion:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  annotations:
    volume-expander-operator.redhat-cop.io/autoexpand: 'true'
    volume-expander-operator.redhat-cop.io/expand-threshold-percent: "85"
    volume-expander-operator.redhat-cop.io/expand-by-percent: "20"
    volume-expander-operator.redhat-cop.io/polling-frequency: "1m"
    volume-expander-operator.redhat-cop.io/expand-up-to: "20Gi"
  name: to-be-expanded
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: "1Gi"

Installation

The volume-expander-operator can be installed via OperatorHub (look for the icon shown below) or via a Helm Chart.

Instructions using either of the supported installation methods can be found here.

Conclusion

The purpose of the volume-expander-operator is to relieve human operators from the toil of monitoring volumes and expanding them when needed. It also provides some protection against run-away processes that would indefinitely increase the consumption of storage space. However, it is always recommended to set a quota on the relative storage class to prevent this situation from occurring.

It is our hope that this operator will increase the adoption rate of stateful workloads on OpenShift.