Scaling the future: How Garanti BBVA manages etcd in massive Red Hat OpenShift environments

At the OpenShift Commons Gathering in Amsterdam on March 23—a Day Zero event for KubeCon + CloudNativeCon Europe 2026—attendees got a deep look into the engine room of 1 of Turkey's largest private banks. Emirhan Bilge Bulut, an Expert System Engineer at Garanti BBVA, joined Gokhan Goksu, a Senior Solution Architect at Red Hat, to discuss the unglamorous but essential infrastructure management required to support 30 million customers and 1.2 billion daily transactions.

Figure 1. Emirhan Bilge Bulut, an Expert System Engineer at Garanti BBVA, joined Gokhan Goksu, a Solution Architect at Red Hat, speaking at the OpenShift Commons Gathering in Amsterdam.

The scale of the challenge

Garanti BBVA operates at a staggering magnitude. Its infrastructure includes:

• 60 Red Hat OpenShift clusters, with 33 serving production environments.
• Over 30,000 pods and 3,100 services across all clusters.
• Up to 2 billion transactions processed per day during peak times.

As organizations scale, they often find that standard tools aren't enough to manage the corresponding growth of their etcd database. In many infrastructures, this database is the single critical component that holds all cluster configuration, secrets, and metadata.

The problem: When etcd hits the wall

For Garanti BBVA, the challenge manifested as uncontrolled etcd size growth. In its non-production environments, which are often more complex due to high developer activity, a single cluster might house 40,000 pods and 10,000 microservices.

Goksu explained that any performance degradation in etcd leads to high API latency. Because nearly every oc or kubectl command interacts with the database, latency creates a systemwide backlog in reconciliation loops and can bottleneck pod scheduling.

The bank initially tried standard optimizations provided by the etcd cluster operator, such as automated defragmentation and history compaction. While these helped in the short term, database growth continued, signaling a need to address root causes rather than symptoms.

Deep dive into optimization

The team identified several factors contributing to the bloat:

• Unrestricted revision history: Deployment objects lacked revision limits, causing historical data to pile up. The team updated configurations to keep only 1 previous revision.
• Secret proliferation: The team discovered over 20,000 unnecessary secrets in their etcd database. By taking advantage of capabilities introduced in Red Hat OpenShift 4.11, they deleted legacy service account tokens no longer required for storage in secrets.
• Duplicated ConfigMaps: CI/CD pipelines were creating redundant configurations across namespaces. The team consolidated these into common ConfigMaps, significantly reducing the data footprint.

Designing a custom solution

When existing open source tools like K8sPurger proved too resource-intensive or limited in their ability to scan custom resource definitions (CRDs), Garanti BBVA decided to build its own solution.

This tool was designed around the principles of low resource consumption and safe deletion. It interacts directly with the Red Hat OpenShift REST API rather than using standard Python libraries, which reduces the processing time necessary to perform its tasks from 30 minutes to just 4 minutes.

The tool also includes a unique estimation mechanism: It decodes etcd data to calculate exactly how much space will be reclaimed before any deletion occurs.

Real-world results

By implementing these strategies and a repeatable, automated process, Garanti BBVA achieved a number of positive outcomes:

• Reclaiming 1.5 to 2 GB of free space in non-production environments.
• Sustainable management: What was once a 1-time fix is now a fully automated, repeatable safety process.

Garanti BBVA was also able to reduce complexity and improve overall system health by separating QA and development environments into distinct clusters. All this was achieved without the need for any major hardware or network upgrades. As Emirhan Bilge Bulut noted, "In large systems, problems are not always solved by adding more resources. They are solved by understanding the system deeply."

Resources

• Missed the live session in Amsterdam? You can watch the full presentation, "How Garanti BBVA solves size challenges in large OpenShift environments," on-demand now. See the technical deep-dive and the live Q&A with OpenShift Commons attendees.
• Optimize your cluster performance. Explore our official guide to achieving reliable etcd performance and scalability.