Chaos engineering in practice: Lessons from enterprise customers

Working with enterprise customers on chaos engineering has reinforced 1 lesson: Resilience is a business requirement. Failed transactions interrupt revenue, infrastructure outages affect multiple workloads, and database failures put critical data at risk. Most importantly, repeated failures erode customer trust—damage that can outlast the outage itself.

Teams turn these business needs into measurable targets through service-level objectives (SLOs), such as availability and transaction latency. Service-level agreements (SLAs) define commitments to customers and the consequences of missing them. Chaos engineering introduces controlled failures to test behavior against SLOs and uncover weaknesses that could put those commitments at risk.

4 use cases. 1 business goal.

Across the enterprise environments we have worked with, several use cases consistently stand out. These anonymized stories show how teams use Krkn to uncover weaknesses, improve recovery, and strengthen service reliability. Krkn Operator helps turn those lessons into reusable workflows to validate resilience before releases and configuration changes. Outcomes reflect the conditions tested.

Table 1: Chaos engineering as applied to 4 enterprise use cases. 

1. Production readiness: Test failure before global trading platform go-live

Before application launch and go-live in production, customers need to understand how the service behaves under load and how it responds when dependencies fail.

For a latency-sensitive trading application on Red Hat OpenShift Service on AWS (ROSA), we collaborated with the customer and tested availability-zone outages, compute-node failures, Amazon Aurora database outages, application programming interface (API) throttling, and storage exhaustion. The experiments exposed reliability and performance bottlenecks earlier testing missed, revealed how dependency outages affected transactions, and guided recovery improvements. Validating and improving workload self-healing under stress gave the team stronger evidence for the launch decision.

Make recovery evidence part of the launch decision. Add a few high-impact failures to release-readiness testing, focusing on dependencies that could interrupt a critical customer journey. Use SLOs aligned with customer SLA commitments and clear recovery limits to decide which gaps must be fixed before launch.

2. Virtualization: Make resilience part of migration

A major institution migrating enterprise virtualized workloads to Red Hat OpenShift Virtualization needed proof that the platform was rock-solid and resilient enough for their most critical applications.

We collaborated with the customer engineering team and tested simultaneous control-plane and worker-node losses, storage disruption, and Domain Name System (DNS) outages while observing Windows and Linux workloads. The findings led to VirtIO driver fixes, Red Hat OpenShift Data Foundation tuning, input/output (I/O) limits, fencing improvements, and better alerts. After hardening, the team reported storage recovery within seconds and no virtual machine (VM) crashes in the validated scenarios.

Validate recovery from the application’s perspective. Test under representative load, traffic patterns, and network topology. Verify application availability, data integrity, and recovery time alongside VM and host health because 1 infrastructure failure can disrupt many business services.

3. Stateful applications: Protect data during recovery

Database recovery must preserve the data and transactions the business depends on.

A team validating critical relational databases on OpenShift Virtualization tested full disks, network partitions that could create split-brain conditions, and abrupt loss of active compute nodes. The findings helped platform engineers and database architects tune storage classes and input/output operations (IOPS) limits for crash recovery. The team reported preserved data and service continuity in the validated scenarios.

Make data integrity a recovery requirement. Start with recovery-time and data-loss objectives, then choose failures that could violate them. Verify acknowledged transactions, replication health, consistency, and application recovery. Storage tuning should be judged by its effect on recovery as well as normal performance.

4. Payment systems: Follow the customer transaction

A payment service can appear healthy while packet loss, slow dependencies, or resource pressure prevent customers from completing transactions.

In 1 financial-services environment, teams used a hub-and-spoke model and GitOps workflows to coordinate chaos experiments for critical payment systems. Network packet loss and central processing unit (CPU) exhaustion tested microservice failover, while regular GameDays exercised troubleshooting and incident response. The team reported improved recovery times through repeated drills and joint root-cause analysis.

Follow the transaction through to completion. Trace a request through the application, its dependencies, the database, and the final confirmation. Evaluate transaction success and latency against SLOs, and check retries, duplicate processing, and recovery time. These measures reveal customer impact that infrastructure health alone can miss.

Include partial failures, such as intermittent packet loss and slow responses. Watch for growing queues, excessive retries, and failures spreading between services. These tests reveal how degradation affects customers before a complete outage occurs.

What these customer experiences teach us

Across these engagements, 3 lessons stand out:

Measure what matters to customers. For trading and payment platforms, success means completing transactions within SLOs. For virtualization, it means keeping applications available when shared infrastructure fails. For databases, it means recovering within agreed limits while preserving data integrity. Use these service outcomes to judge resilience and identify risks to customer SLA commitments.

Validate the team’s response alongside system recovery. The payment GameDays showed the value of testing detection, diagnosis, and incident response together with automated failover. Give each exercise a recovery owner, measure response times, and use the findings to improve alerts and runbooks. Repeat the exercise to verify those improvements.

Save and replay workflows to track resilience with every change. Turn validated experiments into reusable workflows with Krkn Operator and Chaos Studio. Replay them for every release and configuration change, comparing resilience scores against the baseline to catch regressions before rollout. Keep workloads and SLOs consistent, and require transaction, data-integrity, and recovery checks to pass alongside the score.

Put the lessons to work

Start with 1 question: What failure could hurt our customers or business the most?

Choose a critical service, test that risk under representative conditions, and fix what the experiment reveals. Preserve the test so future changes can be checked against the same expectations. This turns customer experience into a repeatable engineering practice that protects availability, revenue, and most importantly customer trust.

Embrace chaos together

Join the Krkn community to discuss your use cases, share what you learn, and collaborate on building more resilient systems.