How Discover cut $1.4 million from its annual AWS budget in two game days

Do you remember shared cell phone plans? At the turn of the century, a whole family might share the same pool of mobile phone minutes, and it only took a few hours of idle chatter on the phone to drain the account of all remaining minutes of talk time for everyone. Well, a similar phenomenon is currently happening in the cloud today. 

Anyone who's ever touched a public cloud has likely encountered an accidental overage payment. That's a problem, but it's even more problematic that it's deceptively easy to overlook a single instance of this when you're dealing in hundreds at a time. At Discover, the financial services company recently acquired by Capital One, developers found a way to solve their overage problems—and implement serious cost savings. It’s not surprising that a company offering credit cards would understand the value of cost optimization in the cloud. What is surprising, however, is that in those two days, they managed to cut their cloud bill by $1.4 million each year.

2 Days, $1.4 million

Discover moved onto Red Hat OpenShift in 2022. Prior to that, the company had been attempting to solidify on CloudFoundry, the Platform-as-a-Service often promoted by Pivotal, VMware, and Broadcom.

Craig Katz, OpenShift Director at Discover, said that Discover now uses OpenShift to manage its container infrastructure inside of Amazon Web Services. In order to migrate their existing 1,200 workloads to OpenShift and Amazon Web Services, the decision was made early on to focus on the speed and quality of the transition, and to push cost concerns to the side.

As Katz put it, "You can choose two: fast, cheap, or good. We chose fast and good."

After the migration was complete, the Discover team built out and deployed an additional 300 green field applications on the platform, and that's when Katz said it was time to think about cost optimization. With approximately 1,500 applications running in AWS, Katz said that there were definitely some holistic cost savings efforts available. The trick was undertaking those optimizations across the organization's portfolio of applications.

In January, a week before the Super Bowl, Katz and his Chicago-based team held what they called "Game Day."

"It was a two-day event, and we were asking people to analyze, fix, and deploy. At the end of two days, based on the changes people had made and promoted through to production development, and production in AWS," said Katz.

Being based in Chicago, the development team went through a lot of Lou Malnati's famous deep dish pizza, and in the end, Katz says, "That two-day effort resulted in approximately $1.4 million in annual AWS cost reductions."

Tagging workloads

To begin with, the primary piece of advice Katz offers for such an endeavour is to institute a policy and automation for tagging workloads. Katz said teams did not implement this initially, and as a result he spent a lot of time in spreadsheets tying workloads to cost centers. Now when a workload is spun up in OpenShift, it defaults to Katz as the owner (and to his department as the cost center). If a developer is spinning up the workload, it's tagged with that developer's name, and so it rolls up to their cost center.

That's important because there is a major FinTech effort inside of Discover, looking to optimize costs and tie IT spending directly to budget lines. There's also an impetus to integrate with Capital One's IT infrastructure due to the acquisition, and tagging workloads aids in that effort as well.

Technology and developer culture

Even before Katz set up the two-day Game Day event, he was already requesting his developers set the Requests metadata for their containers as a best practice. Efforts to spread this tip across the organization hadn't quite succeeded, however. 

While some people prefer to have documentation, others want webinars or classes so they can ask questions, said Katz. And while he did try to offer such options, cost optimizations weren't quite happening.

But when he got everyone into the same room and scheduled speakers to explain just how to optimize each chunk of the application lifecycle, that's when the magic happened. Each speaker would talk and show some slides, then the teams in the audience would put what they learned directly to use in their applications. Then the next speaker began. It was as if they were doing real life optimization passes for two days straight. 

Of course, before the Game Day event, Katz had taken the IT teams through some admin-based AWS cost optimizations as well. "We're using Turbonomic, which is IBM. That's helped us with efficiency of the services we [run]. [It does] the dense packing and moving stuff around to keep the node affinity where it needs to be, and just not wasting space,” said Katz.

"Did we need two days to do this whole thing? No. It was about trying to bring people together and getting them immersed in the culture and why we're doing it. We'd also been at Discover for months saying, update these two values! Look here at this dashboard, update this number! We just didn't get as much traction. So it was kind of like bringing the culture in on top of it because technology is obviously the most important thing, but without culture it sometimes just gets stagnant and unused," said Katz.

Learn more about Red Hat OpenShift Service on AWS