Citigroup: Red Hat Innovation Award Winner

Logo - No Image

June 17, 2010

Customer: Citigroup Global Markets, Inc.

Geography: North America
Country: United States


Business Challenge:

Reconciling two independently developed and supported Linux platforms to run mission-critical applications for Citi's globally distributed business units

Software:

Red Hat Enterprise Linux

Hardware:

x86 servers, IBM System z mainframes

Benefits:

By delivering a common global Linux build across the enterprise that can be leveraged across both x86 and IBM mainframe platforms, Citi has been able to retire a number of one-off infrastructure software products and their associated costs.

More
Background:

As one of the largest and most prestigious financial services firms in the world, Citi has approximately 200 million customer accounts in more than 140 countries. Through Citicorp and Citi Holdings, Citi provides consumers, corporations, governments and institutions with a broad range of financial products and services, including consumer banking and credit, corporate and investment banking, securities brokerage, transaction services and wealth management. The company employs approximately 263,000 staff around the world.

Business Challenge:

Within Citi, there were two independently developed and supported Linux platforms available for the business units to use.

The first was Linux Standard Operating Environment (SOE), which was introduced in 2003 based on Red Hat Enterprise Linux Advanced Server 2.1. Subsequently, the 2.1 version was retired as versions based on Red Hat Enterprise Linux 3 and Red Hat Enterprise Linux 4 were introduced. Most recently, SOE Linux 5 based on Red Hat Enterprise Linux 5 was released for production in May 2008. These operating environments at Citi run on industry standard Intel and AMD x86 servers. The second Linux platform had been developed by Citi’s IBM mainframe organization. Called mainframe Linux, or “zLinux,” the first release at Citi was based on Novell SUSE Linux Enterprise Server (SLES) version 8.

Although IBM provided cost incentives that made zLinux a potentially compelling solution for certain types of workloads, the differences in the technologies and operational models between the two distributions of Linux used had been a barrier to wider adoption of the platform. To address this, Citi initiated a project to identify and close the gaps between the current Linux SOE (distributed) and zLinux (mainframe) operating environments. Such gaps included the security model, the release and patch management processes, software provisioning, and differences in operational model. By closing these gaps, Citi hoped both environments would benefit by increasing consistency and improving portability across x86 and System z architectures.

Additionally, for any operating system platform to be considered a candidate for global certification within Citi, it had to support the firm’s Global Security Framework, a cross-platform security model definition that uses a common stack of internally-engineered and third-party products to achieve compliance with Citi’s internal security standards.

Another requirement was that the new Linux platform had to integrate with Citi’s Data Center Automation project, which seeks to deliver a common interface for software provisioning and patch management functions across all platforms and operating systems used at Citi.

Citigroup’s ultimate goal was too deliver a single Linux operating system environment that could be leveraged on both x86 servers and IBM System z mainframes while still meeting both these enterprise-wide requirements.

Solution:

To achieve its goal, Citi chose Red Hat Enterprise Linux to be the standard Linux distribution across both x86 servers and IBM System z mainframe for a number of reasons.

First and foremost, Citi likened to Red Hat’s approach to supporting its products across multiple hardware platforms. Citi found the Red Hat Enterprise Linux releases for each hardware architecture were built exactly the same, consistently sharing the same package manifests and update schedules. This greatly simplified Citi’s task, since a single Citi internal platform definition could be released for both x86 and IBM System z platforms, thus minimizing the internal certification process.

Red Hat Enterprise Linux also proved to have broad third-party ISV software product support. This was critical in folding these Linux builds into the Global Security Framework initiative, which in turn, greatly simplified obtaining security certification as required by Citi’s internal auditors.

Integrating the Linux builds into the Data Center Automation project also turned out to be surprisingly easy with Red Hat Enterprise Linux. By leveraging the Kickstart and Yum features of Red Hat Enterprise Linux, Citi was able to create an open and flexible interface and tie in to the front-end that the system administrator teams could use to perform key maintenance tasks.

Benefits:

There is a big push within Citi to drive IT optimization and consistency. By delivering a common global Linux build across the enterprise, the firm has been able to retire a number of one-off infrastructure software products and their associated costs. In the process, it has been able to negotiate enterprise-level agreements for a reduced number of third-party products that can be leveraged across both hardware platforms.

Additionally, virtualization has proven to be a successful strategy for optimizing hardware utilization. IBM mainframes have a long tradition of delivering virtualization, and perform especially well for workloads that require heavy input / output, whereas the value proposition for virtualization on x86 mainly comes from the low cost of entry and the ability to scale out massively. With a common Linux build, Citi is able to choose the virtualization that best meets its requirements.

Supporting a common Linux build has also reduced the time that the Citi engineering organization needs to spend on “business as usual” activities, such as certification of operating system updates and security vulnerability solutions. The benefits of this are twofold. First, these certifications are delivered to the business more quickly, thereby reducing risk. Secondly, and just as importantly, Citi was able to redirect those resources to work on strategic solutions such as real-time Linux (MRG) and low-latency networking. This translates to increased competitive advantage for the lines of business.

And Citi also took advantage of the fact that Red Hat does not stop with just packaging and supporting software from “upstream” developers. Instead, Red Hat fully embraces the open source philosophy as reflected by its highly transparent and collaborative way of doing business both with upstream vendors and their customers. This has encouraged the Citi Linux engineering team to model the same behavior in the way it approaches its own work and how it engages with internal clients.

Indeed, the community that has grown around Citi’s internal Linux consolidation project is so strong that engineering leaders of other teams within the firm are now beginning to embrace this model as well. Where a traditional engineering process would have gotten bogged down within organization barriers, Citi instead experienced an infectious enthusiasm to get the job done, both among its internal teams as well as when collaborating with Red Hat and other vendors.

During the early stages of the project, the mainframe team expressed some concern that Red Hat Enterprise Linux did not take advantage of all the advanced features of the System z hardware architecture. To address these concerns, the Red Hat account team brought in a solutions architect who specialized on the System z platform. By being able to speak the mainframe team’s language, he helped bridge the gaps in understanding between the internal Linux teams at Citi.

Today, SOE Linux is used for a wide variety of applications across all of Citi’s lines of business. To support the Global Security Framework, tight integration was required with third party products in the areas of system baselines, auditing, compliance, backup and recovery, capacity planning, and enterprise monitoring.

For the x86 platforms, SOE Linux operating systems are deployed on several thousand servers globally. Operational roll out on System z hardware is just getting started, but based upon the project teams that are already adopting the platform, Citi is projecting to have at least several hundred instances on System z by the end of 2010.

With this project, Citi delivered a family of operating system builds across different hardware architectures that achieved an unanticipated level of consistency. It also beat its initial time-to-market estimate by four months. By emulating the Red Hat open source model of working, Citi also avoided finger pointing, quickly getting the answers it needed to solve some complex issues involving multiple parties both inside and outside of the organization.

Contact Sales

Less