Log in / Register
Contact us


Digital innovation through agile integration

Last Updated:

Software strategy matters for digital success

Everything changes. We are in a period of significant shifts in companies—even entire industries—demonstrated in rankings, like the Fortune Global 500. For the last century, these periods of volatility have been driven by a combination of technological change and capital expansion.

There is obvious competition between direct, traditional market segments, but digital disruption also opens up the ability to compete and gain revenue in new areas. For example, a movie streaming service like Netflix also launches a community around the software it created to run its services, or an online retailer like Amazon also innovates with public cloud management. Innovation requires more than a slick customer user interface (UI).

There has to be a foundation of technology, standardized processes, and open culture that allow an organization to be flexible, to build on its existing knowledge, and to incorporate new ideas.

At a strategic level, today’s software is expected to deliver on a multitude of new and different business objectives, from big data, application programming interfaces (APIs), and edge computing initiatives to omnichannel experiences. In addition, software needs to work across multiple business functions, business models, engagement channels, and stakeholder ecosystems—and it needs to do it all at ever faster rates of change and innovation. What companies expect from their software and hardware systems is the ability to adapt to new market realities, realign to capitalize on opportunities, and do all of this without missing a beat in efficiency and uptime. An organization that can change pricing overnight and make new product options available to its global customers and staff rapidly has an enormous advantage over one that requires a three-month rollout with a cascade of manual verification steps.

The ability to integrate applications and data, known as enterprise integration, is key to realizing different business objectives and delivering competitive services. New, and increasingly difficult, demands are being placed on old approaches as digital innovation and disruption become the norm.

Internal business workflows and customer interactions still rely on core systems of record and their supporting IT infrastructure, but delivering solid internal solutions faster has become radically more difficult. New challenges like the increased adoption of cloud applications, hybrid cloud IT environments, and the need to extend systems to reach partners and customers generate demand for modern applications. This makes enterprise integration even more important and delivering services in a faster, continuous way even more critical.

We believe a better way to address these new and rapidly multiplying challenges is to integrate different applications and information systems using agile integration strategies.

What is agile integration, and how does it relate to CI/CD?

The term “agile integration” commonly relates to continuous integration/continuous delivery (CI/ CD), which represent the integration or combination of different development processes into a continuous process.

In this whitepaper, we define agile integration as an architectural approach that specifically applies to “integration” technologies and processes—taking advantage of agile methods and flexible microservices architectures—so that applications and data across multiple systems and services can be integrated and adapted faster to meet the rapidly changing demands of digital business. Agile integration relies on modern platforms, processes, and technologies that are suited for fast-paced and adaptive solutions. Agile integration approaches can help customers include their integration services as part of CI/CD processes.

Why agile integration?

Enterprise service bus (ESB) and other traditional integration technologies provide the essential capabilities (like transformation, routing, orchestration, and connectivity) required to integrate and connect different applications. ESBs coupled with architectural patterns, like service-oriented architecture (SOA) provide a platform to encapsulate integration logic as reusable services. SOA takes advantage of modular business functions and the ability to reuse these services. But it also comes with challenges: technology and governance complexity, time-consuming implementation cycles, and more emphasis on reusability than agility, for  example.

For years, enterprises looking to mitigate the problems of monolithic application interoperability tried to make sense of the exponential increase in the number of connections between applications—expressed as x^(x-1), where x represented the number of applications in an enterprise. A solution to this problem was to integrate all applications into a single enterprise service bus. However, the multiple connection problem did not go away and instead, the complexity of those connections was constrained into a single box (the ESB), which could only scale vertically and itself became a monolithic application.

This architecture required centralized governance to control the connections inside the ESB. The goal of reducing complexity by forcing all applications into one central “connection box” failed. This solution reduced the agility of new application development. With a centralized ESB architecture and associated development processes, it became difficult to create, modify, and innovate with new services. This problem has become exponentially more complex with the advent of microservices architectures.

A microservices style architecture provides a more agile approach to developing applications by designing and developing application functionality as independently deployable services. Microservices architectures facilitate building agile business systems—systems that allow a business to change quickly, to build new functionality, to experiment, and to be more prepared to address disruptions.

However, as applications are broken into smaller, discrete services, integration capabilities like transformation, orchestration, and connection are still required. Integration is critical to microservices development, but centralized ESB technologies and deployment architectures do not support microservices architectures and the agile development processes that realize them. A different integration approach is required.

An agile integration approach relies on platforms, processes, and technologies that are more suited for adaptive solutions. With an agile integration approach, integrations can be part of application development processes, including microservices architectures, providing more agility. Integration should be a key capability of distributed teams tasked with delivering new systems of innovative solutions. Combining technological capabilities with different organizational and process approaches makes real change possible.

How to implement agile integration

The 3 key capabilities needed for an agile integration architectural approach are:

  1. Distributed integration
  2. Linux containers
  3. APIs


image container

Distributed integration—  flexibility to adapt

With users increasingly engaging through digital channels (mobile, social, messaging, and web) software has shifted toward a more user-centric model with demand for features and services being pulled from the outside in, rather than being pushed from the inside out. This, combined with the ease of access to user-friendly software tools and cloud services, has created a shift in the role of IT to one that is more about collaboration and enablement of the business, and less about centralized control and gatekeeping.

A further outcome of this is the increased prominence of the line of business in technology strategy and purchases. This creates better alignment of business goals with technology decisions.

As a result of these market and organizational movements, IT’s hold on integration has to loosen into a more modular and distributed model while still adhering to the security and governance that remain core to business requirements. The integration competency centers that were the de facto centers of excellence for best practices for enterprise integration are now evolving to a more distributed integration and business-oriented model. Large integration teams in the IT organization are giving way to smaller and more flexible teams that can respond with greater agility.

In progressing toward a more connected world, agile integration needs to be supported by more modular, lightweight, and pattern-based approaches in order to meet the demand for faster and simpler integration of new services and applications. Changing front-end or client-side requirements demands equal flexibility on back-end integrations. However, lightweight approaches to integration are not always as easy as they sound. Traditional ESBs, usually built on proprietary technologies, are heavyweight and share the downsides of other monolithic applications.

A lightweight, flexible integration platform that enables rapid integration across multiple enterprise systems and services, on-premise or in the cloud, is necessary. An integration platform that enables developers to quickly create lightweight API-based integration services, deploy where required, and scale those services as needed is critical.

Take the example of a big brand retailer that sells products through traditional distribution channels that it largely dominates. As this retailer becomes disrupted by nimble online providers, it needs to adapt and develop its digital channels and do so while keeping the lights on in its traditional business models. It does this by creating small teams. These teams, as Amazon CEO Jeff Bezos famously noted, should be small enough that they can be fed by 2 pizzas—4 to 7 people. Teams this small can respond quickly to the business, creating full end-to-end applications enabled by more lightweight integration platforms and APIs.

Containers—scale with demand

Modern applications often need to scale to hundreds of thousands or millions of transactions, frequently in an unplanned and elastic manner. In most cases, these applications need to scale independently of each other to deliver the appropriate data on demand. They also are often subject to continuous updates and development cycles to meet the rapid pace of digital demand.

In an increasingly connected and data-driven world, the scalability and management of underlying infrastructure, the speed of access to data, and continuous development and delivery cycles are critical to success. Integration that is compatible with container-based platforms and technologies—like Kubernetes container orchestration—play a central role in building apps that respond to customer needs.

Containers are fundamental to a distributed integration model outlined in the first agile integration capability (distributed integration) because they avoid the bottlenecks associated with the rigid and monolithic architecture of centralized ESB deployments.

In addition, containers provide the means to create lightweight yet fully tested and validated units of deployment that can be scaled independently on demand. Containers give teams more options for working in innovative ways. For instance, container app dev and integration can provide a robust technical foundation for adopting DevSecOps, which brings dev, security and compliance, and operations teams together to build and deploy apps rapidly and securely. 

Containers also provide an excellent paradigm to divide up complex systems into smaller working units—microservices. Deploying services in containers allows independent teams to accelerate delivery and innovation.

Above all, containers provide an important capability that permits consistent version management and scalability of individual building blocks of a system. This all provides a powerful foundation for distributed integration that meets multiple business needs.

APIs—connect and reuse efficiently

Web and mobile-enabled software is what users expect. The way that customers interface with business processes have become a critical differentiator in the marketplace.

As these endpoints and user interfaces multiply, point-to-point integration is no longer sustainable. Instead, APIs are becoming the accepted means for connecting business assets—IT systems, internal and external personnel, client applications, and customers—to maximize their potential value.

The benefit of APIs is that they can help large, traditional enterprises act like smaller, nimbler firms by increasing their agility. Conversely, APIs can enable start-up organizations to expand their presence rapidly across new territories.

By reducing the complexity of integration and accelerating the creation of applications, APIs can encourage internal innovation, reach new customers, extend products and services, and create vibrant partner ecosystems. This is one reason why teams and organizations are launching API programs to plan and strategically build APIs that help the business connect its data to a larger ecosystem.

The opening of APIs typically enables organizations to provide uniform data and transaction interfaces to internal and external developers, partners, and customers, for improved data access and transactions. Such organizations can also develop software applications to access these APIs to create new functionality and value both for themselves and the wider world. This enables many new classes of applications with the potential to transform the way business is done.

By unlocking data from back-end systems in a secure and repeatable manner, APIs can play a critical role in application development by:

  • Generating new revenue streams through API monetization (for example, by charging for access or enabling e-commerce).
  • Extending customer reach and value by offering existing services on new devices and platforms (for example, consumer IoT products and mobile apps).
  • Stimulating business’ technical innovation by making it easier to implement ideas without changing back-end systems.
  • Enabling the self-service of API-based integrated applications by agile teams.

These 3 capabilities—distributed integration, containers, and APIs—are foundational to efforts to make IT infrastructure more agile because they each raise the level of abstraction at which different teams can work together.

APIs and Linux containers both package specific key assets at a level that can be broadly understood. Treating integration as a distributed set of integrations placed wherever they are needed means they can be addressed as important parts in the overall infrastructure.

The Red Hat approach: Amsterdam Airport Schiphol builds an integrated data ecosystem with open source software

As a major Amsterdam Airport Schiphol has a wealth of data that connects travelers to important information as they travel through the airport. A core business objective was to make accurate, real-time, high-quality airport information from data owned by the group available to the broader ecosystem of internal and external partners in order to create an exceptional customer experience.

Amsterdam Airport Schiphol Group chose Red Hat Fuse, Red Hat 3scale API Management Platform, and Red Hat OpenShift as the key integration, API management, and container technologies, replacing their enterprise service bus, exposing services via RESTful APIs, and creating a multicloud platform.

This resulted in:

  • Reduced development costs for partner integration.
  • Increased revenue streams through partners.
  • Increased visibility and control over functionality and data shared with partners.

Learn more about how Amsterdam Airport Schiphol used Red Hat integration and container technologies to build a robust, vibrant application environment.

Conclusion: Distributed integration supports innovation

In the age of digital services, enterprise software needs to satisfy many new business objectives, incorporating mobile, big data, Internet of Things, cloud, and other transformative technologies as a means of improving business outcomes and remaining competitive. Add to this the faster pace of change that these technologies influence, and it becomes clear that the ability to integrate data from multiple sources at speed is critical to today’s business success.

Traditional integration technologies, such as centralized ESBs, involve technological complexity and long implementation cycles that limit the degree of agility businesses now require. An agile integration approach, backed by platforms specially suited for flexible and adaptive integrated solutions, can help take full advantage of existing and emerging technologies.

An architectural approach, supported by appropriate tools and agile processes, has three key capabilities:

  • Distributed integration—offering the flexibility to adapt at greater speed.
  • APIs—enabling more efficient connectivity.
  • Containers—supporting scaling on demand.

These capabilities help organizations evolve from traditional integration technologies and approaches towards a modern, agile integration model, positioning them for digital business transformation.

Learn how to build and integrate apps using open source tech and innovation in the Red Hat e-book Agile integration: A blueprint for enterprise architecture.