High throughput, resiliency, scalability and speed — the new generation of digital leaders have mastered these capabilities in order to be more competitive, and many of them have accomplished this by leveraging microservices and event-driven architecture (EDA).
EDA is an approach to develop applications in order to maximize the benefits of microservices. Simply put, EDA is a software development method for building applications that asynchronously communicate or integrate with other applications and systems via events.
Events are used as a way of establishing communication between applications, as well as with systems. An event can be any occurrence or change in state that is identified by the application. A good example of an event is the addition of a product to a shopping cart on an ecommerce site. An application designated as a producer detects the event and sends out relevant data in the form of a message. The middleman, an event processor, directs the message to an application designated as a consumer, which receives the message. Multiple consumers can receive the same message and utilize or even process the associated data in their own way, to accomplish the specific job for which the application is designed.
EDA's asynchronous event-based communication differs from synchronous communication in which two applications make a direct connection, most commonly via application programming interfaces (APIs). Conversely, asynchronous communication is event-driven, allowing multiple applications to communicate simultaneously and rapidly, in real time. EDA is considered the best way to enable asynchronous event-driven application interaction.
Today's development model relies on revolutionary technologies — like microservices and cloud-native development — that make distributed systems and applications possible. Distributed means the system or application is built from components physically located in different places. The secret to making distributed computing work is good solid communication between these components.
Traditional synchronous architecture cannot handle the performance and scale needs of distributed systems, and consequently applications based on this more conventional architecture face debilitating challenges in a distributed environment, including communication latency, system degradation, unpredictable failures, and more. Moving to EDA helps developers overcome these challenges. EDA requires minimal coupling between the services, while they can still communicate with each other, which makes EDA optimal for modern, distributed applications.
EDA in the Real World
One of the most appealing aspects of EDA is that it fits well into the real world, supporting many applications that are indispensable in the digital economy. This is because real world interactions are also event driven.
EDA is an ideal architecture for enterprise applications that derive the most benefit from scalable and reliable real-time communication, such as stream processing, data integration, and website activity tracking.
EDA enables developers to build applications designed to react to, process or transform multiple streams of events in real time. A stream is a continuous flow of events sent from a producer or multiple producers. The advantages of EDA for stream processing applications include scalability to handle large amounts of streaming data, and the reliability to ensure the lines of communication are kept open.
One excellent example of stream processing that makes a great fit with EDA is IoT monitoring. The Internet-of-Things (IoT) relies on machine-learning (ML) microservices to process events in a timely manner, to enable proactive repair and replacement of mission-critical devices, before they disrupt the system. This capability is crucial to IoT operation, which relies on the continuous operation of many devices, all delivering up-to-the-minute data. EDA's reliable, low-latency event-streaming architecture makes real-time processing of ML models possible in the IoT environment.
Another compelling example of stream processing that can benefit from EDA is financial fraud detection —the discovery, identification, and remediation of intentional misrepresentation of financial data. Critical fraud detection applications enable real-time and historical analysis of customer behavior across multiple systems such as credit, banking, and trading. Detecting financial fraud requires near zero latency between the actual event — the fraud itself — and the identification and remediation. EDA enables shared, reliable, low-latency event distribution so all stakeholders have access to timely and consistent data, in order to gain maximum visibility.
EDA is well-suited for developing data integration applications responsible for capturing, aggregating, and processing data from multiple diverse sources and presenting it in a single, unified view, or feeding that data to a variety of other unconnected systems. Data integration applications provide indispensable support for decision makers in a variety of industries, with up-to-date, accurate and easily accessible information.
Data integration can be used to process streams of events for real-time monitoring, alerting and analysis, while concurrently sending that same data to another system for retention and historical analysis. Traditional monolithic synchronous applications each store their own data separately, typically in a relational database that other applications cannot easily access. The asynchronous nature of EDA supports a shared data store, enabling applications to more easily access data from multiple sources, and also share data across the entire enterprise. All the related applications and systems can use the same accurate real-time information without compromising the integrity of the data.
One example of a data integration application built upon EDA is a retail dashboard which delivers real-time trends to decision makers while simultaneously saving the same data to enterprise data lakes for later analysis. EDA can help enable the application to process billions of events every day — involving pricing, transactions, inventory and more — to provide a retailer with real-time granular visibility into the metrics that define business success and failure.
Another example of data integration based on EDA would be an application that tracks customer interactions. EDA enables the application to process data concurrently for usage by both internal systems and customer-facing web and mobile applications. While delivering essential customer insights to the business, the application also provides customers with timely information about their accounts and transactions, ensuring that both the company and its customers have access to the same accurate and up-to-date data.
Website Activity Tracking
EDA can be very helpful to developers building applications that track user activity on websites. This type of application provides a company with visibility, in terms of both customer activity and website performance. Each customer interaction on the website — from page views and ad clicks to ecommerce transactions—would be consumed and processed as an event by EDA, and the resulting metrics sent to multiple consumers.
For the business, this can provide insight into the customer's interest, and potentially trigger automated suggestions for additional products and services based on the customer's perceived needs. Concurrently, the data is sent for analysis by other applications, to forecast product demand, for example.
This same type of EDA application can be used to serve ads to website visitors in real time. Ad serving requires split-second event processing with zero latency in order to automatically deliver the right ad to the right customer at the right time.
EDA Delivers Business Advantages
EDA can provide a range of valuable business advantages to the users of event-driven applications.
EDA can enable users to view and analyze the business in real time, and react quickly to the continuously changing market. EDA can be essential for establishing and maintaining real-time connectivity with internal stakeholders and external partners, suppliers and customers.
In the digital world, many applications depend on immediate access to data. By minimizing the need for point-to-point integrations for data sharing, EDA can streamline communications and reduce latency to milliseconds.
Context-Informed Decision Making
With real-time event processing, EDA can help your application to deliver the mission-critical up-to-date and accurate information and situational awareness your users need to make the right business decisions.
Big Data Scalability
EDA can help you to build highly scalable applications, capable of handling the massive volumes of data needed for today's analytics, and offering the flexibility to scale up and down quickly based on need.
With EDA, your applications can handle high throughput with low latency, while independently subscribing to and processing events in parallel.
EDA is designed to ensure reliable communication and operation, mitigating service downtime. In a synchronous model, a service must be available and responsive in order to communicate. When a service is not available, the entire system can stall or even break down.
With EDA, events do not require a reply and are inherently asynchronous, which means events can be consumed and processed later, if a service is busy or unavailable. In addition, services are all decoupled in the EDA model, which means that if a service goes down it does not impact any of the other services in the chain. In this way, EDA offers high resilience to failure, assuring vital business continuity.
Finding the Right EDA Solution
The industry's go-to technology for EDA is Apache Kafka, a publish and subscribe event distribution system that offers fault tolerance, high throughput, low latency, and horizontal scalability. Kafka is the industry standard for event-streaming architectures and stream processing applications.
We see that Kafka is frequently combined with Kubernetes, today's container management system of choice which is used to automate deployment, scaling, and operations of containers across clusters of hosts. Together, we consider Kafka and Kubernetes the state-of-the-art in event-streaming technology. The combination of Kafka and Kubernetes helps developers to create event-driven, real-time data pipelines that stream and integrate data between applications.