* [Topics](/en/topics "Topics") * [Open source](/en/topics/open-source "Open source") * What is Apache Kafka? What is Apache Kafka? ===================== Updated October 10, 2022•*4*-minute read Copy URL Jump to section --------------- OverviewAsynchronous integrationWhen to useHow Kubernetes scales Overview -------- Apache Kafka is a distributed [data streaming](/en/topics/integration/what-is-streaming-data) platform that can publish, subscribe to, store, and process streams of records in real time. It is designed to handle data streams from multiple sources and deliver them to multiple consumers. In short, it moves massive amounts of data—not just from point A to B, but from points A to Z and anywhere else you need, all at the same time. Apache Kafka is an alternative to a traditional enterprise messaging system. It started out as an internal system developed by Linkedin to handle 1.4 trillion messages per day, but now it's an open source data streaming solution with application for a variety of enterprise needs. [Read more about Streams for Apache Kafka](https://access.redhat.com/products/streams-for-apache-kafka "Streams for Apache Kafka") Asynchronous integration with Apache Kafka ------------------------------------------ [Microservices](/en/topics/microservices) have changed the development landscape. They make developers more agile by reducing dependencies, such as shared database tiers. But the distributed applications your developers are building still need some type of integration to share data. One popular integration option, known as the synchronous method, utilizes application programming interfaces (APIs) to share data between different users. A second integration option, the asynchronous method, involves replicating data in an intermediate store. This is where Apache Kafka comes in, streaming data from other development teams to populate the data store, so the data can be shared between multiple teams and their applications. Microservices teams have different requirements for integration than traditional, waterfall development teams. These teams require 3 foundational capabilities: 1. [Distributed integrations](/en/topics/integration): Lightweight, patterns-based integrations that can be continuously deployed where required, and are not limited by centralized ESB type deployments. 2. [APIs](/en/topics/api): API-based services to foster an ecosystem of partners, customers, and developers that can offer reliable and profitable use of services. 3. [Containers](/en/topics/containers): Platform to develop, manage, and scale [cloud-native](/en/topics/cloud-native-apps) and connected applications. Containers enable development of lean artifacts that are individually deployable, part of [DevOps](/en/topics/devops) processes, and supported by out-of-box clustering, ensuring high availability. Red Hat calls this approach "agile integration," which allows integrations to be part of application development processes, providing more agility and adaptive solutions. Part of agile integration is the freedom to use either synchronous or asynchronous integration, depending on the specific needs of the application. Apache Kafka is a great option when using asynchronous [event driven integration](/en/topics/integration/what-is-event-driven-architecture) to augment your use of synchronous integration and APIs, further supporting microservices and enabling agile integration. In this way, Apache Kafka can be an important part of your initiative to streamline the development process, drive innovation, save time, and ultimately speed up time to market for your new features, apps, and services. Recommended for you From legacy to leading edge: A practical guide to industrial modernization -------------------------------------------------------------------------- [Watch the webinar](https://www.redhat.com/en/events/webinar/from-legacy-to-leading-edge-a-practical-guide-to-industrial-modernization?percmp=RHCTG0250000455235) When to use Apache Kafka ------------------------ Apache Kafka is built into streaming data pipelines that share data between systems and/or applications, and it is also built into the systems and applications that consume that data. Apache Kafka supports a range of use cases where high throughput and scalability are vital. Since Apache Kafka minimizes the need for point-to-point integrations for data sharing in certain applications, it can reduce latency to milliseconds. This means data is available to users faster, which can be advantageous in use cases that require real-time data availability, such as IT operations and e-commerce. Apache Kafka can handle millions of data points per second, which makes it well-suited for big data challenges. However, Kafka also makes sense for companies that are not currently handling such extreme data scenarios. In many data processing use cases, such as the [Internet of Things](/en/topics/internet-of-things/what-is-iot) (IoT) and social media, data is increasing exponentially, and may quickly overwhelm an application you are building based on today's data volume. In terms of data processing, you must consider scalability, and that means planning for the increased proliferation of your data. This video can't play due to privacy settings To change your settings, select the "Cookie Preferences" link in the footer and opt in to "Advertising Cookies or try disabling adblockers." ### IT operations IT Operations is all about data. IT Operations needs access to the data, and they need it quickly. This is the only way to keep websites, applications, and systems up and running and performing at all times. Apache Kafka is a good fit for IT Operations functions that rely on collecting data from a variety of sources such as monitoring, alerting, and reporting; log management; and tracking website activity. ### Internet of Things The value of [IoT](/en/topics/internet-of-things/what-is-iot) is the actionable data generated by physical objects connected to the internet. Apache Kafka is designed for scalability to handle the massive amount of data expected from IoT. ### E-commerce E-commerce is a growing opportunity for using Apache Kafka, which can process data such as page clicks, likes, searches, orders, shopping carts, and inventory. How Kubernetes scales Apache Kafka applications ----------------------------------------------- [Kubernetes](/en/topics/containers/what-is-kubernetes) is the ideal platform for Apache Kafka. Developers need a scalable platform to host Kafka applications, and Kubernetes is the answer. Like Apache Kafka, Kubernetes also makes your development process more agile. Kubernetes—the technology behind Google’s [cloud services](/en/topics/cloud-computing/what-are-cloud-services)—is an open source system for managing containerized applications, and it eliminates many of the manual processes associated with containers. Using Apache Kafka in Kubernetes streamlines the deployment, configuration, management, and use of Apache Kafka. By combining Kafka and Kubernetes, you gain all the benefits of Kafka, and also the advantages of Kubernetes: scalability, [high availability](/en/topics/linux/what-is-high-availability), portability and easy deployment. The scalability of Kubernetes is a natural complement to Kafka. In Kubernetes, you can scale resources up and down with a simple command, or scale automatically based on usage as needed to make the best use of your computing, networking, and storage infrastructure. This capability enables Apache Kafka to share a limited pool of resources with other applications. Kubernetes also offers Apache Kafka portability across infrastructure providers and operating systems. With Kubernetes, Apache Kafka clusters can span across on-site and public, private, or hybrid clouds, and use different operating systems. [Learn more about running Apache Kafka on Kubernetes](/en/topics/integration/why-run-apache-kafka-on-kubernetes) The official Red Hat blog ------------------------- Get the latest information about our ecosystem of customers, partners, and communities. [Keep reading](/en/blog "The official Red Hat blog") Recommended for you Developing Event-Driven Applications with Apache Kafka and Red Hat AMQ Streams ------------------------------------------------------------------------------ Develop, scale, and troubleshoot event-driven microservice applications. [View course](https://www.redhat.com/en/services/training/ad482-developing-event-driven-applications-apache-kafka-and-red-hat-amq-streams?percmp=RHCTG0250000455236) Keep reading ------------ ### What is InstructLab? InstructLab simplifies the process of customizing large language models with private data. [Read the article](/en/topics/ai/what-is-instructlab "article | what is instructlab?") ### What is Istio? Find out more about Istio, an open source service mesh that controls how microservices share data with one another. [Read the article](/en/topics/microservices/what-is-istio "article | What is Istio?") ### What is CentOS Stream? CentOS Stream is a Linux® development platform where open source community members can contribute to Red Hat® Enterprise Linux in tandem with Red Hat developers. [Read the article](/en/topics/linux/what-is-centos-stream "product article | what is centos stream") Open source resources --------------------- ### Related content * Case study [Turkcell builds resilient telco cloud with Red Hat](/en/resources/turkcell-emea-case-study) * Blog post [Bringing intelligent, efficient routing to open source AI with vLLM Semantic Router](/en/blog/bringing-intelligent-efficient-routing-open-source-ai-vllm-semantic-router) * Blog post [KServe joins CNCF as an incubating project](/en/blog/kserve-joins-cncf-incubating-project) * Case study [HCA Healthcare develops predictive analytics using Red Hat software](/en/resources/hca-healthcare-case-study) ### Related articles * [What is InstructLab?](/en/topics/ai/what-is-instructlab) * [What is Istio?](/en/topics/microservices/what-is-istio) * [What is a service mesh?](/en/topics/microservices/what-is-a-service-mesh) * [What is CentOS Stream?](/en/topics/linux/what-is-centos-stream) * [What is KVM?](/en/topics/virtualization/what-is-KVM) * [What is Podman Desktop?](/en/topics/containers/what-is-podman-desktop) * [What is CentOS?](/en/topics/linux/what-is-centos) * [What are CentOS replacements?](/en/topics/linux/centos-alternatives) * [What is Podman?](/en/topics/containers/what-is-podman) * [What is application integration?](/en/topics/cloud-native-apps/application-integration) * [What is Helm?](/en/topics/devops/what-is-helm) * [What is Argo CD?](/en/topics/devops/what-is-argocd) * [What is MicroShift?](/en/topics/edge-computing/microshift) * [OpenJDK versus Oracle JDK](/en/topics/application-modernization/openjdk-vs-oracle-jdk) * [What is Cloud Foundry?](/en/topics/application-modernization/what-is-cloud-foundry) * [What is Buildah?](/en/topics/containers/what-is-buildah) * [Understanding Ansible, Terraform, Puppet, Chef, and Salt](/en/topics/automation/understanding-ansible-vs-terraform-puppet-chef-and-salt) * [Ansible vs. Chef: What you need to know](/en/topics/automation/ansible-vs-chef) * [Ansible vs. Salt: What you need to know](/en/topics/automation/ansible-vs-salt) * [What's the best Linux distro for you?](/en/topics/linux/whats-the-best-linux-distro-for-you) * [What is Linux?](/en/topics/linux/what-is-linux) * [What is middleware?](/en/topics/middleware/what-is-middleware) * [Ansible vs. Puppet: What you need to know](/en/topics/automation/ansible-vs-puppet) * [Red Hat OpenShift vs. OKD](/en/topics/containers/red-hat-openshift-okd) * [Spring on Kubernetes with Red Hat OpenShift](/en/technologies/cloud-computing/openshift/spring) * [Why run Apache Kafka on Kubernetes?](/en/topics/integration/why-run-apache-kafka-on-kubernetes) * [Ansible vs. Terraform, clarified](/en/topics/automation/ansible-vs-terraform) * [Ansible vs. Red Hat Ansible Automation Platform](/en/technologies/management/ansible/ansible-vs-red-hat-ansible-automation-platform) * [Why choose Red Hat for middleware?](/en/topics/middleware/why-choose-red-hat-middleware) * [What is Skopeo?](/en/topics/containers/what-is-skopeo) * [Using Helm with Red Hat OpenShift](/en/technologies/cloud-computing/openshift/helm) * [What is an API?](/en/topics/api/what-are-application-programming-interfaces) * [What is Grafana?](/en/topics/data-services/what-is-grafana) * [What is open source software?](/en/topics/open-source/what-is-open-source-software) * [Open source vs. proprietary software in vehicles](/en/topics/open-source/open-source-vs-proprietary-software-in-vehicles) * [Red Hat OpenShift application services](/en/technologies/cloud-computing/openshift/application-services) * [What is an independent software vendor (ISV)?](/en/topics/digital-transformation/isv-partners) * [What is a Kafka service?](/en/topics/integration/what-is-a-kafka-service) * [What is KubeLinter?](/en/topics/containers/what-is-kubelinter) * [What is streaming data?](/en/topics/integration/what-is-streaming-data) * [Do banking APIs benefit from cloud technology?](/en/topics/api/banking-api) * [What is a service registry?](/en/topics/integration/what-is-a-service-registry) * [What is an event mesh?](/en/topics/integration/what-is-an-event-mesh) * [How microservices support IT integration in healthcare](/en/topics/microservices/microservices-in-healthcare) * [What is RKT?](/en/topics/containers/what-is-rkt) * [What is Kogito?](/en/topics/automation/what-is-kogito) * [What was CoreOS and CoreOS container Linux](/en/technologies/cloud-computing/openshift/what-was-coreos) * [What is change data capture?](/en/topics/integration/what-is-change-data-capture) * [Why Red Hat for APIs?](/en/topics/api/why-choose-red-hat-apis) * [What is service-oriented architecture?](/en/topics/cloud-native-apps/what-is-service-oriented-architecture) * [What is the Kubernetes API?](/en/topics/containers/what-is-the-kubernetes-api) * [What is a REST API?](/en/topics/api/what-is-a-rest-api) * [What is Jaeger?](/en/topics/microservices/what-is-jaeger) * [What is open source?](/en/topics/open-source/what-is-open-source) * [What is event-driven architecture?](/en/topics/integration/what-is-event-driven-architecture) * [What is API monetization?](/en/topics/api/what-is-api-monetization) * [REST vs. SOAP](/en/topics/integration/whats-the-difference-between-soap-rest) * [Why choose Red Hat for integration?](/en/topics/integration/why-choose-red-hat-integration) * [What is Clair?](/en/topics/containers/what-is-clair) * [What is API security?](/en/topics/security/api-security) * [What is GraphQL?](/en/topics/api/what-is-graphql) * [What is API design?](/en/topics/api/what-is-api-design) * [What does an API gateway do?](/en/topics/api/what-does-an-api-gateway-do) * [What is API management?](/en/topics/api/what-is-api-management) * [What is etcd?](/en/topics/containers/what-is-etcd) * [What is Knative?](/en/topics/microservices/what-is-knative) * [What is Docker?](/en/topics/containers/what-is-docker) * [What is integration?](/en/topics/integration/what-is-integration) [More about this topic](/en/topics/open-source "More about this topic")