Event-driven data processing
Artificial intelligence and machine learning (AI/ML) have triggered a revolution with their ability to increase data value and make diverse information more useful and actionable. Serverless models and distributed event streaming technologies like Apache Kafka make AI/ML technologies even more powerful and scalable. Given the massive amounts of data flowing into organizations, software-defined storage technology has emerged as a critical component for automating and scaling data processing, pipeline applications, and infrastructure.
Red Hat® OpenShift® Container Storage combined with Red Hat OpenShift and Red Hat AMQ (part of Red Hat Integration) delivers a powerful foundation and architecture for automating data pipelines. Along with storage-based event generation in Red Hat Ceph® Storage, these technologies can aggregate, ingest, prepare, and manage data from its inception—automating data pipelines and scaling infrastructure based on demand.
Scalable event-driven data pipeline architectures
Notification-driven or event-driven architectures are increasingly important data processing tools. By connecting data ingest services to Red Hat OpenShift Serverless functions, data pipelines can automatically scale to meet requirements, growing or shrinking application infrastructure to process incoming data based on your specific organizational needs. Ingest notifications can move fresh data into data pipelines automatically, with the ability to customize metadata insertion upon ingest. With these innovations, organizations can respond rapidly to changing information or variations in customer behavior, providing the organization with timely insights based on the latest data for a range of use cases.
- Manufacturers can detect anomalies for quality assurance, manage retail product replenishment, and understand logistics.
- Healthcare facilities can automatically process images, using AI inference for detection and alerting—simultaneously anonymizing data for researchers to improve processes or accelerate cures.
- Financial services institutions can accelerate payments or utilize fraud detection to better serve their customers.
- Public sector use cases range from equipment maintenance to automatically detecting geospatial changes in satellite imagery.
Containers and Kubernetes orchestration are vital for deploying AI/ML in hybrid clouds.1 Figure 1 shows how a combination of Red Hat technologies can be used to build an ingest data pipeline for a financial services application. The solution includes integrated object bucket notifications in the Red Hat Ceph Storage RADOS gateway (RGW), data streaming services provided by Red Hat AMQ streams, and Serverless capabilities in Red Hat OpenShift.