A quick guide to Automated Data-Streaming with the Flink Operator on OpenShift

What is the Flink?

What is Apache Flink? Apache Flink is a framework that enables real-time data stream processing. Real-time data processing is integral to meeting customer expectations. With Apache Flink, companies can provide their customers with real-time information, whether it be delayed shipments or fraudulent card transactions. Customers receive instant access to their data. Check out the resources at the end of the article for more information on Apache Flink.

What is an Operator? 

An Operator is a Kubernetes native application that extends upon the controller concept found in Kubernetes resources. An Operator consists of three key components: custom resource, custom controller, and application-specific knowledge. Utilizing existing Kubernetes controller concepts, an Operator is a custom controller that manages the state of a cluster based on the application-specific knowledge embedded in your custom resource definition. Operators are personalized controllers that can handle the complete lifecycle of an application, from deployment to troubleshooting to autoscaling. It is a powerful tool that takes the heavy lifting from application management.

What is the Flink Operator?

The Flink Operator is a control plane that deploys and manages the entire lifecycle of Apache Flink applications. The goal of the Flink Operator is to manage applications as a human operator would. It handles cluster startup, deploys jobs, updates apps, and resolves prevalent problems. It can automate operational tasks and comprehensively manage Apache Flink applications.

Install the Flink Operator on OpenShift

Installing the Flink Operator on Red Hat OpenShift Container Platform (RHOCP) is a simple process. All you need is an OpenShift cluster and admin privileges. Follow the steps below to get the Flink Operator on your cluster. 

Step 1: Install the Flink Operator

Starting from the RHOCP homepage: 

• Navigate to the sidebar and click Operators.
• In the dropdown, select OperatorHub.
• Once you've completed that, search for Flink in the search box and select the Flink Kubernetes Operator (it is a community operator and will be labeled so).
• Select Install, and install the operator with the default values unless you have other preferences.
• After the Flink Operator installs, navigate to the operator via View Operator or Operators > Installed Operators > Flink Kubernetes Operator.
• On the operator details page, create an instance of both the Flink Deployment and Flink Session Job.

Once you create those instances, you have successfully created an Apache Flink application. Hurray!

Step 2: Access the Apache Flink web dashboard

To access your web dashboard, simply port-forward the service: 

oc port-forward svc/basic-example-rest 8081

You can also create a route to view the web dashboard if you don't want to keep a terminal running. In your terminal, apply this resource to create a route resource.

oc apply -f -<< EOF 
kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: flink-route
  namespace: openshift-operators
spec:
  to:
    kind: Service
    name: basic-example-rest
  tls: null
  port:
    targetPort: 8081
EOF

Once you create the route, use the command oc get route to retrieve the Apache Flink web dashboard URL, which can be found under HOST/PORT. 

That concludes the setup, but feel free to explore the dashboard and see what Apache Flink offers.

Wrap up

Apache Flink is a mighty tool for data streaming and will continue to be a shining asset to companies who use it. Providing real-time data to customers is essential to meeting customer expectations. The Flink Operator—and Operators, in general—automate and ease the job of developers and human operators, freeing up time and resources for developers to focus and work on more important matters. 

Besides the Flink Operator, many other operators are available on RHOCP in OperatorHub that can be useful to you, and you can install them at the click of a button. Feel free to browse the large variety of community, certified, and marketplace operators available in OperatorHub. Explore the Flink Operator further and ease the workflow and management of your Apache Flink applications. Sit back and relax as the Flink Operator manages your application for you!

Resources

Apache Flink: Intro to Stream Processing with Apache Flink | Apache Flink 101

Red Hat OpenShift: Red Hat OpenShift Container Platform