In the last few years, several Red Hat customers have asked how to add a Web Application Firewall (WAF) to the OpenShift ingress to protect all externally facing applications.
A WAF is a Layer 7 capability that protects applications against some types of web-based attacks, including but not limited to Cross Site Request Forgery (CRSF), Cross-Site Scripting (XSS) and SQL injection (for a more comprehensive list of all known web based attacks, see here).Unfortunately, OpenShift does not have these capabilities included within the default ingress router, and as a result, alternate solutions must be pursued.
But now, with the advent of the Gateway API and a new ingress stack for OpenShift, things are changing. In this article, we will demonstrate how you can protect your applications with a WAF.
OpenShift, Gateway API and WASM
Note: As of this writing, the OpenShift Gateway API implementation is in a dev preview support status and the functionality may change in the future, including the underlying technologies.
OpenShift, as a product, will adopt the new Gateway API standard. This feature is in dev preview behind a feature gate and can be activated as described in this article.
For the purpose of this article, we do not need to analyze all of the new features of the Gateway API (a good summary for that can be found here). However, the feature that we are most concerned with is that the reverse proxy implementation of the Gateway API for OpenShift is Envoy. In fact, by enabling the Gateway API feature, a clusterwide OpenShift Service Mesh is currently installed automatically.
By having the power of the Istio at our disposal, we can easily inject new behavior in Envoy using WebAssembly (WASM) packaged plugins.WASM is a technology that compiles code such that it runs safely in a lightweight stack-based virtual machine.WASM was originally created to inject behavior in the browser to support sophisticated websites, but it turned out to be ideal for all kinds of plugin systems.To be compatible with Envoy, a WASM program must implement the proxy-wasm Application Binary Interface (ABI). This interface specification ensures that Envoy understands how to call the WASM program.
ModSecurity, OWASP WAF core rule set and Coraza
ModSecurity is a mature and stable Apache module that adds WAF capabilities to an Apache instance. ModSecurity has been adopted over time by other http servers and proxies products, such as ngnix via an appropriate connector.ModSecurity has recently been donated to Open Web Application Security Project (OWASP) and under this organization, will be maintained as an open source project with open and transparent governance.
One of the features of ModSecurity is that it can be configured to check for a set of rules via a Domain Specific Language (DSL) called SecLang. The OWASP organization maintains a default set of rules that cover the most common attack types. This set of rules is referred to as the OWASP core rule set.When configuring a WAF based on ModSecurity, it is a good practice to start with the OWASP core rule set and then tweak the ruleset based on one’s needs.
Coraza is a port of ModSecurity to the golang language (ModSecurity is originally written in C) with the objective of making ModSecurity more adopted by enterprises. Coraza Proxy WASM is an adaptation of Coraza to the proxy-wasm-ABI.
That being said, by leveraging these solutions, it should start to be clear how we are going to enhance OpenShift ingress with a WAF.
Putting it all together
The following diagram depicts the approach for injecting Coraza in the Envoy proxy which manages the ingress traffic:
In the diagram above, we can see in yellow, the CRs that we created, and in green, pods or groups of pods that are running. You can find this implementation in a declarative GitOps format within this repository.
At the bottom of the diagram, we can find the GatewayClass CR. When configured appropriately, it triggers the deployment of an ServiceMesh (Istio) control plane. Moving up higher in the diagram, we can see the Gateway CR. This manifest creates an Istio ingress gateway pod. When the Gateway CR is deployed the OpenShift Ingress Operator creates a wildcard DNS entry for this ingress endpoint. Additionally, we created a wildcard certificate using cert-manager to enable TLS for all the subdomains managed by this endpoint.
The last piece of configuration that the administrator needs to do is to inject the Coraza WAF code as a WASM plugin. This can be achieved with a WasmPlugin CR. Notice that this manifest references an OCI artifact containing the WASM code.
On the tenant side, we can deploy a simple application that demonstrates the use of a WAF now that the capability has been enabled. The example application can be found here. The key component in this application is the HttpRoute. By specifying the parentRefs section, we request the newly created Gateway to serve this route.
Now that we have enabled the Gateway API within OpenShift and configured the cluster to support our use case, and deployed the sample application, we can finally test and validate the functionality.
Let’s first get the current cluster base domain:
export cluster_base_domain=$(oc get ingress.config.openshift.io cluster --template={{.spec.domain}} | sed -e "s/^apps.//")
Then we can issue a request:
curl -I https://nginx-echo-headers.gwapi.${cluster_base_domain}
HTTP/2 200
server: istio-envoy
date: Fri, 10 May 2024 18:23:50 GMT
content-type: text/plain
x-envoy-upstream-service-time: 2
Now, let’s make a request that looks a little more suspicious:
curl -I "https://nginx-echo-headers.gwapi.${cluster_base_domain}/?arg=<script>alert(0)</script>"
HTTP/2 403
date: Fri, 10 May 2024 18:25:21 GMT
server: istio-envoy
The request was denied with a 403 error.
Within the Istio ingress gateway pod log, messages similar to the following will be displayed:
024-05-10T18:25:21.318531Z critical envoy wasm wasm log openshift-ingress.coraza-waf: [client "100.64.0.5"] Coraza: Warning. XSS Attack Detected via libinjection [file "@owasp_crs/REQUEST-941-APPLICATION-ATTACK-XSS.conf"] [line "7663"] [id "941100"] [rev ""] [msg "XSS Attack Detected via libinjection"] [data "Matched Data: XSS data found within ARGS_GET:arg: <script>alert(0)</script>"] [severity "critical"] [ver "OWASP_CRS/4.0.0-rc2"] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-xss"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/152/242"] [hostname "10.129.2.11"] [uri "/?arg=<script>alert(0)</script>"] [unique_id "cmQXIPQegrompxpYJnD"]
126
…
Load test
By now, we know that the OWASP core rule set is being applied by our WAF./ But, was there a performance hit against the application? Let’s try to measure the delta with and without Coraza enabled.
To accomplish this task, we need to deploy another gateway without the plugin.
To generate load, we are going to use cassowary, a cross platform load testing tool. Here is our baseline, without Coraza:
podman run rogerw/cassowary:v0.14.1 -u https://nginx-echo-headers.gwapi-no-coraza.${cluster_base_domain} -c 10 -n 1500 -d 3
0 --disable-keep-alive
Starting Load Test with 1500 requests using 10 concurrent users
99% |███████████████████████████████████████ | [29s:0s] 30.056327229s
TCP Connect.....................: Avg/mean=17.74ms Median=17.00ms p(95)=22.00ms
Server Processing...............: Avg/mean=19.43ms Median=19.00ms p(95)=25.00ms
Content Transfer................: Avg/mean=0.00ms Median=0.00ms p(95)=0.00ms
Summary:
Total Req.......................: 1500
Failed Req......................: 0
DNS Lookup......................: 1.00ms
Req/s...........................: 49.91
Now, let’s see how the same load performs with Coraza enabled:
podman run rogerw/cassowary:v0.14.1 -u https://nginx-echo-headers.gwapi.${cluster_base_domain} -c 10 -n 1500 -d 30 --disable-keep-alive
Starting Load Test with 1500 requests using 10 concurrent users
99% |███████████████████████████████████████ | [29s:0s] 30.075051016s
TCP Connect.....................: Avg/mean=18.14ms Median=18.00ms p(95)=23.00ms
Server Processing...............: Avg/mean=33.46ms Median=33.00ms p(95)=40.00msEnvoy
Content Transfer................: Avg/mean=0.00ms Median=0.00ms p(95)=0.00ms
Summary:
Total Req.......................: 1500
Failed Req......................: 0
DNS Lookup......................: 1.00ms
Req/s...........................: 49.88
We can see that for this simple use case, the latency introduced by the WAF was around 14ms (33ms-19ms).
Also, we noticed that on the resource consumption side, the ingress gateway pod with Coraza (top in the picture below) consumes approximately one order of magnitude (x10) more memory and CPU.
Conclusions
In this article, we demonstrated an approach to add a WAF to our OpenShift ingress stack using the new Gateway API and injection via a WASM plugin. We used Coraza as the WAF implementation and the OWASP core rule set as our WAF policies. When attempting to deploy these assets in your own environment, please keep in mind that the Gateway API is still in dev preview in OpenShift and that the Coraza WASM plugin is not currently supported by Red Hat.
In addition to WAF and the core rule set, OWASP recommends sanitizing the HTTP headers following the OWASP Secure Headers Project “best practices”. These best practices can be tested at develop time, but one can also decide to sanitize the headers on the fly at run time. We built a custom WASM plugin to demonstrate this use case. The plugin can be found in this repository. This sample plugin attempts to satisfy most recommendations from the published HTTP Security Response Headers Cheat Sheet.
In our opinion, WASM is a promising approach for enhancing the behavior of the Envoy reverse proxy especially when Envoy is controlled by Istio (multiple plugins can be attached to the same Envoy instance). We also recommend investigating project Kuadrant in detail which uses WASM to add authz/authn and rate-limiting capabilities (and more) to an Envoy proxy.
About the authors
Trevor Box has been a Red Hatter since 2018 and concentrates on container platforms and their adoption.
Raffaele is a full-stack enterprise architect with 20+ years of experience. Raffaele started his career in Italy as a Java Architect then gradually moved to Integration Architect and then Enterprise Architect. Later he moved to the United States to eventually become an OpenShift Architect for Red Hat consulting services, acquiring, in the process, knowledge of the infrastructure side of IT.
Currently Raffaele covers a consulting position of cross-portfolio application architect with a focus on OpenShift. Most of his career Raffaele worked with large financial institutions allowing him to acquire an understanding of enterprise processes and security and compliance requirements of large enterprise customers.
Raffaele has become part of the CNCF TAG Storage and contributed to the Cloud Native Disaster Recovery whitepaper.
Recently Raffaele has been focusing on how to improve the developer experience by implementing internal development platforms (IDP).
Browse by channel
Automation
The latest on IT automation for tech, teams, and environments
Artificial intelligence
Updates on the platforms that free customers to run AI workloads anywhere
Open hybrid cloud
Explore how we build a more flexible future with hybrid cloud
Security
The latest on how we reduce risks across environments and technologies
Edge computing
Updates on the platforms that simplify operations at the edge
Infrastructure
The latest on the world’s leading enterprise Linux platform
Applications
Inside our solutions to the toughest application challenges
Original shows
Entertaining stories from the makers and leaders in enterprise tech
Products
- Red Hat Enterprise Linux
- Red Hat OpenShift
- Red Hat Ansible Automation Platform
- Cloud services
- See all products
Tools
- Training and certification
- My account
- Customer support
- Developer resources
- Find a partner
- Red Hat Ecosystem Catalog
- Red Hat value calculator
- Documentation
Try, buy, & sell
Communicate
About Red Hat
We’re the world’s leading provider of enterprise open source solutions—including Linux, cloud, container, and Kubernetes. We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.
Select a language
Red Hat legal and privacy links
- About Red Hat
- Jobs
- Events
- Locations
- Contact Red Hat
- Red Hat Blog
- Diversity, equity, and inclusion
- Cool Stuff Store
- Red Hat Summit