Rate Limit your services in the mesh like the old days
Rate limiting is a critical aspect of managing traffic in a mesh network, and we’re excited to announce that it’s now easier than ever to implement in OpenShift Service Mesh (OSSM) with Kuadrant. Developed by Red Hat, Kuadrant is a Kubernetes-native solution that integrates seamlessly with OpenShift Service Mesh and provides robust rate limiting and authorization capabilities to help you maintain control over your cluster’s resources. Kuadrant is currently in developer preview, and is not yet officially supported.
Kuadrant bundles Limitador, the rate limiting service, and Authorino, our lightweight authorization service, which are leveraged within Red Hat OpenShift application services today. The Kuadrant Operator, available from Operator Hub, will take care of all the wiring and deployments of the services required for applying Rate Limits and Authorization Policies to your infrastructure.
Once installed, Policy Custom Resource Definitions let cluster operators apply a cluster-wide Rate Limit policy to protect the infrastructure from DDoS attacks; while application developers can use advanced features to control usages of their API endpoints based on API keys handed out to their users. Kuadrant supports a wide range of use cases, from simple Rate Limiting to ones that span across authorization and identity-based Rate Limits.
Advanced Authnz workflows in OSSM
OpenShift Service Mesh already offers support for protecting APIs based on built-in Envoy filters for simple, opinionated use-cases such as JSON Web Token (JWT) authentication and authorization based on limited condition matchers, via Istio Authorization Policies. However, as developers, we are frequently asked to go the extra mile in order to support more complex real-world use cases while not giving up on the benefits of relying on proven standards and patterns for implementing authentication and authorization properly.
This is where Authorino, enabled by Kuadrant on OpenShift Service Mesh, comes in. Kuadrant Authorization Policies will not only handle the wiring of auth filtering for your network traffic but will also open up options for authentication and authorization based on API keys (for simpler and quicker onboarding), Kubernetes RBAC (for a truly Kubernetes-native setup), and Open Policy Agent (OPA) (for full-power, complex authorization rules), among others.
Here’s a practical example. Imagine you have an API that you want to be accessed by users authenticated by an Single-Sign On (SSO) server that implements OpenID Connect (OIDC) but also consumed within the cluster by other workloads authenticated using Kubernetes Service Account tokens. Moreover, say you want to enforce Role-Based Access Control (RBAC) with permissions (roles and role bindings) managed as Kubernetes resources; plus rate limit requests to your API per user. You can achieve all that with a single Kuadrant AuthPolicy custom resource.
Gateway API & Policy attachment
Kuadrant leverages the Kubernetes Gateway API, which is currently under development by the community. This API introduces a range of new network resources, including the Gateway, which represents an ingress point into your cluster, and the HTTPRoute, which connects a Gateway to a Kubernetes service for HTTP traffic. The API introduces more new resources, but most importantly all these resources are role-oriented. Managing the Gateway resource, for example, would be more targeted at cluster operators, while the HTTPRoute is targeted at application developers deploying Services on the cluster.
OpenShift Service Mesh 2.2 added support for the Kubernetes Gateway API as a tech preview. Leveraging that very same API, Kuadrant uses the Policy Attachment mechanism of the Gateway API to provide both Rate Limiting and Authorization. The Kuadrant policies, which are custom resources, represent the policy to apply to the network resource it points to, e.g. a RateLimitPolicy attached to an HTTPRoute that routes traffic to a given back-end service to a maximum given rate of 100 requests per second.
Secure Your Services in the Mesh with Kuadrant: A Hands-On Guide
Pre requisites
- OpenShift cluster
- oc CLI tool
- Admin privileges to the OpenShift cluster
- The following operators installed in the cluster
Limitations
As of today, but will be addressed in the near future
- 1 Kuadrant instance (CR) per cluster
- 1 AuthPolicy and RateLimitPolicy per network resource
- Namespace of policies needs to be the same of its target network resources
- Gateways that’ll be managed by Kuadrant should be already in the cluster
Diagram of what we’ll be doing
Prepare your service mesh
1. Login to your cluster
export CLUSTER_DOMAIN=<openshift-cluster-domain>
oc login --token=<secret> --server=https://api.$CLUSTER_DOMAIN:6443
kubectl get crd gateways.gateway.networking.k8s.io || { kubectl kustomize "github.com/kubernetes-sigs/gateway-api/config/crd?ref=v0.4.0" | kubectl apply -f -; }
3. Create the service mesh control plane (SMCP)
Create the namespace where the SMCP will be installed:
kubectl create namespace istio-system
Apply the ServiceMeshControlPlane
custom resource:
kubectl apply -n istio-system -f -<<EOF
apiVersion: maistra.io/v2
kind: ServiceMeshControlPlane
metadata:
name: istiocontrolplane # set the ISTIOOPERATOR_NAME env var in the Kuadrant Operator deployment if this is named otherwise
spec:
runtime:
components:
pilot:
container:
env:
PILOT_ENABLE_GATEWAY_API: "true"
PILOT_ENABLE_GATEWAY_API_DEPLOYMENT_CONTROLLER: "true"
PILOT_ENABLE_GATEWAY_API_STATUS: "true"
version: v2.3
policy:
type: Istiod
telemetry:
type: Istiod
addons:
prometheus:
enabled: false
kiali:
enabled: false
grafana:
enabled: false
EOF
4. Deploy the gateway
kubectl apply -n istio-system -f -<<EOF
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: Gateway
metadata:
labels:
istio: ingressgateway
name: istio-ingressgateway
spec:
gatewayClassName: istio
listeners:
- name: default
port: 80
protocol: HTTP
allowedRoutes:
namespaces:
from: All
addresses:
- value: istio-ingressgateway.istio-system.svc.cluster.local
type: Hostname
EOF
Set up Kuadrant
1. Install the Kuadrant Operator
Install Kuadrant from the your OpenShift cluster UI
2. Request a Kuadrant instance
Create the namespace:
kubectl create namespace kuadrant
Apply the Kuadrant
custom resource:
kubectl apply -n kuadrant -f -<<EOF
apiVersion: kuadrant.io/v1beta1
kind: Kuadrant
metadata:
name: kuadrant
spec: {}
EOF
Add an application to protect
1. Deploy an example application
Create the namespace where the application, its networking objects and protection policies will be installed:
kubectl create namespace toystore
As an example, one could use our testing multiuse “echo” API, and call it “toystore”:
kubectl apply -n toystore -f https://raw.githubusercontent.com/Kuadrant/kuadrant-operator/3e05cf590dab0faeca63c4b13b88771676beddc2/examples/toystore/toystore.yaml
2. Onboard the application into the mesh
Add the application namespace (toystore) to the mesh:
kubectl apply -n toystore -f -<<EOF
apiVersion: maistra.io/v1
kind: ServiceMeshMember
metadata:
name: default
spec:
controlPlaneRef:
namespace: istio-system
name: istiocontrolplane
EOF
Annotate the deployment for the Istio sidecar injection:
kubectl patch -n toystore deployment/toystore -p '{"spec": {"template":{"metadata":{"annotations":{"sidecar.istio.io/inject":"true"}}}} }'
Route the application through the gateway:
kubectl apply -n toystore -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
name: toystore
labels:
app: toystore
spec:
parentRefs:
- name: istio-ingressgateway
namespace: istio-system
hostnames: ["*.toystore.apps.$CLUSTER_DOMAIN"]
rules:
- matches:
- path:
type: PathPrefix
value: "/toy"
method: GET
backendRefs:
- name: toystore
port: 80
EOF
3.Expose the application
Expose the application for N/S traffic:
oc expose -n istio-system service/istio-ingressgateway --port 8080 --name toystore --hostname=api.toystore.apps.$CLUSTER_DOMAIN
Make sure the application is reachable (unprotected for now):
curl http://api.toystore.apps.$CLUSTER_DOMAIN/toy -i
# HTTP/1.1 200 OK
Protect the application with Kuadrant
1. Require API keys to authenticate to the application
Let’s keep things simple and protect the application with API keys.
Create API keys for users Bob and Alice:
kubectl -n toystore apply -f -<<EOF
---
apiVersion: v1
kind: Secret
metadata:
annotations:
secret.kuadrant.io/user-id: bob
name: bob-key
labels:
authorino.kuadrant.io/managed-by: authorino
app: toystore
stringData:
api_key: IAMBOB
type: Opaque
---
apiVersion: v1
kind: Secret
metadata:
annotations:
secret.kuadrant.io/user-id: alice
name: alice-key
labels:
authorino.kuadrant.io/managed-by: authorino
app: toystore
stringData:
api_key: IAMALICE
type: Opaque
EOF
Create a Kuadrant AuthPolicy
custom resource to configure the authentication:
kubectl -n toystore apply -f - <<EOF
---
apiVersion: kuadrant.io/v1beta1
kind: AuthPolicy
metadata:
name: toystore
spec:
targetRef:
group: gateway.networking.k8s.io
kind: HTTPRoute
name: toystore
rules:
authScheme:
identity:
- name: friends
apiKey:
allNamespaces: true
selector:
matchLabels:
app: toystore
credentials:
in: authorization_header
keySelector: APIKEY
response:
- json:
properties:
- name: userID
valueFrom:
authJSON: auth.identity.metadata.annotations.secret\.kuadrant\.io/user-id
name: rate-limit-apikey
wrapper: envoyDynamicMetadata
wrapperKey: ext_auth_data
EOF
Verify that the endpoint is now protected:
curl http://api.toystore.apps.$CLUSTER_DOMAIN/toy -i
# HTTP/1.1 401 Unauthorized
You should be able to access the application supplying either Alice’s or Bob’s API Key:
curl -H 'Authorization: APIKEY IAMBOB' http://api.toystore.apps.$CLUSTER_DOMAIN/toy -i
# HTTP/1.1 200 OK
2. Rate limit the application
Let’s rate limit per API key (also known as “authenticated rate limiting”):
User | Limit |
---|---|
Bob |
2 reqs / 10 secs (0.2 rps) |
Alice |
5 reqs / 10 secs (0.5 rps) |
kubectl -n toystore apply -f -<<EOF
---
apiVersion: kuadrant.io/v1beta1
kind: RateLimitPolicy
metadata:
name: toystore
spec:
targetRef:
group: gateway.networking.k8s.io
kind: HTTPRoute
name: toystore
rateLimits:
- configurations:
- actions:
- metadata:
descriptor_key: "userID"
default_value: "no-user"
metadata_key:
key: "envoy.filters.http.ext_authz"
path:
- segment:
key: "ext_auth_data"
- segment:
key: "userID"
limits:
- conditions:
- "userID == bob"
maxValue: 2
seconds: 10
variables: []
- conditions:
- "userID == alice"
maxValue: 5
seconds: 10
variables: []
EOF
Note that it may take up to a couple of minutes depending on your cluster to the RLP to be applied.
Only 2 requests out of every 10 are allowed for Bob:
while :; do curl --write-out '%{http_code}' --silent --output /dev/null -H 'Authorization: APIKEY IAMBOB' -X GET http://api.toystore.apps.$CLUSTER_DOMAIN/toy | egrep --color "\b(429)\b|$"; sleep 1; done
Only 5 requests out of every 10 are allowed for Alice:
while :; do curl --write-out '%{http_code}' --silent --output /dev/null -H 'Authorization: APIKEY IAMALICE' -X GET http://api.toystore.apps.$CLUSTER_DOMAIN/toy | egrep --color "\b(429)\b|$"; sleep 1; done
Thanks for giving Kuadrant a try!
Be sure to share any feedback or issues you might encounter through Github or Slack.
About the authors
More like this
Browse by channel
Automation
The latest on IT automation for tech, teams, and environments
Artificial intelligence
Updates on the platforms that free customers to run AI workloads anywhere
Open hybrid cloud
Explore how we build a more flexible future with hybrid cloud
Security
The latest on how we reduce risks across environments and technologies
Edge computing
Updates on the platforms that simplify operations at the edge
Infrastructure
The latest on the world’s leading enterprise Linux platform
Applications
Inside our solutions to the toughest application challenges
Original shows
Entertaining stories from the makers and leaders in enterprise tech
Products
- Red Hat Enterprise Linux
- Red Hat OpenShift
- Red Hat Ansible Automation Platform
- Cloud services
- See all products
Tools
- Training and certification
- My account
- Customer support
- Developer resources
- Find a partner
- Red Hat Ecosystem Catalog
- Red Hat value calculator
- Documentation
Try, buy, & sell
Communicate
About Red Hat
We’re the world’s leading provider of enterprise open source solutions—including Linux, cloud, container, and Kubernetes. We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.
Select a language
Red Hat legal and privacy links
- About Red Hat
- Jobs
- Events
- Locations
- Contact Red Hat
- Red Hat Blog
- Diversity, equity, and inclusion
- Cool Stuff Store
- Red Hat Summit