Guide to Kubernetes Egress Network Policies

A few months ago, we published a guide to setting up Kubernetes network policies, which focused exclusively on ingress network policies. This follow-up post explains how to enhance your network policies to also control allowed egress.

A Brief Recap: What are Network Policies?

Network policies are used in Kubernetes to specify how groups of pods are allowed to communicate with each other and with external network endpoints. They can be thought of as the Kubernetes equivalent of a firewall. As with most Kubernetes objects, network policies are extremely flexible and powerful – if you know the exact communications patterns of the services in your application, you can use network policies to restrict communications to exactly what’s required and nothing more.

Ingress vs. Egress

Network policies can be used to specify both allowed ingress to pods and allowed egress from pods. These specifications work as one would expect:

• traffic to a pod from an external network endpoint outside the cluster is allowed if ingress from that endpoint is allowed to the pod.
• traffic from a pod to an external network endpoint outside the cluster is allowed if egress is allowed from the pod to that endpoint.
• traffic from one pod (A) to another (B) is allowed if and only if egress is allowed from A to B and ingress is allowed to B from A. Note that controls are unidirectional – for traffic from B to be allowed to initiate a connection to A, egress must be allowed from B to A and ingress to B from A.

Set Up Ingress First!

First things first - we recommend setting up network policies for ingress and operationalizing it successfully before setting up egress network policies.

Why? First, it is simpler to not do both at once, since otherwise it can be hard to know whether a network connection was blocked because of ingress or egress configurations. Second, and more important, egress network policies are typically harder to operationalize. Restricting egress often breaks apps in unexpected ways. While it is usually relatively straightforward to figure out from which network endpoints we expect communications to a pod, it is, in practice, usually much harder to figure out to which network endpoints connections from a pod go. This challenge arises because:

1. deployments often query a bunch of external services as part of regular functioning. Depending on how they handle timeouts when reaching out to these services (for example, whether it is best effort or they fail hard), functionality can be impacted in subtle and hard-to-observe ways.
2. deployments need to be able to talk to a DNS server to be able to talk to anything else, unless they are reaching out to services directly by IP.

To set up ingress policies, you can follow our aforementioned guide.

Isolate your pods for egress

Each network policy has a podSelector field, which selects a group of (zero or more) pods. When a pod is selected by a network policy, the network policy is said to apply to it.

Further, each network policy can apply to ingress, egress, or both, depending on the value of the policyTypes field (if this field is not specified in the YAML, its value defaults based on the presence of ingress and egress rules in the policy; since the defaulting logic is subtle, we recommend always specifying it explicitly).

In what follows, we will use the term “egress network policy” to denote any network policy that applies to egress (irrespective of whether the policy also applies to ingress).

By default, if no egress network policy applies to a pod, it is non-isolated for egress. (Note that isolation is evaluated independently for ingress and egress; it is possible for a pod to be isolated for neither, for exactly one, or for both). When a pod is non-isolated for egress, all network egress is allowed from the pod.

The moment one egress network policy applies to a pod, the pod is isolated for egress. For isolated pods, network egress is allowed only if it is permitted by at least one of the egress network policies that applies to it (that is, network policies are by allow list-only).

Therefore, the first step to setting up egress network policies is to isolate your pods for egress. We recommend starting out by applying a “default-deny-all” policy, which will isolate all pods for egress.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all-egress
spec:
  podSelector: {}
  egress:
  - to:
    ports:
    - protocol: TCP
      port: 53
    - protocol: UDP
      port: 53
  policyTypes:
  - Egress

Note that this policy allows connections to port 53 on any IP by default, to facilitate DNS lookups. As a consequence, while it guards against accidental unintended egress, it is not a general protection against data exfiltration since attackers can just exfil to port 53.

If you know, for certain, what DNS servers your pods will use, you can scope down this access further.

For example, to narrow DNS down to the kube-dns service only, you can do the following:

1. Label the kube-system namespace: kubectl label namespace kube-system networking/namespace=kube-system
2. Apply the following network policy:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all-egress
spec:
  podSelector: {}
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          networking/namespace: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: TCP
      port: 53
    - protocol: UDP
      port: 53
  policyTypes:
  - Egress

Important Note: Since network policies are namespaced resources, you will need to create this policy for each namespace. You can do so by running

 kubectl -n <namespace> create -f <filename> 

for each namespace. Do not apply this on the kube-system namespace unless you know what you’re doing, since it can break cluster functionality.

Explicitly allow Internet egress for pods that need it

With just the default-deny-all-egress policy in place in every namespace, none of your pods will be able to reach out to the Internet, but in most applications, at least some pods will need to. To permit this setup, one approach would be to designate labels that are applied to those pods for which Internet egress is allowed, and to create a network policy that targets those labels. For example, the following network policy allows traffic from pods having the networking/allow-internet-egress=true label to all network endpoints (including those external to the cluster). Note that again, as in the previous section, you will have to create this policy for every namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: internet-egress
spec:
  podSelector:
    matchLabels:
      networking/allow-internet-egress: "true"
  egress:
  - {}
  policyTypes:
  - Egress

For a more locked-down set of policies, you would ideally want to specify finer-grained CIDR blocks as well as explicitly list out allowed ports and protocols. However, this policy provides a good starting point, with much greater security than the default.

Explicitly allow necessary pod-to-pod communications

If you isolated your pods for ingress, and then explicitly allowed pod-to-pod communications until your app worked, you probably found that all such communications was blocked the moment you applied the default-deny-all-egress policy. This behavior is expected: every connection that you allow in the ingress direction, you will now need to add to the allow list in the egress direction as well. Whether you followed one of our suggestions (such as allowing all intra-namespace communications, using a source-sink approach, or adding connections between individual pods to an allow list), or went with a custom approach, simply follow the below algorithm for every ingress policy you constructed to construct corresponding egress policies.

Constructing the complementary egress policy

For an ingress policy that allows communications from one group of pods to the other, the complementary egress policy is fairly straightforward to construct. First, change the policyTypes field to be an array containing only Egress. Take the spec.podSelector, and put it inside a spec.egress.to.podSelector block. Remove the ingress.from section, but take the ingress.from.podSelector from there and make it the spec.podSelector of the new egress policy. If the ingress policy was an intra-namespace policy, you’re done!

For a cross-namespace policy, assuming that you have labeled each namespace with the network/namespace: <ns-name> label as we suggested in our ingress guide (recall that you can do so by running

kubectl label namespace <name> networking/namespace=<name>

) you need to put the namespace selected in ingress.from.namespaceSelector in the ingress policy as metadata.namespace of the egress policy and select the namespace specified in the metadata.namespace of the ingress policy in the egress policy’s egress.to.namespaceSelector field.

Here’s an example:

Summary

Just as we noted in our ingress post, these recommendations provide a good starting point, but network policies are much more complicated. If you’re interested in exploring them in more detail, be sure to check out the Kubernetes tutorial as well as some handy network policy recipes.

Here at Red Hat, we’ve spent a lot of time thinking about how to operationalize network policies. The StackRox Kubernetes Security Platform automatically suggests and can generate network policies that enable just those communications paths your applications need. You can learn more about these capabilities in our network policy enforcement discussion.