Simple use of selectors to get pods to land on the desired nodes

In OpenShift testing (and production), it is useful to have pods land on specific sets of nodes for monitoring or isolation purposes.  There are discussions on how to do this in a few different locations such as:

https://blog.openshift.com/deploying-applications-to-specific-nodes/

https://docs.openshift.com/enterprise/3.2/admin_guide/managing_projects.html#using-node-selectors

This post will describe a simple practical example where

• The OpenShift router and docker-registry pods will go to a set of infrastructure nodes
• The OpenShift metrics pods will go to a second set of metrics nodes
• Applications will run on the remaining worker nodes

Some of this can be accomplished during the OpenShift V3 install by making use of the openshift_router_selector and openshift_registry_selector Ansible inventory parameters, but I will assume nothing was done during the install and that the cluster is up and running with the normal default (router and registry pods) and openshift-infra (metrics pods) projects.

Step 1:  Label your nodes

First we need to label sets of nodes which will run the desired types of OpenShift pods.  The overwrite parameter below handles the case where the node already has a region label

For each infrastructure node where the docker-registry or router pods will run:  oc label node <nodename> “region=infra” --overwrite

For each metrics node where the metrics pods will run:  oc label node <nodename> “region=metrics”

For each application node :  oc label node <nodename> “region=primary”

This labelling can also be done in the Ansible inventory at installation time.   Example:

[nodes]
ip-10-0-0-[1:3].us-west-2.compute.internal openshift_node_labels="{'region': 'infra', 'zone': 'default'}"
ip-10-0-0-[4:6].us-west-2.compute.internal openshift_node_labels="{'region': ‘metrics’, 'zone': 'default'}"
ip-10-0-0-21[0:9].us-west-2.compute.internal openshift_node_labels="{'region': 'primary', 'zone': 'default'}"

Step 2: Set a cluster default selector for normal application pods

To keep everyday applications off of the infra and metrics pods, set a default selector for the cluster:

Edit /etc/origin/master/master-config.yaml and find projectConfig.   Set the value:  defaultNodeSelector: "region=primary"

systemctl restart atomic-openshift-master (or atomic-openshift-master-api and atomic-openshift-master-controllers for HA clusters)

Step 3: Set node selectors for the special projects

We want pods in the default project to land on the region=infra nodes and pods in the openshift-infra project to land on the region=metrics nodes.  To do this, set selectors for these projects which override the cluster-wide default selector:

oc edit namespace default

Add the following annotation with the other annotations and save:

openshift.io/node-selector: region=infra

oc edit namespace openshift-infra

Add the following annotation with the other annotations and save:

openshift.io/node-selector: region=metrics

That should do it.  You can get a lot fancier with things like selectors at the deployment configuration level, using  combinations of selectors, or the  affinity/anti-affinity features of OpenShift.  But for many use cases where you need to know the basic set of nodes where a pod will land, this method works just fine.