Red Hat OpenShift Container Platform DNS deep dive: DNS changes in Red Hat OpenShift Container Platform 3.6

DNS in Red Hat OpenShift Container Platform?

Regarding Red Hat OpenShift Container Platform, there are two locations of DNS resolution to be concerned with:

• Corporate DNS (outside Red Hat OpenShift Container Platform) for Master internal/external hostname and router wildcard name.

• OpenShift DNS for communication between internal services.

In this blog, I plan to talk about the latter DNS, which is based on skyDNS.

For demonstration purpose, a simple application “hello openshift” will be used. This is the script to deploy it.

# Create DNS Demo Project
$ oc new-project dns-project

# Deploy hello world pod
$ curl https://raw.githubusercontent.com/openshift/origin/release-3.6/examples/hello-openshift/hello-pod.json|oc create -f -

# Check there is no svc
$ oc get svc
No resources found.

# Access the pod with Pod IP (Container IP - 10.x.x.x)
$ oc get pod -o wide
NAME             READY     STATUS    RESTARTS   AGE              IP                NODE
hello-openshift   1/1         Running             0            5s        10.130.2.13         XXXX

$ curl 10.130.2.13:8080
Hello OpenShift!

# Tip - use this command to use the Pod IP from your environment
$ curl $(oc get pod hello-openshift --template '{{ .status.podIP}}):8080
Hello OpenShift!

# Expose Service from Pod
$ oc expose pod hello-openshift 
service "hello-openshift" exposed

$  oc get svc
NAME              CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
hello-openshift   172.30.12.46   <none>        8080/TCP   5s

Internal DNS: Why do we need it?

In order to answer this question, I am going to use a simple example. When it comes to communication between applications in the VM world (e.g. Red Hat Virtualization, VMware,), each application uses the IP address of the guest, which typically does not change. However, in the container-centric world, the container gets a new IP address whenever it spins up.

Kubernetes, which orchestrates containers, needs a static IP for the container. The Kubernetes Service Object, which provides a Service IP, was created to do this, among other things. The Service Object has a hostname and an IP address. The hostname will not change, but the IP address will change if the Service Object is recreated. With this Service hostname, stable internal communication between pods (services) becomes possible.

Because the Service IP can change, we don’t want to specify a static IP address in the application or DeploymentConfig environment. What if the Service Object is recreated without the application team being notified? A new Service IP will be assigned to the service and the applications that use the Service IP will fail.

For these reasons, we normally use the Service hostname to avoid the use of a hardcoded IP address in the application. The internal DNS in Red Hat OpenShift Container Platform provides this functionality. It uses dynamic DNS, so whenever the Service Object is recreated, the DNS will be updated with new records. With this feature, each Service Object hostname (that has a specific format - refer to Red Hat OpenShift Container Platform DNS) has a unique and dynamic Service IP address. As a result, we recommend using the hostname to communicate between internal services.

Check Pod IP and Service IP/Hostname:

# Check Service IP
$ oc get svc
NAME       CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
hello-openshift   172.30.12.46     <none>        8080/TCP   7s

$  dig hello-openshift.dns-project.svc.cluster.local

;; QUESTION SECTION:
;hello-openshift.dns-project.svc.cluster.local. IN A

;; ANSWER SECTION:
hello-openshift.dns-project.svc.cluster.local. 30 IN A 172.30.12.46

;; Query time: 1 msec
;; SERVER: 10.10.181.97#53(10.10.181.97)
;; WHEN: Mon Oct 30 15:56:58 EDT 2017
;; MSG SIZE  rcvd: 79

# Access the pod with Service IP.
# hello openshift application can be accessible using service ip from another application like following.   
$ curl 172.30.12.46:8080
Hello OpenShift!


# Access the pod with Service Hostname

# Also, you can get the same result using service hostname.

$ curl hello-openshift.dns-project.svc.cluster.local:8080

Hello OpenShift!

Internal DNS: Where is it?

Internally, the DNS uses SkyDNS, which uses etcd. Since Red Hat OpenShift 3 was released, the internal DNS has changed twice, in Red Hat OpenShift Container Platform 3.2 and 3.6. Prior to 3.6, SkyDNS always ran on the master nodes (‘masters’), so pods in infrastructure/application nodes (‘nodes’) had to access one of the masters in order to resolve Service hostnames.

Main changes in 3.2

• Dnsmasq is installed by default on masters and nodes

• NetworkManager is required on masters and nodes

• SkyDNS listens on port 8053 on the masters

• All nodes connect to the masters on TCP/8053

• Dnsmasq routes queries for cluster.local to Kubernetes Service IP (172.30.0.1:53)

Note: OpenShift 3.2 - 3.5 has the DNS structure below

Main changes in Red Hat OpenShift Container Platform 3.6

• Dnsmasq was optional but now is mandatory.

• SkyDNS runs on masters and nodes

  • On masters, SkyDNS listens on port 8053 to avoid port conflict

  • On nodes, SkyDNS listens on port 53

• Dnsmasq routes queries for cluster.local and in-addr.arpa to 127.0.0.1:53

Note: OpenShift 3.6 has the DNS structure below

Deep Dive

Daemons and port mapping:

As the above diagram (Figure 3) shows, there are three DNS daemons on masters. Let’s check which process uses each DNS port. The following output shows detailed information.

# Show daemons that listen on *53
# netstat -tunlp|grep 53
tcp        0      0 0.0.0.0:8053            0.0.0.0:*               LISTEN      47645/openshift     
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      47678/openshift     
tcp        0      0 10.10.181.97:53       0.0.0.0:*              LISTEN      47644/dnsmasq       
….

# Show process names that use *53 ports
# ps -ef|grep openshift
root      47645      1  2 13:15 ?        00:02:08 /usr/bin/openshift start master api ...
root      47678      1  4 13:15 ?        00:03:17 /usr/bin/openshift start node ..

Dispatch Nameserver by Dnsmasq

Dnsmasq is still responsible for dispatching queries inside pods to the right nameserver. Until OpenShift 3.5, if it queried `cluster.local`, Dnsmasq was sending it to SkyDNS on master nodes, but now it goes to 127.0.01:53 because skyDNS is on all nodes in Red Hat OpenShift Container Platform 3.6. Plus, if it queries `in-addr,arpa`, it also returns the Service hostname. From dnsmasq configuration files, we can see that it forwards the queries that contain `cluster.local` and ` in-addr,arpa`  to 127.0.0.1:53.

# # Show all configuration about dnsmasq
# grep . /etc/dnsmasq.d/*
/etc/dnsmasq.d/node-dnsmasq.conf:server=/in-addr.arpa/127.0.0.1
/etc/dnsmasq.d/node-dnsmasq.conf:server=/cluster.local/127.0.0.1
/etc/dnsmasq.d/origin-dns.conf:no-resolv
/etc/dnsmasq.d/origin-dns.conf:domain-needed
/etc/dnsmasq.d/origin-dns.conf:no-negcache
/etc/dnsmasq.d/origin-dns.conf:max-cache-ttl=1
/etc/dnsmasq.d/origin-dns.conf:enable-dbus
/etc/dnsmasq.d/origin-dns.conf:bind-interfaces
/etc/dnsmasq.d/origin-dns.conf:listen-address=10.10.181.97 # default route interface IP
/etc/dnsmasq.d/origin-upstream-dns.conf:server=10.10.182.21 # original name server IP

# # Check if *.cluster.local can be resolved
# dig hello-openshift.dns-project.svc.cluster.local

;; QUESTION SECTION:
;hello-openshift.dns-project.svc.cluster.local. IN A

;; ANSWER SECTION:
hello-openshift.dns-project.svc.cluster.local. 30 IN A 172.30.12.46

;; Query time: 1 msec
;; SERVER: 10.10.181.97#53(10.10.181.97)

# # Check if *.in-addr.arpa  can be resolved
#  dig -x 172.30.12.46

;; QUESTION SECTION:
;46.12.30.172.in-addr.arpa. IN PTR

;; ANSWER SECTION:
46.12.30.172.in-addr.arpa. 30 IN PTR hello-openshift.dns-project.svc.cluster.local.

;; Query time: 1 msec
;; SERVER: 10.10.181.97#53(10.10.181.97)

NetworkManager with Dnsmasq

NetworkManager dispatcher 99-origin-dns.sh replicates the functionality of NetworkManager's dns=dnsmasq. With this script, it makes the default route IP to Dnsmasq listen IP. Container uses this IP as a default nameserver.

99-origin-dns.sh logs to journald on the unit NetworkManager-dispatcher

This dispatch script:

• creates dnsmasq conf files :

  • node-dnsmasq.conf

  • origin-dns.conf  

  • origin-upstream-dns.conf

• starts Dnsmasq daemon by default when NetworkManager start.

• sets host default route IP to Dnsmasq listen IP.

• updates /etc/resolv.conf using host default route IP.

• creates /etc/origin/node/resolv.conf

DNS query flow Condition in OpenShift 3.6

DNS query flow in OpenShift 3.6

Note: No need to reach the Master from the Node to get SVC DNS data

Debugging DNS Flow with tcpdump

Test env:

• Master Node

  • IP: 10.10.181.97

  • Hostname: dhcp181-97.gsslab.example.com

• App Node

  • IP: 10.10.181.196

  • Hostname: dhcp181-196.gsslab.example.com

• SkyDNS

  • IP: 127.0.0.1

• Upstream Nameserver

  • IP: 10.10.182.21

Executing tcpdump command fo

# On App Node 
$  tcpdump  -xx -vvvv  -s 0 -l -n -i any port 53 -w test.pcap

Scenario 1. Resolve Master hostname from one of app nodes

Node query will access Dnsmasq and forward it to upstream DNS.

# Command
$ dig dhcp181-97.gsslab.example.com

# Result
1 10.10.181.196   10.10.181.196  DNS  … dhcp181-97.gsslab.example.com OPT
2 10.10.181.196   10.10.182.21   DNS ... dhcp181-97.gsslab.example.com OPT
3 10.10.182.21    10.10.181.196   DNS ...  dhcp181-97.gsslab.example.com A 10.10.181.97 NS ns01.xxx.redhat.com NS ns02.xxx.redhat.com A x.x.x.x A x.x.x.x OPT
4 10.10.181.196   10.10.181.196   DNS dhcp181-97.gsslab.example.com A 10.10.181.97 NS ns01.xxx.redhat.com NS ns02.xxx.redhat.com A x.x.x.x A x.x.x.x OPT

Scenario 2. Resolve Service hostname from one of app nodes

Node query will access Dnsmasq and forward it to SkyDNS.

# Command 
$ dig  hello-openshift.dns-project.svc.cluster.local

#Result 

1 10.10.181.196 10.10.181.196DNS hello-openshift.dns-project.svc.cluster.local OPT
2 127.0.0.1            127.0.0.1     DNS hello-openshift.dns-project.svc.cluster.local OPT
3 127.0.0.1            127.0.0.1     DNS hello-openshift.dns-project.svc.cluster.local A 172.30.12.46
4 10.10.181.196 10.10.181.196DNS hello-openshift.dns-project.svc.cluster.local A 172.30.12.46

Scenario 3. Resolve Service hostname inside docker container from one of app nodes

Pod query will access Dnsmasq and forward it to SkyDNS.

# Command inside container
sh-4.2# ip a
   ….
   inet 10.131.2.17/23 scope global eth0
  ...

sh-4.2# dig  hello-openshift.dns-project.svc.cluster.local

# Result
1 10.131.2.17   10.10.181.196   DNS Standard query A hello-openshift.dns-project.svc.cluster.local OPT
2 10.131.2.17   10.10.181.196   DNS Standard query A hello-openshift.dns-project.svc.cluster.local OPT
3 127.0.0.1     127.0.0.1       DNS Standard query A hello-openshift.dns-project.svc.cluster.local OPT
4 127.0.0.1     127.0.0.1       DNS Standard query response A hello-openshift.dns-project.svc.cluster.local A 172.30.12.46
5 10.10.181.196 10.131.2.17     DNS Standard query response A hello-openshift.dns-project.svc.cluster.local A 172.30.12.46
6 10.10.181.196 10.131.2.17     DNS Standard query response A hello-openshift.dns-project.svc.cluster.local A 172.30.12.46

Reference:

• network_config.go

• https://wiki.archlinux.org/index.php/dnsmasq#NetworkManager

Interested in learning more about Red Hat OpenShift Container Platform? Join us at the Red Hat OpenShift Roadshow Event in London Jan. 18, 2018. Learn more and register here.

Jooho Lee is a senior OpenShift Technical Account Manager (TAM) in Toronto supporting middleware products(EAP/ DataGrid/ Web Server) and cloud products (Docker/ Kubernetes/ OpenShift/ Ansible). He is an active member of JBoss User Group Korea and Openshift / Ansible Group. Find more posts by Jooho at https://www.redhat.com/en/blog/authors/jooho-lee​

Connect with TAMs at a Red Hat Convergence event near you! Red Hat Convergence is a free, invitation-only event offering technical users an opportunity to deepen their Red Hat product knowledge and discover new ways to apply open source technology to meet their business goals. These events travel to cities around the world to provide you with a convenient, local one-day experience to learn and connect with Red Hat experts and industry peers.