DNS troubleshooting with record inspection

In the early days of the Internet, it was obvious there was a need for a way to identify devices easily. With the evolution of IP addressing and the steadily increasing number of things that might need to connect, an easier way to remember all of them seemed like a good idea. At first, someone had to have the job of writing all these names and numbers down and making sure everyone else knew what they were.

Eventually, this just became too big a job for one person, then a group of people, and then even a whole organization to manage on their own. Finally, in 1983, Request for Comment (RFC) 882 and 883 were issued by the IETF. In November 1987, RFC 1034[1] and RFC 1035[3] superseded the 1983 DNS specifications. Several additional RFCs have proposed extensions to the core DNS protocols since then if you are ever looking for some light reading on the topic, and the Domain Name System (DNS) was born. Several competing software solutions were considered, and eventually, a program created by some engineers from UC Berkeley called Internet Name Domain- which became known as Berkeley Internet Name Domain (BIND)- began to be implemented. BIND (with many successive improvements since then, of course) is to this day the most common application used to find other machines on the Internet.

This software and those RFCs form one of the most essential functions for Internet use: how do we find the needle of the machine we want to talk to in the haystack of the Internet.

A little review

Let's quickly review, Introduction to DNS troubleshooting, which is part one of this two-part DNS series. When we look for an address using a name, an application called a ‘resolver’ looks locally on your machine and tries to translate that to an address. If that fails, it goes to the next address(es) you’ve told it to check.

If that DNS server/resolver (it could be a caching resolver, or it might be able to do what we call recursion) doesn’t have the answer, it goes all the way to the top of the DNS hierarchy. It then systematically looks for an authoritative DNS server for each part of the address until it finds a one that has the answer.

When it breaks

As you can imagine, a number of things can go wrong along the way. The DNS system is designed in such a way that the resolution process shouldn’t go on too long or require too many referrals to be completed, but any number of delays or malfunctions can occur.

If you really want to understand how DNS works, I recommend you take a look at the O’Reilly book DNS and BIND, which is still considered an essential textbook for the subject. Also, take a look at the BIND manual on the ISC website. Better still, build a couple of DNS servers and see how they break.

Previously, we used the ping and host commands to do some quick checking and we talked about some of the things we can learn using dig.

The next level

Now we’re going to take a closer look at those tools. Let’s look at Red Hat's domain on the Internet and see what other information we can learn:

[root@showme1 ~]# host redhat.com
redhat.com has address 209.132.183.105
redhat.com mail is handled by 10 us-smtp-inbound-2.mimecast.com.
redhat.com mail is handled by 10 us-smtp-inbound-1.mimecast.com.

The main address (A record) for redhat.com is 209.132.183.105, and there are two other records. These look like MX (mail server) records. Let’s use dig to confirm that information:

[root@showme1 ~]# dig redhat.com -t mx +short
10 us-smtp-inbound-2.mimecast.com.
10 us-smtp-inbound-1.mimecast.com.

We use the -t to specify the type of record we want, and +short so dig shows us the results and not all the other stuff. So, for redhat.com, those are the addresses of the servers that handle mail for redhat.com.

NS records are the DNS nameservers for the domain:

[root@showme1 ~]# dig redhat.com -t NS +short
a16-67.akam.net.
a10-65.akam.net.
a13-66.akam.net.
a9-65.akam.net.
a1-68.akam.net.
a28-64.akam.net.

Getting the details

With this information, we could check that we can reach those servers to get addresses in the redhat domain using ping, for example. In this case, those servers happen to be caching/recursive DNS servers, which is a common way for large organizations to manage DNS.

We can also use the -t option to look at other specific records:

• TXT records are often used to insert information into DNS records that don’t have another specific format but are used by other services that interact with a domain.
• SRV records are for other commonly used services on the domain.
• CNAME records are alternate names for servers or endpoints. Usually, these are used to give a site a more friendly name or direct an old address to a new location.
• DS, CDSKEY, and CDS records are used in DNSSEC validation.
• AAAA is the IPv6 IP information for the A record. Not all sites have them yet, but they are being implemented more and more.

Other useful output:

• +trace shows us the delegation path of a query. We should be able to use this to find where the recursion is taking us, which would help if we have results coming up different from different locations or ISPs. This one’s actually kind of interesting, but the output is usually pretty long. You can see the query working it’s way down the tree from the root domain:

-    .            86400    IN    NS    c.root-servers.net.
-    .            86400    IN    NS    k.root-servers.net.

To the  .COM ‘tld’ name servers:

-    com.            172800    IN    NS    i.gtld-servers.net.
-    com.            172800    IN    NS    h.gtld-servers.net.

It then moves to the recursive layer servers in the Red Hat domain:

-    redhat.com.        172800    IN    NS    a13-66.akam.net.
-    redhat.com.        172800    IN    NS    a10-65.akam.net.

In the complete output, you can see all the servers in each group of servers, which can be up to a dozen or so. You can also see the RRSIG records, which show you the DNSSEC signatures used to validate that the source of the information can be trusted.

• +qr prints the query as it is sent. This can help give you an idea if there’s a problem with search domain configurations, for example.
• +multiline shows the long human-readable version of input. Sometimes this is helpful in understanding query results.

Wrap up

That’s a little more about understanding DNS and troubleshooting it. It’s an essential service for the way the Internet works. It’s a surprisingly complex and useful tool but we’ve only scratched the surface in these two articles. I encourage you to build your own DNS servers and experiment and learn more about how it works and how you can use these tools.

[ Want more for your network? Download a free ebook on network automation with Ansible. ]