[Freeipa-users] Can not post to list - email floats off into cyberspace

[nathan at nathanpeters.com]

We have a FreeIPA domain running IPA server 4.1.4 on CentOS 7.

We have no per zone forwarding enabled, only a single global forwarder.
This seems to work fine, but then after a while (several weeks I think)
will randomly stop working.

We had this issue several weeks ago on a different IPA domain (identical
setup) in our production network but it was ignored because a server
restart fixed it.

This issue then re-surfaced in our development domain today (different
network, different physical hardware, same OS and IPA versions).

I received a report today from a developer that he could not ping a
machine in another domain so I verified network connectivity and
everything was fine.  When I tried to resolve the name from the IPA dc
using ping it would fail, but nslookup directly to the forward server
worked fine.

ipactl showed no issues, and only after I restarted the server did the
lookups start working again.

Console log below :

Using username "myipausername".
Last login: Thu Oct  1 16:36:51 2015 from 10.5.5.57
[myipausername at dc1 ~]$ sudo su -
Last login: Tue Sep 29 19:03:39 UTC 2015 on pts/3

ATTEMPT FIRST PING TO UNRESOLVABLE HOST
=======================================
[root at dc1 ~]# ping artifactory.externaldomain.net
ping: unknown host artifactory.externaldomain.net

CHECK IPA STATUS
================
[root at dc1 ~]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
ipa_memcached Service: RUNNING
httpd Service: RUNNING
pki-tomcatd Service: RUNNING
smb Service: RUNNING
winbind Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful

ATTEMPT PING OF GLOBAL FORWARDER
================================
[root at dc1 ~]# ping 10.21.0.14
PING 10.21.0.14 (10.21.0.14) 56(84) bytes of data.
64 bytes from 10.21.0.14: icmp_seq=1 ttl=64 time=0.275 ms
64 bytes from 10.21.0.14: icmp_seq=2 ttl=64 time=0.327 ms
^C
--- 10.21.0.14 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.275/0.301/0.327/0.026 ms

MANUAL NSLOOKUP OF DOMAIN ON GLOBAL FORWARDER FROM IPA DC
=========================================================
[root at dc1 ~]# nslookup
> server 10.21.0.14
Default server: 10.21.0.14
Address: 10.21.0.14#53
> artifactory.externaldomain.net
Server:         10.21.0.14
Address:        10.21.0.14#53

Non-authoritative answer:
artifactory.externaldomain.net     canonical name =
van-artifactory1.externaldomain.net.
Name:   van-artifactory1.externaldomain.net
Address: 10.20.10.14

RE-ATTEMPT PING SINCE WE KNOW THAT NAME RESOLUTION (at least via nslookup
IS WORKING FROM THIS MACHINE
======================================================================================================
> ^C[root at dc1 ~]# ping artifactory.externaldomain.net
ping: unknown host artifactory.externaldomain.net
[root at dc1 ~]# ping van-artifactory1.externaldomain.net
ping: unknown host van-artifactory1.externaldomain.net

RESTART IPA SERVICES
====================
[root at dc1 ~]# ipactl restart
Restarting Directory Service
Restarting krb5kdc Service
Restarting kadmin Service
Restarting named Service
Restarting ipa_memcached Service
Restarting httpd Service
Restarting pki-tomcatd Service
Restarting smb Service
Restarting winbind Service
Restarting ipa-otpd Service
Restarting ipa-dnskeysyncd Service
ipa: INFO: The ipactl command was successful
[root at dc1 ~]# ipa dnsconfig-show
ipa: ERROR: did not receive Kerberos credentials
[root at dc1 ~]# kinit myipausername
Password for myipausername at ipadomain.NET:

OUTPUT GLOBAL FORWARDER CONFIG FOR TROUBLESHOOTING
==================================================
[root at dc1 ~]# ipa dnsconfig-show
  Global forwarders: 10.21.0.14
  Allow PTR sync: TRUE

PING NOW WORKS BECAUSE IPA SERVICES WERE RESTARTED
==================================================
[root at dc1 ~]# ping artifactory.externaldomain.net
PING van-artifactory1.externaldomain.net (10.20.10.14) 56(84) bytes of data.
64 bytes from 10.20.10.14: icmp_seq=1 ttl=60 time=3.00 ms
64 bytes from 10.20.10.14: icmp_seq=2 ttl=60 time=1.42 ms
64 bytes from 10.20.10.14: icmp_seq=3 ttl=60 time=2.39 ms
^C
--- van-artifactory1.externaldomain.net ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 1.420/2.274/3.004/0.653 ms
[root at dc1 ~]#

Here are some strange enties from my /var/log/messages relating to errors
from today :

Oct  1 20:39:31 dc1 named-pkcs11[15066]: checkhints: unable to get root NS
rrset from cache: not found
Oct  1 20:39:17 dc1 named-pkcs11[15066]: error (network unreachable)
resolving 'pmdb1.ipadomain.net/A/IN': 2001:500:2f::f#53
Oct  1 20:39:17 dc1 named-pkcs11[15066]: error (network unreachable)
resolving 'pmdb1.ipadomain.net/AAAA/IN': 2001:500:2f::f#53

Looking at the log entries, it appears that there may have been a network
connectivity 'blip' (maybe a switch or router was restarted) at some point
and even after connectivity was restored, the global forwarding was
failing because the "we can't contact our forwarder" status seemed to get
stuck in memory.