Linux sysadmin basics: Troubleshooting known_hosts failures

SSH is easy to use and is generally trouble-free. But, there's this little file in your home directory that can become a rare pain point for this ubiquitous protocol and tool. The offending entity is the ~/.ssh/known_hosts file. If you didn't know about it, you should acquaint yourself now. This file will come up at some point in your career.

If you've ever seen the following message, you either can freak out that someone is attempting to hack you, or you can realize that DHCP or some other anomaly has done a dirt cheap dirty deed on you:

$ ssh 192.168.1.84

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:I02UyJs2vS0ym4jWn5upAWZDqwu5RjMg4aM9hPq8G1k.
Please contact your system administrator.
Add correct host key in /Users/khess/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /Users/khess/.ssh/known_hosts:4
ECDSA host key for 192.168.1.84 has changed and you have requested strict checking.
Host key verification failed.

What has happened here is that you've attempted to connect to a system that no longer has the IP address 192.168.1.84. A different system has that IP address and SSH is complaining that you might be getting hacked. The reality is often much less exciting. Your known_hosts file may not sync with actual host reality. 

Multiple things can cause this IP address musical chairs game. One of the systems could have been down for an extended period of time and when another's lease was up, it grabbed the next available IP address. A patching event and subsequent reboot caused some of your systems to acquire different IP addresses. The system could have been reinstalled and a new identity generated, or perhaps someone wised up and gave static IP addresses to your server systems. (It's my opinion that all servers should have static IP addresses. The same goes for any support components, such as routers, switches, access points, IoT devices, cameras, conference room systems, and even printers. Only end-user devices should have the dubious privilege of acquiring random IP addresses. I have my reasons for this strong opinion, which I'll share in another article later.)

Regardless of how it happened, the system you were interested in connecting to has a new IP address. One that is also known to your system, but with a different fingerprint. 

The aforementioned known_hosts file is a simple text file that lives inside a hidden directory (.ssh) in your home directory. To view its contents, enter the following command:

$ cat .ssh/known_hosts

192.168.1.97 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMJWpXf8Ou/XJ1Q5YcIS5Me4GgN1F7AYHbvIPayfGqwd90kvhQygrPCYEqVdw44lEOIZe+DEKP4F6Otdt38yf68=

192.168.1.101 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJRNTZm8M9IYYN3uJL/yCPv4EJRMDZkkcnt1CY8L6OR84wstYdhqtNqV0v/LQiz/AoRbHKxPTdjXfIOrc1vYDHc=

ken,192.168.1.69 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB53lh2RyFiMfJFKBkLa643lS4GGILG6oVGq33KyBSgaDU2ZHryYak1FU1HcPa6Xb/xYqEUugv9cOXFRwajbrmg=

192.168.1.84 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMGjMsJPX4bfdEnVy59Uf2VhH1oAt7Zemd5bfSoSGxX69HBcjkekO/LkEZUlhVplBlHFqJBqs2gWYp3zVDCTwr4=

192.168.1.64 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIkK9WylcZcgjT7gUz9DMLehUBApION1CLiBVc3Pa/YfAbIDUlIWta12zL5B12RmBXgoYvntdW5rcowPiqjL0/o=

The easy solution to this problem is to remove the known_hosts file and allow new keys to be generated for each host you connect to. For this demonstration, I remove the 192.168.1.84 entry from the known_hosts file and then connect to the new host at that address, so you can see a comparison of what happens in the process. First, remove a known_hosts entry with the ssh-keygen command:

$ ssh-keygen -R 192.168.1.84 -f .ssh/known_hosts

 Host 192.168.1.84 found: line 4
.ssh/known_hosts updated.
Original contents retained as .ssh/known_hosts.old

The original entry is saved to .ssh/known_hosts.old in case it's needed in the future.

After removing the 192.168.1.84 entry from the  known_hosts file, I attempted the connection again to host 192.168.1.84:

$ ssh 192.168.1.84
The authenticity of host '192.168.1.84 (192.168.1.84)' can't be established.
ECDSA key fingerprint is SHA256:I02UyJs2vS0ym4jWn5upAWZDqwu5RjMg4aM9hPq8G1k.
Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added '192.168.1.84' (ECDSA) to the list of known hosts.

You can see that the key I added now matches the one listed in the error at the beginning of the article. To verify a host's key, issue the following command on the remote host:

$ ssh-keygen -lf /etc/ssh/ssh_host_ecdsa_key.pub
256 SHA256:I02UyJs2vS0ym4jWn5upAWZDqwu5RjMg4aM9hPq8G1k no comment (ECDSA)

Wrapping up

Unless you use static IP addresses and your systems never change, you'll have this problem at some point in your career. It happens to me all of the time because I create and destroy virtual machines several times a month. In fact, it happens so often that by now I should have a script that removes the known_hosts file at the beginning of every month.

Ideally, you would never have to remove an entry from the known_hosts file (or the entire file). But if you have to, you've just learned how to do it.

Takeaways: known_hosts, ssh-keygen

[Want to try out Red Hat Enterprise Linux? Download it now for free.]