The chances are that you've seen references to hashes or checksums when you've downloaded software from the Internet. Often, the software will be displayed, and then near the link is a checksum. The checksum may be labeled as MD5, SHA, or with some other similar name. Here is an example using one of my favorite old games from the 1990s named Nethack:
Many people don't know exactly what this information means or how to work with it. In this article, I discuss the purpose of hashing, along with how to use it.
[ Readers also enjoyed: Getting started with GPG (GnuPG) ]
Goals of cryptography
In this first section, I want you to unlearn something. Specifically, I want you to break the association in your head between the word encryption and the word confidential. Many of us conceive of these two words as being synonymous when that is actually not the case. Cryptography, which includes encryption, can provide confidentiality, but it can also satisfy other goals.
Cryptography actually has three goals:
- Confidentiality - to keep the file content from being read by unauthorized users
- Authenticity - to prove where a file originated
- Integrity - to prove that a file has not changed unexpectedly
It is that third concept, integrity, that we are interested in here. In this context, integrity means to prove that data has not changed unexpectedly. Proving integrity is useful in many scenarios:
- Internet downloads such as Linux distributions, software, or data files
- Network file transfers via NFS, SSH, or other protocols
- Verifying software installations
- Comparing a stored value, such as a password, with a value entered by a user
- Backups that compare two files to see whether they've changed
What is hashing?
Cryptography uses hashing to confirm that a file is unchanged. The simple explanation is that the same hashing method is used on a file at each end of an Internet download. The file is hashed on the web server by the web administrator, and the hash result is published. A user downloads the file and applies the same hash method. The hash results, or checksums, are compared. If the checksum of the downloaded file is the same as that of the original file, then the two files are identical, and there have been no unexpected changes due to file corruption, man-in-the-middle attacks, etc.
Hashing is a one-way process. The hashed result cannot be reversed to expose the original data. The checksum is a string of output that is a set size. Technically, that means that hashing is not encryption because encryption is intended to be reversed (decrypted).
What kind of hash cryptography might you use with Linux?
Message Digest and Secure Hash Algorithm
In Linux, you're likely to interact with one of two hashing methods:
These cryptography tools are built into most Linux distributions, as well as macOS. Windows does not typically include these utilities, so you must download them separately from third party vendors if you wish to use this security technique. I think it's great that security tools such as these are part of Linux and macOS.
Message Digest versus Secure Hash Algorithm
What's the difference between the message digest and secure hash algorithms? The difference is in the mathematics involved, but the two accomplish similar goals. Sysadmins might prefer one over the other, but for most purposes, they function similarly. They are not, however, interchangeable. A hash generated with MD5 on one end of the connection will not be useful if SHA256 is used on the other end. The same hash method must be used on both sides.
SHA256 generates a bigger hash, and may take more time and computing power to complete. It is considered to be a more secure approach. MD5 is probably good enough for most basic integrity checks, such as file downloads.
Where do you find hashing in Linux?
Linux uses hashes in many places and situations. Checksums can be generated manually by the user. You'll see exactly how to do that later in the article. In addition, hash capabilities are included with
rsync, and other utilities.
For example, the passwords stored in the
/etc/shadow file are actually hashes. When you sign in to a Linux system, the authentication process compares the stored hash value against a hashed version of the password you typed in. If the two checksums are identical, then the original password and what you typed in are identical. In other words, you entered the correct password. This is determined, however, without ever actually decrypting the stored password on your system. Check the first two characters of the second field for your user account in
/etc/shadow. If the two characters are $1, your password is encrypted with MD5. If the characters are $5, your password is encrypted with SHA256. If the value is $6, SHA512 is being used. SHA512 is used on my Fedora 33 virtual machine, as seen below:
How to manually generate checksums
Using the hash utilities is very simple. I will walk you through a very easy scenario to accomplish on a lab computer or whatever Linux system you have available. The purpose of this scenario is to determine whether a file has changed.
First, open your favorite text editor and create a file named
original.txt with a line of text that reads: Original information.
[damon@localhost ~]$ vim original.txt [damon@localhost ~]$ cat original.txt Original information. [damon@localhost ~]$
Next, run the file through a hash algorithm. I'll use MD5 for now. The command is
md5sum. Here is an example:
[damon@localhost ~]$ md5sum original.txt 80bffb4ca7cc62662d951326714a71be original.txt [damon@localhost ~]$
Notice the resulting checksum value. This value is large enough that it's difficult to work with. Let's store that value for future use by redirecting it into a file:
[damon@localhost ~]$ md5sum original.txt > hashes.txt [damon@localhost ~]$ cat hashes.txt 80bffb4ca7cc62662d951326714a71be original.txt [damon@localhost ~]$
At this point, you have an original file. Copy that file to the
/tmp directory with the name
duplicate.txt. Copy the file by using the following command (be sure to copy, not move):
[damon@localhost ~]$ cp original.txt /tmp/duplicate.txt [damon@localhost ~]$
Run the following command to create a checksum of the copied file:
[damon@localhost ~]$ md5sum /tmp/duplicate.txt 80bffb4ca7cc62662d951326714a71be /tmp/duplicate.txt [damon@localhost ~]$
Next, append the hash result to our
hashes.txt file and then compare the two. Be very careful to use the
>> append redirect operator here, because
> will overwrite the hash value of the
Run the following command:
[damon@localhost ~]$ md5sum /tmp/duplicate.txt >> hashes.txt [damon@localhost ~]$ cat hashes.txt 80bffb4ca7cc62662d951326714a71be original.txt 80bffb4ca7cc62662d951326714a71be /tmp/duplicate.txt [damon@localhost ~]$
The two hash results are identical, so the file did not change during the copy process.
Next, simulate a change. Type the following command to change the
/tmp/duplicate.txt file contents, and then rerun the
md5sum command with the
>> append operator:
[damon@localhost ~]$ hostname >> /tmp/duplicate.txt [damon@localhost ~]$ md5sum /tmp/duplicate.txt >> hashes.txt [damon@localhost ~]$
You know that the
duplicate.txt file is no longer identical to the
original.txt file, but let's prove that:
[damon@localhost ~]$ cat hashes.txt 80bffb4ca7cc62662d951326714a71be original.txt 80bffb4ca7cc62662d951326714a71be /tmp/duplicate.txt 1f59bbdc4e80240e0159f09ecfe3954d /tmp/duplicate.txt [damon@localhost ~]$
The two checksum values are not identical, and therefore the two files from which the checksums were generated are not identical.
In the above example, you manually compared the hash values by displaying them with
cat. You can use the
--check option to have
md5sum do the comparison for us. I've included both methods below:
[damon@localhost ~]$ cat hashes.txt 80bffb4ca7cc62662d951326714a71be original.txt 80bffb4ca7cc62662d951326714a71be /tmp/duplicate.txt 1f59bbdc4e80240e0159f09ecfe3954d /tmp/duplicate.txt [damon@localhost ~]$ md5sum --check hashes.txt original.txt: OK /tmp/duplicate.txt: FAILED /tmp/duplicate.txt: OK md5sum: WARNING: 1 computed checksum did NOT match [damon@localhost ~]$
You can repeat the above steps substituting
sha256sum for the
md5sum command to see how the process works using the SHA algorithm. The
sha256sum command also includes a
--check checksum option that compares the resulting hashes and displays a message for whether the files differ.
Note: If you transfer files between Linux, macOS, and Windows, you can still use hashing to verify the files' integrity. To generate a hash value on macOS, run the
md5 command. To do this in Windows, you must download a third party program. Personally, I use md5checker. Be sure to understand licensing for these utilities. You may be able to use the PowerShell cmdlet
get-filehash, depending on the version of PowerShell you have installed.
[ Free course: Red Hat Satellite Technical Overview. ]
Hashing confirms that data has not unexpectedly changed during a file transfer, download, or other event. This concept is known as file integrity. Hashing does not tell you what changed, just that something changed. Once hashing tells you two files are different, you can use commands such as
diff to discover what differences exist.