But, what about root passwords?

If you’ve walked long enough into your enterprise identity management journey you might reach this question: How will root passwords be managed? Having centralized user and group IDs, your access policies—Host Based Access Control (HBAC) and Role Based Access Control (RBAC)—in Red Hat Identity Management (IdM) or any similar solution might still leave root passwords unmanaged.

Consider the below PCI-DSS requirements for example. How would you apply that on root?

8.2.4 Change user passwords/passphrases at least once every 90 days.

8.2.6 Set passwords/passphrases for first-time use and upon reset to a unique value for each user, and change immediately after the first use.

In this post, we’ll cover some approaches to manage root passwords efficiently, on a large scale.

Privileged Access Management (PAM)

The concept of managing privileged accounts, such as root, is a thing among operating systems. There is a wide range of commercial products to help with that. Many of them are cross platform products—they help manage different types of operating systems.

Privileged accounts are those accounts you most definitely never want to lose control over.

The National Institute of Standards and Technology (NIST) issues regular publications around security standards, guidelines and recommended security controls. One of them, publication 800-53 (NIST 800-53) deals with Security and Privacy Controls for Information Systems and Organizations. We will not delve into 800-53 here, but if we summarize the strategy to manage privileged accounts we’ll get:

• Identify and classify: while root is the obvious privileged account, your organization might have other accounts to look after.

• Develop explicit policies for privileged accounts: Privileged accounts require a seperate set of policies. Clearly define these policies and the 5Ws (What, Why, When, Where, and Who) aspects of these policies. For example, who is allowed access to root, and under which circumstances (when).

• Monitor and record privileged accounts sessions: All activities on privileged accounts should be recorded to enforce proper behavior and in some situations, such as with PCI-DSS, it is a compliance requirement.

• Track and alert on user’s behavior: Since we’re monitoring, might as well get alerts on anomalies. Define your organization's baseline and set alerts for what’s out of the baseline.

• Principle of least privilege: Allow users to escalate privilege (sudo) only for the commands that they need to handle their day-to-day operations.

• Review and audit: Regularly review the generated reports for privileged accounts to identify anomalies. Once again, this might be a compliance requirement as well.

In this post we will be focusing only on managing root passwords, on a scale, with technologies available already within Red Hat Enterprise Linux. 

Public cloud versus on-premise

Public clouds’ approach is to lock privileged accounts like root and offload access to another user with ssh-key authentication. Below are examples from Amazon Elastic Compute Cloud (Amazon EC2), Google Cloud Platform (GCP), and Microsoft Azure

Amazon EC2

# grep ec2 /etc/shadow
 ec2-user:!!:18588:0:99999:7:::

# grep root /etc/shadow
 root:!!:18569:0:99999:7:::

# grep ec2-user /etc/sudoers.d/90-cloud-init-users
 # User rules for ec2-user
 ec2-user ALL=(ALL) NOPASSWD:ALL

# grep myuser /etc/shadow
 myuser:*:18588:0:99999:7:::

# grep root /etc/shadow
 root:*::0:99999:7:::

# cat /etc/sudoers.d/google_sudoers
 %google-sudoers ALL=(ALL:ALL) NOPASSWD:ALL

$ id
uid=1000(myuser) gid=1001(myuser)
groups=1001(myuser),4(adm),39(video),1000(google-sudoers)

Microsoft Azure

# grep myuser /etc/shadow
myuser:!!:18592:0:99999:7:::

# grep root /etc/shadow
root:*LOCK*:14600::::::

# grep myuser  /etc/sudoers.d/90-cloud-init-users
# User rules for myuser
myuser ALL=(ALL) NOPASSWD:ALL

Private clouds or on-premise installations, whether physical or virtual machines, on the other hand can have a multitude of possible setup variations, for example:

• SSH as root disabled, one or more accounts are set up with unrestricted sudo using password for authentication.

• SSH as root disabled, one or more accounts are set up with unrestricted sudo and use SSH keys or 2FA.

• SSH as root enabled from specific IPs only, with two-factor authentication or SSH keys and those keys reside only on the machines behind those IPs.

While there is a resemblance in some of these examples and the public cloud’s approach in  having no root password set, and shifting the privileged access to users other than root, there is one big difference. Many physical and virtual provisioning workflows for on-prem will include setting up a default root password for a variety of reasons, but those reasons are beyond the scope of this blog post.

Why is a root password still needed?

With proper sudo privileges in place, you might be left out with maintenance mode situations that need root password, for example when a server fails to boot properly. Those most likely fall into two categories:

• kernel/initramfs problems.

• /etc/fstab mount problems, like a missing or corrupt disk.

The kernel situations might be easier to handle via rebooting to a previous kernel, but for other situations like mounting disk problems, some manual intervention might still be needed and root password will be needed accordingly.

Let’s explore different approaches to manage root passwords, while keeping scale problems, like managing thousands of Linux machines, in mind.

Unknown root password

In this approach, the root password will be set to a random, long, unknown value after the initial provisioning workflow finishes, and proper sudo privileges are in place. An unknown password doesn’t need to be managed.

If root password is required, reset it via booting into single user mode or using rescue media. Then, randomize it again once the need is fulfilled.

However, this might not be efficient to tackle situations on a large scale. Imagine booting thousands of machines into single user mode, one by one, to reset root passwords. Not the best use of time and skills of a sysadmin.

Decouple

In this approach, we’ll get the maintenance mode to require a different password other than root password.

Create a drop-in file for emergency service.

# cp /usr/lib/systemd/system/emergency.service \
       /etc/systemd/system/emergency.service

Edit the drop-in file /etc/systemd/system/emergency.service, comment out ExecStartPre and ExecStart and add new lines like below.

#ExecStartPre=-/bin/echo -e 'Welcome to emergency mode! After logging in, type "journalctl -xb" to view\\nsystem logs, "systemctl reboot" to reboot, "systemctl default" or ^D to\\ntry again to boot into default mode.'
#ExecStart=-/bin/sh -c "/usr/sbin/sulogin; /usr/bin/systemctl --fail --no-block default"
ExecStartPre=-/bin/echo -e 'Enter emergency admin password to get shell'
ExecStart=-/bin/sh -c "/root/emergency.sh; /usr/bin/systemctl --fail --no-block default"

Reload systemctl daemon.

# systemctl daemon-reload

What we’ve done here is configure the emergency mode to run a file called /root/emergency.sh instead of the normal systemd sulogin.

Generate password hash to be used for the emergency hash $emergehash

# echo "mypass"|sha256sum

Now we add the /root/emergency.sh script.

while true; do
    read -s -p "Emergency Password: " pass
    hash=`/bin/echo "${pass}" | /usr/bin/sha256sum | \
        /bin/awk '{print $1}'`
    # echo "mypass"|sha256sum

emergehash="c206e94efcffe47a03694f92bc94a392cadb79ec0895e07b71e1e2275dea3463"

    if [ "${hash}" == "${emergehash}" ];then
    /bin/echo -e "\npass OK"
    exec /bin/bash
    else
    /bin/echo "wrong pass, try again"
    /bin/sleep 3
    fi
done

Set file permissions to protect its content.

# chmod 700 /root/emergency.sh

Now, if we test booting into emergency mode by editing kernel cmdline as below.

We are introduced with our emergency.sh script and it’s password instead.

What we’ve added here is a password useful only for the emergency shell and has no other use.

This approach can be used with both Red Hat Enterprise Linux (RHEL) 7 and RHEL 8.

Rotate root password and store it in IdM’s vault

There might still be situations that mandate setting the root password, and rotating it regularly. For that we can leverage IdM’s vault capability.

IdM vault is designed to be a secure location for storing, retrieving, and sharing secrets. We will use it to share a file generated by the script that contains root passwords, only among the group of users allowed.

We will use a script that rotates passwords concurrently on servers, assuming the following:

• The password rotation is conducted on a machine enrolled to IdM, and designated only for the purpose of password rotation, and has a minimal set of installed packages. In the steps below this is server gondor.example.com. This is to reduce the attack surface.

• The account used for password rotation (passadmin) is used only for that purpose. It is not used for daily admin tasks, and it has sudo privileges to run two commands only on all servers, /usr/sbin/chpasswd, and /usr/bin/timeout. This can be fulfilled centrally with IdM. Again, this is to reduce risk, if one of the admin accounts, used in daily tasks, is compromised, that will not affect the passadmin account.

• The vault will be shared among members of the user group sysadmins.

I can hear you thinking already, why a script, why not Ansible?

While working on this exercise, I found out that Ansible’s user module reveals the password hash in the running processes list (ps auxf), as seen in the image below. I've reported this issue upstream, but in the meantime I decided to write a script that does this in a discreet way instead.

The script does not fall under Red Hat’s support coverage.

 Let’s set up vault on IdM first, if it is not set up already.

[root@idm1 ~]# ipa-kra-install

As IdM admin, create a shared vault.

[root@idm1 ~]# ipa vault-add root_passwords --shared --type standard
----------------------------
Added vault "root_passwords"
----------------------------
  Vault name: root_passwords
  Type: standard
  Owner users: admin
  Shared vault: True

Add sysadmins to the vault.

[root@idm1 ~]# ipa vault-add-member root_passwords --shared --groups=sysadmins
  Vault name: root_passwords
  Type: standard
  Owner users: admin
  Shared vault: True
  Member groups: sysadmins
-------------------------
Number of members added 1
-------------------------

Now, on gondor.example.com we set up the script to be run by the sysadmins group.

[root@gondor ~]# wget https://raw.githubusercontent.com/anazmy/rotatepass/main/rotatepass.py -O /usr/local/bin/rotatepass.py
[root@gondor ~]# chown sysadmin1.sysadmins /usr/local/bin/rotatepass.py
[root@gondor ~]# chmod 750 /usr/local/bin/rotatepass.py

copy_file(sudo_user, host, helper_script)
cmd = """/usr/bin/ssh -o SendEnv=LC_NAME -o BatchMode=yes -o ConnectTimeout=%s -p %s %s@%s \
'export SUDO_ASKPASS=%s ;echo $LC_NAME|sudo -A timeout %s chpasswd -e' """ % (ssh_timeout,ssh_port, sudo_user, host, helper_script_path, fork_timeout)

$ kinit passadmin
Password for passadmin@EXAMPLE.COM:

$ cat serverlist
server1.example.com
Server2.example.com
[...]
server10.example.com

$ time rotatepass.py -i serverlist -p
Enter remote user sudo password:
Successfully reset root's password on server1.example.com
[...]
Successfully reset root's password on server10.example.com
All passwords changed successfully
Ensure secure handling of host_pass.enc!!

real    0m15.423s
user    0m0.441s
sys     0m0.254s

$ cat host_pass.enc
server1.example.com   3LoU6FkrWTbSns5tPm2AJCXw
server2.example.com   9KnaxZwjdZ2X8ptk72WZBB3e
[...]

$ kdestroy

$ kinit
Password for sysadmin1@EXAMPLE.COM:

$ ipa vault-archive root_passwords --shared --in host_pass.enc
-----------------------------------------
Archived data into vault "root_passwords"
-----------------------------------------

$ scrub -p dod -r host_pass.enc
scrub: using DoD 5220.22-M patterns
scrub: padding host_pass.enc with 3626 bytes to fill last fs block
scrub: scrubbing host_pass.enc 4096 bytes
scrub: random  |................................................|
scrub: 0x00 |................................................|
scrub: 0xff |................................................|
scrub: verify  |................................................|
scrub: unlinking host_pass.enc

$ klist
Ticket cache: KEYRING:persistent:410400005:410400005
Default principal: sysadmin2@EXAMPLE.COM
[..]

$ ipa vault-retrieve root_passwords --shared --out secret_exported.txt
------------------------------------------
Retrieved data from vault "root_passwords"
------------------------------------------
$ cat secret_exported.txt
server1.example.com   3LoU6FkrWTbSns5tPm2AJCXw
server2.example.com   9KnaxZwjdZ2X8ptk72WZBB3e
[...]