Anmelden / Registrieren Konto

If you’ve walked long enough into your enterprise identity management journey you might reach this question: How will root passwords be managed? Having centralized user and group IDs, your access policies—Host Based Access Control (HBAC) and Role Based Access Control (RBAC)—in Red Hat Identity Management (IdM) or any similar solution might still leave root passwords unmanaged.

Consider the below PCI-DSS requirements for example. How would you apply that on root?

8.2.4 Change user passwords/passphrases at least once every 90 days.

8.2.6 Set passwords/passphrases for first-time use and upon reset to a unique value for each user, and change immediately after the first use.

In this post, we’ll cover some approaches to manage root passwords efficiently, on a large scale.

Privileged Access Management (PAM)

The concept of managing privileged accounts, such as root, is a thing among operating systems. There is a wide range of commercial products to help with that. Many of them are cross platform products—they help manage different types of operating systems.

Privileged accounts are those accounts you most definitely never want to lose control over.

The National Institute of Standards and Technology (NIST) issues regular publications around security standards, guidelines and recommended security controls. One of them, publication 800-53 (NIST 800-53) deals with Security and Privacy Controls for Information Systems and Organizations. We will not delve into 800-53 here, but if we summarize the strategy to manage privileged accounts we’ll get:

  • Identify and classify: while root is the obvious privileged account, your organization might have other accounts to look after.

  • Develop explicit policies for privileged accounts: Privileged accounts require a seperate set of policies. Clearly define these policies and the 5Ws (What, Why, When, Where, and Who) aspects of these policies. For example, who is allowed access to root, and under which circumstances (when).

  • Monitor and record privileged accounts sessions: All activities on privileged accounts should be recorded to enforce proper behavior and in some situations, such as with PCI-DSS, it is a compliance requirement.

  • Track and alert on user’s behavior: Since we’re monitoring, might as well get alerts on anomalies. Define your organization's baseline and set alerts for what’s out of the baseline.

  • Principle of least privilege: Allow users to escalate privilege (sudo) only for the commands that they need to handle their day-to-day operations.

  • Review and audit: Regularly review the generated reports for privileged accounts to identify anomalies. Once again, this might be a compliance requirement as well.

In this post we will be focusing only on managing root passwords, on a scale, with technologies available already within Red Hat Enterprise Linux. 

Public cloud versus on-premise

Public clouds’ approach is to lock privileged accounts like root and offload access to another user with ssh-key authentication. Below are examples from Amazon Elastic Compute Cloud (Amazon EC2), Google Cloud Platform (GCP), and Microsoft Azure

Amazon EC2

# grep ec2 /etc/shadow
 ec2-user:!!:18588:0:99999:7:::

# grep root /etc/shadow
 root:!!:18569:0:99999:7:::

# grep ec2-user /etc/sudoers.d/90-cloud-init-users
 # User rules for ec2-user
 ec2-user ALL=(ALL) NOPASSWD:ALL

GCP

# grep myuser /etc/shadow
 myuser:*:18588:0:99999:7:::

# grep root /etc/shadow
 root:*::0:99999:7:::

# cat /etc/sudoers.d/google_sudoers
 %google-sudoers ALL=(ALL:ALL) NOPASSWD:ALL

$ id
uid=1000(myuser) gid=1001(myuser)
groups=1001(myuser),4(adm),39(video),1000(google-sudoers)

Microsoft Azure

# grep myuser /etc/shadow
myuser:!!:18592:0:99999:7:::

# grep root /etc/shadow
root:*LOCK*:14600::::::

# grep myuser  /etc/sudoers.d/90-cloud-init-users
# User rules for myuser
myuser ALL=(ALL) NOPASSWD:ALL
In the above, both "myuser" and "ec2-user" have unrestricted sudo privileges, and both have  a private SSH key setup. 

Private clouds or on-premise installations, whether physical or virtual machines, on the other hand can have a multitude of possible setup variations, for example:

  • SSH as root disabled, one or more accounts are set up with unrestricted sudo using password for authentication.

  • SSH as root disabled, one or more accounts are set up with unrestricted sudo and use SSH keys or 2FA.

  • SSH as root enabled from specific IPs only, with two-factor authentication or SSH keys and those keys reside only on the machines behind those IPs.

While there is a resemblance in some of these examples and the public cloud’s approach in  having no root password set, and shifting the privileged access to users other than root, there is one big difference. Many physical and virtual provisioning workflows for on-prem will include setting up a default root password for a variety of reasons, but those reasons are beyond the scope of this blog post.

Why is a root password still needed?

With proper sudo privileges in place, you might be left out with maintenance mode situations that need root password, for example when a server fails to boot properly. Those most likely fall into two categories:

  • kernel/initramfs problems.

  • /etc/fstab mount problems, like a missing or corrupt disk.

The kernel situations might be easier to handle via rebooting to a previous kernel, but for other situations like mounting disk problems, some manual intervention might still be needed and root password will be needed accordingly.

Let’s explore different approaches to manage root passwords, while keeping scale problems, like managing thousands of Linux machines, in mind.

Unknown root password

In this approach, the root password will be set to a random, long, unknown value after the initial provisioning workflow finishes, and proper sudo privileges are in place. An unknown password doesn’t need to be managed.

If root password is required, reset it via booting into single user mode or using rescue media. Then, randomize it again once the need is fulfilled.

However, this might not be efficient to tackle situations on a large scale. Imagine booting thousands of machines into single user mode, one by one, to reset root passwords. Not the best use of time and skills of a sysadmin.

Decouple

In this approach, we’ll get the maintenance mode to require a different password other than root password.

Create a drop-in file for emergency service.

# cp /usr/lib/systemd/system/emergency.service \
       /etc/systemd/system/emergency.service

Edit the drop-in file /etc/systemd/system/emergency.service, comment out ExecStartPre and ExecStart and add new lines like below.

#ExecStartPre=-/bin/echo -e 'Welcome to emergency mode! After logging in, type "journalctl -xb" to view\\nsystem logs, "systemctl reboot" to reboot, "systemctl default" or ^D to\\ntry again to boot into default mode.'
#ExecStart=-/bin/sh -c "/usr/sbin/sulogin; /usr/bin/systemctl --fail --no-block default"
ExecStartPre=-/bin/echo -e 'Enter emergency admin password to get shell'
ExecStart=-/bin/sh -c "/root/emergency.sh; /usr/bin/systemctl --fail --no-block default"

Reload systemctl daemon.

# systemctl daemon-reload

What we’ve done here is configure the emergency mode to run a file called /root/emergency.sh instead of the normal systemd sulogin.

Generate password hash to be used for the emergency hash $emergehash

# echo "mypass"|sha256sum

Now we add the /root/emergency.sh script.

while true; do
    read -s -p "Emergency Password: " pass
    hash=`/bin/echo "${pass}" | /usr/bin/sha256sum | \
        /bin/awk '{print $1}'`
    # echo "mypass"|sha256sum

emergehash="c206e94efcffe47a03694f92bc94a392cadb79ec0895e07b71e1e2275dea3463"

    if [ "${hash}" == "${emergehash}" ];then
    /bin/echo -e "\npass OK"
    exec /bin/bash
    else
    /bin/echo "wrong pass, try again"
    /bin/sleep 3
    fi
done

Note: Make sure to use something other than mypass for  $emergehash

Set file permissions to protect its content.

# chmod 700 /root/emergency.sh

Now, if we test booting into emergency mode by editing kernel cmdline as below.
Root Passwords 1

We are introduced with our emergency.sh script and it’s password instead.

Root Passwords 2

What we’ve added here is a password useful only for the emergency shell and has no other use.

This approach can be used with both Red Hat Enterprise Linux (RHEL) 7 and RHEL 8.

Rotate root password and store it in IdM’s vault

There might still be situations that mandate setting the root password, and rotating it regularly. For that we can leverage IdM’s vault capability.

IdM vault is designed to be a secure location for storing, retrieving, and sharing secrets. We will use it to share a file generated by the script that contains root passwords, only among the group of users allowed.

We will use a script that rotates passwords concurrently on servers, assuming the following:

  • The password rotation is conducted on a machine enrolled to IdM, and designated only for the purpose of password rotation, and has a minimal set of installed packages. In the steps below this is server gondor.example.com. This is to reduce the attack surface.

  • The account used for password rotation (passadmin) is used only for that purpose. It is not used for daily admin tasks, and it has sudo privileges to run two commands only on all servers, /usr/sbin/chpasswd, and /usr/bin/timeout. This can be fulfilled centrally with IdM. Again, this is to reduce risk, if one of the admin accounts, used in daily tasks, is compromised, that will not affect the passadmin account.

  • The vault will be shared among members of the user group sysadmins.

I can hear you thinking already, why a script, why not Ansible?

While working on this exercise, I found out that Ansible’s user module reveals the password hash in the running processes list (ps auxf), as seen in the image below. I've reported this issue upstream, but in the meantime I decided to write a script that does this in a discreet way instead.

The script does not fall under Red Hat’s support coverage.

Root Passwords 3 Let’s set up vault on IdM first, if it is not set up already.

[root@idm1 ~]# ipa-kra-install

As IdM admin, create a shared vault.

[root@idm1 ~]# ipa vault-add root_passwords --shared --type standard
----------------------------
Added vault "root_passwords"
----------------------------
  Vault name: root_passwords
  Type: standard
  Owner users: admin
  Shared vault: True

Add sysadmins to the vault.

[root@idm1 ~]# ipa vault-add-member root_passwords --shared --groups=sysadmins
  Vault name: root_passwords
  Type: standard
  Owner users: admin
  Shared vault: True
  Member groups: sysadmins
-------------------------
Number of members added 1
-------------------------

Now, on gondor.example.com we set up the script to be run by the sysadmins group.

[root@gondor ~]# wget https://raw.githubusercontent.com/anazmy/rotatepass/main/rotatepass.py -O /usr/local/bin/rotatepass.py
[root@gondor ~]# chown sysadmin1.sysadmins /usr/local/bin/rotatepass.py
[root@gondor ~]# chmod 750 /usr/local/bin/rotatepass.py

Edit the script, setting variables like sudo_user: passadmin.

What does the script actually do and how is it different from Ansible?

Here is a brief of the script’s logic:

  • For every host create a unique random password.

  • Hash the password.

  • Assign "root:hashed_password" to environment variable LC_NAME.

  • On the remote host, write a helper script with a random name that echoes back sudo password and self deletes.

  • ssh to remote host as passadmin and pass along the LC_NAME variable.

  • Assign the variable SUDO_ASKPASS on the remote host to the helper script path, so sudo runs it to get sudo password, which deletes the file afterwards.

  • The script uses Linux pipes to send the password hash to chpasswd command so it is not visible in the process list.

  • Update root password on file in case of success, otherwise leave it as is.

The below two lines show the heart of the code.

copy_file(sudo_user, host, helper_script)
cmd = """/usr/bin/ssh -o SendEnv=LC_NAME -o BatchMode=yes -o ConnectTimeout=%s -p %s %s@%s \
'export SUDO_ASKPASS=%s ;echo $LC_NAME|sudo -A timeout %s chpasswd -e' """ % (ssh_timeout,ssh_port, sudo_user, host, helper_script_path, fork_timeout)

Now, let's see the script in action.

As user sysadmin1, we obtain a passadmin kerberos ticket and run the script.

$ kinit passadmin
Password for passadmin@EXAMPLE.COM:

$ cat serverlist
server1.example.com
Server2.example.com
[...]
server10.example.com

$ time rotatepass.py -i serverlist -p
Enter remote user sudo password:
Successfully reset root's password on server1.example.com
[...]
Successfully reset root's password on server10.example.com
All passwords changed successfully
Ensure secure handling of host_pass.enc!!

real    0m15.423s
user    0m0.441s
sys     0m0.254s

$ cat host_pass.enc
server1.example.com   3LoU6FkrWTbSns5tPm2AJCXw
server2.example.com   9KnaxZwjdZ2X8ptk72WZBB3e
[...]

Note: You can use ssh-keys instead of kerberos, if needed.

In about 15 seconds, we’ve reset root passwords on 10 machines and saved them in a plain text file (since we used (-p) argument).

As user sysadmin1, we store the file in IdM vault.

$ kdestroy

$ kinit
Password for sysadmin1@EXAMPLE.COM:

$ ipa vault-archive root_passwords --shared --in host_pass.enc
-----------------------------------------
Archived data into vault "root_passwords"
-----------------------------------------

The data stored in IdM’s vault is encrypted at rest—at the time of writing—with AES256-CBC encryption.

And now, scrub the file from disk.

$ scrub -p dod -r host_pass.enc
scrub: using DoD 5220.22-M patterns
scrub: padding host_pass.enc with 3626 bytes to fill last fs block
scrub: scrubbing host_pass.enc 4096 bytes
scrub: random  |................................................|
scrub: 0x00 |................................................|
scrub: 0xff |................................................|
scrub: verify  |................................................|
scrub: unlinking host_pass.enc

Later on, if another member of  sysadmins group needs the file, they can retrieve it.

$ klist
Ticket cache: KEYRING:persistent:410400005:410400005
Default principal: sysadmin2@EXAMPLE.COM
[..]

$ ipa vault-retrieve root_passwords --shared --out secret_exported.txt
------------------------------------------
Retrieved data from vault "root_passwords"
------------------------------------------
$ cat secret_exported.txt
server1.example.com   3LoU6FkrWTbSns5tPm2AJCXw
server2.example.com   9KnaxZwjdZ2X8ptk72WZBB3e
[...]

Final thoughts

While these scenarios do not fulfill all requirements, such as a check-in and check-out facility for passwords, they still cover many sides of privileged account management with existing capabilities within RHEL and without the need for a separate piece of software.  


About the author

Ahmed Nazmy is a Principal Technical Account Manager (TAM) in the EMEA region. He has expertise in various industry domains like Security, Automation and Scalability. Ahmed has been a Linux geek since the late 90s, having spent time in the webhosting industry prior to joining Red Hat.

Read full bio