Using Nmap results to help harden Linux systems

System security is not a one-and-done task. Rather, there are numerous layers to an organization's approach to security. Some of those layers are physical security to the datacenters, regular patching and maintenance of the infrastructure, continuing user awareness education, and scanning systems for issues. This article discusses how to use the nmap and nc commands to scan a system so that you can determine the appropriate next steps. I use a few systems in my examples here. The system that does the scanning is my local Red Hat Enterprise Linux (RHEL) 8.3 computer, opendemo.usersys.redhat.com is the Red Hat Satellite 6.8 system used because it has several open ports, and I have various target systems.

[ You might also like: Sysadmin security: 8 Linux lockdown controls ]

Basic scans

To see the ports in use on my Satellite server, I SSH to the server and then use netstat:

[root@opendemo ~]# netstat -tlpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:27017         0.0.0.0:*               LISTEN      1443/mongod        
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      1197/redis-server 1
tcp        0      0 0.0.0.0:5646            0.0.0.0:*               LISTEN      1132/qdrouterd     
tcp        0      0 127.0.0.1:8751          0.0.0.0:*               LISTEN      1194/python        
tcp        0      0 0.0.0.0:5647            0.0.0.0:*               LISTEN      1132/qdrouterd     
tcp        0      0 127.0.0.1:19090         0.0.0.0:*               LISTEN      1237/cockpit-ws    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1175/sshd          
tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN      1242/postmaster    
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1396/master        
tcp        0      0 0.0.0.0:9090            0.0.0.0:*               LISTEN      1138/ruby          
tcp        0      0 127.0.0.1:45285         0.0.0.0:*               LISTEN      28650/Passenger Rack
tcp        0      0 127.0.0.1:5671          0.0.0.0:*               LISTEN      1140/qpidd         
tcp        0      0 0.0.0.0:8008            0.0.0.0:*               LISTEN      1240/ruby          
tcp        0      0 127.0.0.1:5672          0.0.0.0:*               LISTEN      1140/qpidd         
tcp6       0      0 :::8140                 :::*                    LISTEN      2101/java          
tcp6       0      0 127.0.0.1:61613         :::*                    LISTEN      1135/java          
tcp6       0      0 :::5646                 :::*                    LISTEN      1132/qdrouterd     
tcp6       0      0 :::5647                 :::*                    LISTEN      1132/qdrouterd     
tcp6       0      0 :::80                   :::*                    LISTEN      1131/httpd         
tcp6       0      0 :::22                   :::*                    LISTEN      1175/sshd          
tcp6       0      0 ::1:5432                :::*                    LISTEN      1242/postmaster    
tcp6       0      0 :::3128                 :::*                    LISTEN      1258/(squid-1)     
tcp6       0      0 ::1:25                  :::*                    LISTEN      1396/master        
tcp6       0      0 127.0.0.1:8443          :::*                    LISTEN      1135/java          
tcp6       0      0 :::443                  :::*                    LISTEN      1131/httpd         
tcp6       0      0 :::9090                 :::*                    LISTEN      1138/ruby          
tcp6       0      0 127.0.0.1:8005          :::*                    LISTEN      1135/java          
tcp6       0      0 ::1:5671                :::*                    LISTEN      1140/qpidd         
tcp6       0      0 :::8008                 :::*                    LISTEN      1240/ruby          
tcp6       0      0 ::1:5672                :::*                    LISTEN      1140/qpidd         
tcp6       0      0 :::5000                 :::*                    LISTEN      1131/httpd         
[root@opendemo ~]#

However, some of those are limited to the localhost, 127.0.0.1. To see which ports are publicly visible, I start by using a default nmap scan from my local system:

[pgervase@pgervase ~]$ nmap opendemo.usersys.redhat.com
Starting Nmap 7.70 ( https://nmap.org ) at 2021-01-07 20:28 EST
Nmap scan report for opendemo.usersys.redhat.com (10.19.47.240)
Host is up (0.041s latency).
Not shown: 993 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
80/tcp   open  http
443/tcp  open  https
3128/tcp open  squid-http
5000/tcp open  upnp
8008/tcp open  http
9090/tcp open  zeus-admin

Nmap done: 1 IP address (1 host up) scanned in 3.81 seconds
[pgervase@pgervase ~]$

This output shows that my local system can see fewer public ports than what I could see when I was SSH'd into the server. Some of those non-public ports are 25, which is used by master, and 8005, 8140, 8443, and 61613, which are used by java. Looking at ps output and grepping for the PID of master from that netstat output, master is the postfix mailer:

[root@opendemo ~]# ps auxww | grep 1396
root      1396  0.0  0.0  89740  2188 ?        Ss   Jan05   0:00 /usr/libexec/postfix/master -w
root     29665  0.0  0.0 112816   968 pts/0    R+   20:32   0:00 grep --color=auto 1396
[root@opendemo ~]#

That (master) is running locally so that mail can get sent to internal addresses but is not listening for incoming email, nor is it sending anything to any other host.

The other ports mentioned were for java. When you look at the netstat output, two different java processes are responsible for those ports:

[root@opendemo ~]# netstat -tlpn | grep java
tcp6       0      0 :::8140                 :::*                    LISTEN      2101/java          
tcp6       0      0 127.0.0.1:61613         :::*                    LISTEN      1135/java          
tcp6       0      0 127.0.0.1:8443          :::*                    LISTEN      1135/java          
tcp6       0      0 127.0.0.1:8005          :::*                    LISTEN      1135/java          
[root@opendemo ~]#

When you look at ps output for PID 1135, it’s used by tomcat:

[root@opendemo ~]# ps auxww | grep 1135
tomcat    1135  0.3  3.5 12409252 2165668 ?    Ssl  Jan05   9:25 /usr/lib/jvm/jre/bin/java -Xms1024m -Xmx4096m -Djava.security.auth.login.config=/usr/share/tomcat/conf/login.config -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start
root     31507  0.0  0.0 112816   968 pts/0    S+   20:53   0:00 grep --color=auto 1135
[root@opendemo ~]#

When I look in the/usr/share/tomcat/conf/server.xml file, it has content such as:

<Server port="8005" shutdown="SHUTDOWN">
...
    <Connector port="8443"
               address="localhost"
               protocol="HTTP/1.1"
               SSLEnabled="true"
               maxThreads="150" scheme="https" secure="true"
               clientAuth="want"
               sslProtocols="TLSv1.2"
               sslEnabledProtocols="TLSv1.2"
....

This shows how the ports that will get used are defined in the config file.

When I look at the other java process I mentioned, PID 2101 for port 8140, I see this is used by puppet:

[root@opendemo ~]# ps auxww | grep 2101
puppet    2101  0.2  2.5 9787492 1545188 ?     Sl   Jan05   7:14 /usr/bin/java -Xms2G -Xmx2G -Djruby.logger.class=com.puppetlabs.jruby_utils.jruby.Slf4jLogger -XX:OnOutOfMemoryError="kill -9 %p" -XX:ErrorFile=/var/log/puppetlabs/puppetserver/puppetserver_err_pid%p.log -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar:/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter.jar:/opt/puppetlabs/server/data/puppetserver/jars/* clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d --bootstrap-config /etc/puppetlabs/puppetserver/services.d/,/opt/puppetlabs/server/apps/puppetserver/config/services.d/ --restart-file /opt/puppetlabs/server/data/puppetserver/restartcounter
root     31696  0.0  0.0 112816   968 pts/0    S+   20:55   0:00 grep --color=auto 2101
[root@opendemo ~]#

Based on the netstat output, port 8140 should be visible to the public, but nmap from my local system didn't report it in its results. Here again, is the netstat output from the Satellite server:

[root@opendemo ~]# netstat -tunap| grep 8140
tcp6       0      0 :::8140                 :::*                    LISTEN      2101/java          
[root@opendemo ~]#

and the nmap from my local server:

[pgervase@pgervase ~]$ nmap opendemo.usersys.redhat.com | grep 8140
[pgervase@pgervase ~]$

However, I can force nmap to check on a specific port or range of ports:

[pgervase@pgervase ~]$ nmap -p 8140 opendemo.usersys.redhat.com
Starting Nmap 7.70 ( https://nmap.org ) at 2021-01-07 21:07 EST
Nmap scan report for opendemo.usersys.redhat.com (10.19.47.240)
Host is up (0.039s latency).

PORT     STATE SERVICE
8140/tcp open  puppet

Nmap done: 1 IP address (1 host up) scanned in 0.39 seconds
[pgervase@pgervase ~]$ nmap -p 8000-9000 opendemo.usersys.redhat.com
Starting Nmap 7.70 ( https://nmap.org ) at 2021-01-07 21:07 EST
Nmap scan report for opendemo.usersys.redhat.com (10.19.47.240)
Host is up (0.040s latency).
Not shown: 999 closed ports
PORT     STATE SERVICE
8008/tcp open  http
8140/tcp open  puppet

Nmap done: 1 IP address (1 host up) scanned in 2.12 seconds
[pgervase@pgervase ~]$

By forcing nmap to check those ports, I was able to see the :8140 port which a basic nmap scan did not report. This shows that a default nmap scan without extra arguments might be good enough for a first look at the system but might miss ports that are actually open.

This information is important in security testing so that sysadmins can identify potential vulnerabilities. From the nmap output scan, run locally on my system, you saw the ports that were publicly open. Prior versions of Satellite had tomcat configured so that some of those ports were public when that was not necessary. To read over some of the discussion for that issue, you can read the Bugzilla where this was resolved.

Verify certificates

Another issue that nmap can help with is to verify the certs being used on those various ports. Using nmap, you saw the open ports. Using those ports, you can use OpenSSL to see the cert used on the port. A number of those port are using self-signed certificates. To use nmap and OpenSSL together to check the ports on a remote system, you could do something like:

$ for port in `nmap -p 1-5000 opendemo.usersys.redhat.com | grep " open " | cut -d "/" -f 1`
> do echo checking on port: $port
> echo | openssl s_client -showcerts -connect opendemo.usersys.redhat.com:$port
> done &> opendemo.certs.txt.`date +%Y%m%d`

In my opendemo.certs.txt.20210127 file, it would have content like:

checking on port: 443
depth=1 C = US, ST = North Carolina, L = Raleigh, O = Katello, OU = SomeOrgUnit, CN = opendemo.usersys.redhat.com
verify return:1
depth=0 C = US, ST = North Carolina, O = Katello, OU = SomeOrgUnit, CN = opendemo.usersys.redhat.com
verify return:1
CONNECTED(00000003)
….
SSL handshake has read 3476 bytes and written 463 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported

Use that output file to verify that the certs in use are the correct TLS version.

If you use nc (or ncat), you might see more information than presented in the web UI. For this example, I used nc to connect to a web server:

$ nc 10.19.47.242 80
asdf
HTTP/1.1 400 Bad Request
Date: Sat, 09 Jan 2021 01:25:40 GMT
Server: Apache/2.4.37 (Red Hat Enterprise Linux)
Content-Length: 226
Connection: close
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
</p>
</body></html>

From that output, I can see the version of Apache that was installed. With that information, an attacker could learn what exploits the server was vulnerable to. Because of this, a web server should limit the amount of information that displayed:

[pgervase@pgervase ~]$ nc opendemo.usersys.redhat.com 443
GET / HTTP/1.1
HTTP/1.1 400 Bad Request
Date: Fri, 08 Jan 2021 02:33:08 GMT
Server: Apache
Content-Length: 362
Connection: close
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
Reason: You're speaking plain HTTP to an SSL-enabled server port.<br />
 Instead use the HTTPS scheme to access this URL, please.<br />
</p>
</body></html>

[pgervase@pgervase ~]$

Note that in this output, there is no version information for Apache.

In this next example, I use nc to connect to port 21 on my client system, which I can see is open:

[pgervase@pgervase ~]$ nmap 10.19.47.242
Starting Nmap 7.70 ( https://nmap.org ) at 2021-01-08 21:02 EST
Nmap scan report for 10.19.47.242
Host is up (0.039s latency).
Not shown: 996 closed ports
PORT    STATE SERVICE
21/tcp  open  ftp
22/tcp  open  ssh
80/tcp  open  http
111/tcp open  rpcbind

Nmap done: 1 IP address (1 host up) scanned in 0.83 seconds
[pgervase@pgervase ~]$ nc 10.19.47.242 21
220 (vsFTPd 3.0.3)

That 3.0.3 version is confirmed when I SSH to the system and use the rpm command:

[root@vulnerable ~]# rpm -q vsftpd
vsftpd-3.0.3-32.el8.x86_64
[root@vulnerable ~]# rpm -qi vsftpd
Name        : vsftpd
Version     : 3.0.3
Release     : 32.el8
<snipped>

Again, just like with learning the Apache version on the device, being able to do reconnaissance in your environment so that you know what a potential attacker can learn about your systems is important.

Scanning from Kali

In the next section, I show some results from scanning a system from a Kali server. In this example, I know that the target server has distccd running on port 3632, but, like earlier, nmap does not detect that port by default, and so I had to check for it specifically:

Now that you know distccd is open, you can use nmap's built-in capabilities to determine where it could potentially be exploited:

If you'd used only a plain nmap scan, you'd have missed that exploitable vulnerability. In my example, I ran uname -a on the remote system, but I could have run any command.

Identifying services

One final way to use nmap is with the -sV option, which probes the open ports and determines the service or version information. For this example, I changed the port that Apache runs on from 80 to 90 and then restarted the service. Below you can see the difference between a plain nmap scan and then using the -sV option, which correctly determined the service as httpd rather than dnsix:

[root@pgervase ~]# nmap 10.19.47.242

Starting Nmap 7.70 ( https://nmap.org ) at 2021-01-09 19:57 EST
Nmap scan report for 10.19.47.242
Host is up (0.043s latency).

Not shown: 996 closed ports

PORT    STATE SERVICE
21/tcp  open  ftp
22/tcp  open  ssh
90/tcp  open  dnsix
111/tcp open  rpcbind

Nmap done: 1 IP address (1 host up) scanned in 1.80 seconds

[root@pgervase ~]# nmap -sV 10.19.47.242
Starting Nmap 7.70 ( https://nmap.org ) at 2021-01-09 19:52 EST
Nmap scan report for 10.19.47.242
Host is up (0.040s latency).

Not shown: 996 closed ports

PORT    STATE SERVICE VERSION
21/tcp  open  ftp     vsftpd 3.0.3
22/tcp  open  ssh     OpenSSH 8.0 (protocol 2.0)
90/tcp  open  http    Apache httpd 2.4.37 ((Red Hat Enterprise Linux))
111/tcp open  rpcbind 2-4 (RPC #100000)
Service Info: OS: Unix

[ Want to learn more about security? Check out the IT security and compliance checklist. ] 

Wrap up

Now that you've been able to get a detailed report of what's running on your systems, what do you do next? The first thing is to be sure that there are no unexpected ports open. For this, verify with the applications team, security teams, and your coworkers might be appropriate. Next is to ensure that the exposed services are properly secured. This means taking steps such as making sure that all software is updated, updated ciphers are supported, insecure protocols are not in use, and default passwords for the services have been changed.

This article is an introduction to investigating your servers. Use nc and nmap to verify which ports are open and use the ps command to trace back the processes using those ports. I've also provided an example of how you can use nmap with the --script argument to test your systems. To continue on this learning path, one possible next step is to research using nmap as an attack engine by investigating the default scripts in /usr/share/nmap/scripts/.