Using Nmap results to help harden Linux systems
System security is not a one-and-done task. Rather, there are numerous layers to an organization's approach to security. Some of those layers are physical security to the datacenters, regular patching and maintenance of the infrastructure, continuing user awareness education, and scanning systems for issues. This article discusses how to use the
nc commands to scan a system so that you can determine the appropriate next steps. I use a few systems in my examples here. The system that does the scanning is my local Red Hat Enterprise Linux (RHEL) 8.3 computer,
opendemo.usersys.redhat.com is the Red Hat Satellite 6.8 system used because it has several open ports, and I have various target systems.
[ You might also like: Sysadmin security: 8 Linux lockdown controls ]
To see the ports in use on my Satellite server, I SSH to the server and then use
[root@opendemo ~]# netstat -tlpn Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN 1443/mongod tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN 1197/redis-server 1 tcp 0 0 0.0.0.0:5646 0.0.0.0:* LISTEN 1132/qdrouterd tcp 0 0 127.0.0.1:8751 0.0.0.0:* LISTEN 1194/python tcp 0 0 0.0.0.0:5647 0.0.0.0:* LISTEN 1132/qdrouterd tcp 0 0 127.0.0.1:19090 0.0.0.0:* LISTEN 1237/cockpit-ws tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1175/sshd tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN 1242/postmaster tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1396/master tcp 0 0 0.0.0.0:9090 0.0.0.0:* LISTEN 1138/ruby tcp 0 0 127.0.0.1:45285 0.0.0.0:* LISTEN 28650/Passenger Rack tcp 0 0 127.0.0.1:5671 0.0.0.0:* LISTEN 1140/qpidd tcp 0 0 0.0.0.0:8008 0.0.0.0:* LISTEN 1240/ruby tcp 0 0 127.0.0.1:5672 0.0.0.0:* LISTEN 1140/qpidd tcp6 0 0 :::8140 :::* LISTEN 2101/java tcp6 0 0 127.0.0.1:61613 :::* LISTEN 1135/java tcp6 0 0 :::5646 :::* LISTEN 1132/qdrouterd tcp6 0 0 :::5647 :::* LISTEN 1132/qdrouterd tcp6 0 0 :::80 :::* LISTEN 1131/httpd tcp6 0 0 :::22 :::* LISTEN 1175/sshd tcp6 0 0 ::1:5432 :::* LISTEN 1242/postmaster tcp6 0 0 :::3128 :::* LISTEN 1258/(squid-1) tcp6 0 0 ::1:25 :::* LISTEN 1396/master tcp6 0 0 127.0.0.1:8443 :::* LISTEN 1135/java tcp6 0 0 :::443 :::* LISTEN 1131/httpd tcp6 0 0 :::9090 :::* LISTEN 1138/ruby tcp6 0 0 127.0.0.1:8005 :::* LISTEN 1135/java tcp6 0 0 ::1:5671 :::* LISTEN 1140/qpidd tcp6 0 0 :::8008 :::* LISTEN 1240/ruby tcp6 0 0 ::1:5672 :::* LISTEN 1140/qpidd tcp6 0 0 :::5000 :::* LISTEN 1131/httpd [root@opendemo ~]#
However, some of those are limited to the localhost, 127.0.0.1. To see which ports are publicly visible, I start by using a default
nmap scan from my local system:
[pgervase@pgervase ~]$ nmap opendemo.usersys.redhat.com Starting Nmap 7.70 ( https://nmap.org ) at 2021-01-07 20:28 EST Nmap scan report for opendemo.usersys.redhat.com (10.19.47.240) Host is up (0.041s latency). Not shown: 993 closed ports PORT STATE SERVICE 22/tcp open ssh 80/tcp open http 443/tcp open https 3128/tcp open squid-http 5000/tcp open upnp 8008/tcp open http 9090/tcp open zeus-admin Nmap done: 1 IP address (1 host up) scanned in 3.81 seconds [pgervase@pgervase ~]$
This output shows that my local system can see fewer public ports than what I could see when I was SSH'd into the server. Some of those non-public ports are 25, which is used by master, and 8005, 8140, 8443, and 61613, which are used by java. Looking at
ps output and grepping for the PID of master from that
netstat output, master is the postfix mailer:
[root@opendemo ~]# ps auxww | grep 1396 root 1396 0.0 0.0 89740 2188 ? Ss Jan05 0:00 /usr/libexec/postfix/master -w root 29665 0.0 0.0 112816 968 pts/0 R+ 20:32 0:00 grep --color=auto 1396 [root@opendemo ~]#
That (master) is running locally so that mail can get sent to internal addresses but is not listening for incoming email, nor is it sending anything to any other host.
The other ports mentioned were for java. When you look at the
netstat output, two different java processes are responsible for those ports:
[root@opendemo ~]# netstat -tlpn | grep java tcp6 0 0 :::8140 :::* LISTEN 2101/java tcp6 0 0 127.0.0.1:61613 :::* LISTEN 1135/java tcp6 0 0 127.0.0.1:8443 :::* LISTEN 1135/java tcp6 0 0 127.0.0.1:8005 :::* LISTEN 1135/java [root@opendemo ~]#
When you look at
ps output for PID 1135, it’s used by tomcat:
[root@opendemo ~]# ps auxww | grep 1135 tomcat 1135 0.3 3.5 12409252 2165668 ? Ssl Jan05 9:25 /usr/lib/jvm/jre/bin/java -Xms1024m -Xmx4096m -Djava.security.auth.login.config=/usr/share/tomcat/conf/login.config -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start root 31507 0.0 0.0 112816 968 pts/0 S+ 20:53 0:00 grep --color=auto 1135 [root@opendemo ~]#
When I look in the
/usr/share/tomcat/conf/server.xml file, it has content such as:
<Server port="8005" shutdown="SHUTDOWN"> ... <Connector port="8443" address="localhost" protocol="HTTP/1.1" SSLEnabled="true" maxThreads="150" scheme="https" secure="true" clientAuth="want" sslProtocols="TLSv1.2" sslEnabledProtocols="TLSv1.2" ....
This shows how the ports that will get used are defined in the config file.
When I look at the other java process I mentioned, PID 2101 for port 8140, I see this is used by puppet:
[root@opendemo ~]# ps auxww | grep 2101 puppet 2101 0.2 2.5 9787492 1545188 ? Sl Jan05 7:14 /usr/bin/java -Xms2G -Xmx2G -Djruby.logger.class=com.puppetlabs.jruby_utils.jruby.Slf4jLogger -XX:OnOutOfMemoryError="kill -9 %p" -XX:ErrorFile=/var/log/puppetlabs/puppetserver/puppetserver_err_pid%p.log -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar:/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter.jar:/opt/puppetlabs/server/data/puppetserver/jars/* clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d --bootstrap-config /etc/puppetlabs/puppetserver/services.d/,/opt/puppetlabs/server/apps/puppetserver/config/services.d/ --restart-file /opt/puppetlabs/server/data/puppetserver/restartcounter root 31696 0.0 0.0 112816 968 pts/0 S+ 20:55 0:00 grep --color=auto 2101 [root@opendemo ~]#
Based on the
netstat output, port 8140 should be visible to the public, but
nmap from my local system didn't report it in its results. Here again, is the
netstat output from the Satellite server:
[root@opendemo ~]# netstat -tunap| grep 8140 tcp6 0 0 :::8140 :::* LISTEN 2101/java [root@opendemo ~]#
nmap from my local server:
[pgervase@pgervase ~]$ nmap opendemo.usersys.redhat.com | grep 8140 [pgervase@pgervase ~]$
However, I can force
nmap to check on a specific port or range of ports:
[pgervase@pgervase ~]$ nmap -p 8140 opendemo.usersys.redhat.com Starting Nmap 7.70 ( https://nmap.org ) at 2021-01-07 21:07 EST Nmap scan report for opendemo.usersys.redhat.com (10.19.47.240) Host is up (0.039s latency). PORT STATE SERVICE 8140/tcp open puppet Nmap done: 1 IP address (1 host up) scanned in 0.39 seconds [pgervase@pgervase ~]$ nmap -p 8000-9000 opendemo.usersys.redhat.com Starting Nmap 7.70 ( https://nmap.org ) at 2021-01-07 21:07 EST Nmap scan report for opendemo.usersys.redhat.com (10.19.47.240) Host is up (0.040s latency). Not shown: 999 closed ports PORT STATE SERVICE 8008/tcp open http 8140/tcp open puppet Nmap done: 1 IP address (1 host up) scanned in 2.12 seconds [pgervase@pgervase ~]$
nmap to check those ports, I was able to see the :8140 port which a basic
nmap scan did not report. This shows that a default
nmap scan without extra arguments might be good enough for a first look at the system but might miss ports that are actually open.
This information is important in security testing so that sysadmins can identify potential vulnerabilities. From the
nmap output scan, run locally on my system, you saw the ports that were publicly open. Prior versions of Satellite had tomcat configured so that some of those ports were public when that was not necessary. To read over some of the discussion for that issue, you can read the Bugzilla where this was resolved.
Another issue that
nmap can help with is to verify the certs being used on those various ports. Using
nmap, you saw the open ports. Using those ports, you can use OpenSSL to see the cert used on the port. A number of those port are using self-signed certificates. To use
nmap and OpenSSL together to check the ports on a remote system, you could do something like:
$ for port in `nmap -p 1-5000 opendemo.usersys.redhat.com | grep " open " | cut -d "/" -f 1` > do echo checking on port: $port > echo | openssl s_client -showcerts -connect opendemo.usersys.redhat.com:$port > done &> opendemo.certs.txt.`date +%Y%m%d`
opendemo.certs.txt.20210127 file, it would have content like:
checking on port: 443 depth=1 C = US, ST = North Carolina, L = Raleigh, O = Katello, OU = SomeOrgUnit, CN = opendemo.usersys.redhat.com verify return:1 depth=0 C = US, ST = North Carolina, O = Katello, OU = SomeOrgUnit, CN = opendemo.usersys.redhat.com verify return:1 CONNECTED(00000003) …. SSL handshake has read 3476 bytes and written 463 bytes Verification: OK --- New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256 Server public key is 2048 bit Secure Renegotiation IS supported
Use that output file to verify that the certs in use are the correct TLS version.
If you use
ncat), you might see more information than presented in the web UI. For this example, I used
nc to connect to a web server:
$ nc 10.19.47.242 80 asdf HTTP/1.1 400 Bad Request Date: Sat, 09 Jan 2021 01:25:40 GMT Server: Apache/2.4.37 (Red Hat Enterprise Linux) Content-Length: 226 Connection: close Content-Type: text/html; charset=iso-8859-1 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>400 Bad Request</title> </head><body> <h1>Bad Request</h1> <p>Your browser sent a request that this server could not understand.<br /> </p> </body></html>
From that output, I can see the version of Apache that was installed. With that information, an attacker could learn what exploits the server was vulnerable to. Because of this, a web server should limit the amount of information that displayed:
[pgervase@pgervase ~]$ nc opendemo.usersys.redhat.com 443 GET / HTTP/1.1 HTTP/1.1 400 Bad Request Date: Fri, 08 Jan 2021 02:33:08 GMT Server: Apache Content-Length: 362 Connection: close Content-Type: text/html; charset=iso-8859-1 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>400 Bad Request</title> </head><body> <h1>Bad Request</h1> <p>Your browser sent a request that this server could not understand.<br /> Reason: You're speaking plain HTTP to an SSL-enabled server port.<br /> Instead use the HTTPS scheme to access this URL, please.<br /> </p> </body></html> [pgervase@pgervase ~]$
Note that in this output, there is no version information for Apache.
In this next example, I use
nc to connect to port 21 on my client system, which I can see is open:
[pgervase@pgervase ~]$ nmap 10.19.47.242 Starting Nmap 7.70 ( https://nmap.org ) at 2021-01-08 21:02 EST Nmap scan report for 10.19.47.242 Host is up (0.039s latency). Not shown: 996 closed ports PORT STATE SERVICE 21/tcp open ftp 22/tcp open ssh 80/tcp open http 111/tcp open rpcbind Nmap done: 1 IP address (1 host up) scanned in 0.83 seconds [pgervase@pgervase ~]$ nc 10.19.47.242 21 220 (vsFTPd 3.0.3)
That 3.0.3 version is confirmed when I SSH to the system and use the
[root@vulnerable ~]# rpm -q vsftpd vsftpd-3.0.3-32.el8.x86_64 [root@vulnerable ~]# rpm -qi vsftpd Name : vsftpd Version : 3.0.3 Release : 32.el8 <snipped>
Again, just like with learning the Apache version on the device, being able to do reconnaissance in your environment so that you know what a potential attacker can learn about your systems is important.
Scanning from Kali
In the next section, I show some results from scanning a system from a Kali server. In this example, I know that the target server has
distccd running on port 3632, but, like earlier,
nmap does not detect that port by default, and so I had to check for it specifically:
Now that you know
distccd is open, you can use nmap's built-in capabilities to determine where it could potentially be exploited:
If you'd used only a plain
nmap scan, you'd have missed that exploitable vulnerability. In my example, I ran
uname -a on the remote system, but I could have run any command.
One final way to use
nmap is with the
-sV option, which probes the open ports and determines the service or version information. For this example, I changed the port that Apache runs on from 80 to 90 and then restarted the service. Below you can see the difference between a plain
nmap scan and then using the
-sV option, which correctly determined the service as
httpd rather than
[root@pgervase ~]# nmap 10.19.47.242 Starting Nmap 7.70 ( https://nmap.org ) at 2021-01-09 19:57 EST Nmap scan report for 10.19.47.242 Host is up (0.043s latency). Not shown: 996 closed ports PORT STATE SERVICE 21/tcp open ftp 22/tcp open ssh 90/tcp open dnsix 111/tcp open rpcbind Nmap done: 1 IP address (1 host up) scanned in 1.80 seconds [root@pgervase ~]# nmap -sV 10.19.47.242 Starting Nmap 7.70 ( https://nmap.org ) at 2021-01-09 19:52 EST Nmap scan report for 10.19.47.242 Host is up (0.040s latency). Not shown: 996 closed ports PORT STATE SERVICE VERSION 21/tcp open ftp vsftpd 3.0.3 22/tcp open ssh OpenSSH 8.0 (protocol 2.0) 90/tcp open http Apache httpd 2.4.37 ((Red Hat Enterprise Linux)) 111/tcp open rpcbind 2-4 (RPC #100000) Service Info: OS: Unix
[ Want to learn more about security? Check out the IT security and compliance checklist. ]
Now that you've been able to get a detailed report of what's running on your systems, what do you do next? The first thing is to be sure that there are no unexpected ports open. For this, verify with the applications team, security teams, and your coworkers might be appropriate. Next is to ensure that the exposed services are properly secured. This means taking steps such as making sure that all software is updated, updated ciphers are supported, insecure protocols are not in use, and default passwords for the services have been changed.
This article is an introduction to investigating your servers. Use
nmap to verify which ports are open and use the
ps command to trace back the processes using those ports. I've also provided an example of how you can use
nmap with the
--script argument to test your systems. To continue on this learning path, one possible next step is to research using
nmap as an attack engine by investigating the default scripts in