In parts one and two of this series, I walked you through hardening your system by identifying unneeded services, and then segmenting and firewalling. Now that you’re all locked down, let’s talk about how we might validate that all of that hard work is actually providing the desired results. In this final installment, we'll talk about how to scan your network for open ports, and we’ll even touch on vulnerability scanning.
Why we validate
Attackers are constantly scanning your network. In my article Locking down sshd, I placed a system on Digital Ocean with SSH open to the world and waited for someone to attack. The goal there was to see how long it would take for the brute force attempts to start, and that was about an hour, but the first scan happened about 20 minutes in. That result reminds me of the old days when the internet was stricken by worms attacking Windows servers. If you connected an unpatched Windows machine to the internet, it was infected in minutes.
If the bad guys are scanning your network to see what you’ve got open for attack, shouldn’t you do the same thing? The answer is "yes," but there are two different avenues. We’ll look at both approaches today.
Your network is like a castle. It has an outer wall around the outside that the world sees. If you’ve segmented your network, there are doors and walls within that border that allow or deny access to different areas. Behind these are the inner doorways and passages that your users and servers see.
The outer wall is the simplest, so I’ll start there. If you have a public service, like your company’s website, that site doesn’t do you much good if the internet can’t reach it. But, of course, you don’t want wide-open access from the internet. That’s where firewalling comes in. The perimeter firewall, like the wall around a castle, should only allow access to the ports required for your internet-accessible services to work. It’s like opening the drawbridge for things you’re expecting. We covered that functionality in part two of this series.
So, you’ve configured your firewall. You’re good, right? Maybe, but how do you know? You scan. Scanning your network is like having an inspector check your outer wall to see if the drawbridge opens for the right people, and not for others.
Scanning for open ports
There are a wide variety of tools you can choose to scan your network. Whether you use one or a combination, it's good to understand what each can do for you and what they can't.
Let's start with Shodan, which is a service with both free and paid access that's always scanning the internet and maintains a database of all of the open ports it sees. This database is not exactly up to the minute, but if you put a service online, you can bet that sooner rather than later, Shodan will know about it. So, a good first step would be to go to Shodan and ask it what it knows about your network. I looked up the IP of my own website, here’s just a tiny snippet of the information Shodan returned:
Shodan's output is much more thorough than that, however. It also provides a decent amount of potential vulnerability information, but we’ll talk more about that later.
Shodan is great, and maybe it’s enough to do your external scan. You could also fire up a cloud instance and scan your network using a tool like
nmap. Nmap is a great free tool and it's included in many Linux distributions. You can even script it to receive output in a friendly, easy-to-parse format.
A friend of mine is in the process of writing a cloud scanner with
nmap as its back-end, that he calls "DABS." This scanner is designed to run on cloud providers and scan whatever you point it at, logging its results into a database.
The beauty of this sort of system is that you can put agents at different vantage points in your network, which helps you validate your network segmentation and firewalling by scanning each network segment from other network segments. A tool working at this level of detail is like having an inspector inside your castle, verifying that they can’t get into your treasury unless they have the right key.
Place a scanning agent in, say, your web server network, and have it scan your database network. Check those results against your expected firewall rules and make sure they match. You’d also do this from your various user networks. Can your desktops reach the proper services? Can they reach things they shouldn’t? Just because they’re your employees, that doesn’t mean you can implicitly trust their systems. It’s like making sure the average citizens inside of your castle walls can’t walk into the king’s bedroom or the armory but can walk up to the fish merchant and buy dinner.
Once you’ve scanned from all of the places you’d like to scan from, check the results against your expectations, fix anything that looks out of place, and scan again. Once you’re happy with the results, call that your baseline. A baseline is a known good configuration that you can compare against in the future.
If you’re scanning regularly, you can compare future scans against your baseline and flag any changes as something you should either look into or add to your baseline. Did some new port you didn’t expect show up in your scan? That might be something to worry about.
Scanning for vulnerabilities
If you’d like to take this process a step further, you should consider not just scanning for open ports, but also using a vulnerability assessment tool. Vulnerability assessment tools actually look at the services they find on these open ports and report on what sort of nasty things they might be susceptible to. To stick with our castle analogy, it’s like your inspector not just checking if they can reach the fish merchant, but actually verifying that the fish merchant sells fish.
Tenable produces a for-pay vulnerability assessment tool called Nessus, and OpenVAS is a comparable free tool. OpenVAS, Nessus, and other vulnerability assessment tools scan for open ports and then:
- Actually look at the services running on those ports.
- Try to figure out what these services are.
- Report back on what they find.
These services will even do some low impact probing to see if these services are vulnerable to any known issues, which is like your inspector trying to steal a fish from the fish merchant to see if the guards stop them. This probing is usually based on what version of a piece of software they detect, but it can get more advanced and try automated exploit code. If you’d like, you can even pay a professional to attack your services to see how far they can get, but that’s a little further than I’m going with this article.
Vulnerability assessment tools can also be used to get different viewpoints across your network, as I described earlier. Many of them are designed to be run in an automated fashion, so you can have a near real-time look at your network and the state of your services. I wouldn’t call these monitoring tools, but instead a piece of your infosec strategy.
Managing the complexity
You might be reading this article and thinking, "This is great, but when the heck would I have time to do all of this?" We’re all busy as sysadmins, and this is just one more thing on the plate, right?
I have just one response: Automation is key. Nmap can be automated. DABS is designed to be automated. OpenVAS and Nessus can be automated. As for remediation of the issues that these scans uncover? Imagine an automated scan finding some new problem, and firing off a Red Hat Ansible playbook to remediate the issue immediately. (I can’t say that I know of a tool that does this just yet, but it sure would be a great next step.)
Thanks for reading this series. I hope I’ve shared a decent path to improving your security posture, and protecting what’s important. Here are links to the tools I’ve talked about in this article:
And check out this great talk on baselining your network from DerbyCon 1.0.