A hole in one: Firewall exception testing and troubleshooting

A firewall or a packet filter, in general, is a network security system that prevents all network communications except those connections that are explicitly allowed. Network firewalls filter traffic between networks and host-based firewalls filter incoming and outgoing traffic on a host machine.

For the following example, keep in mind that by default, no traffic is allowed to pass a firewall. The client in Image1 is not allowed to send a request to the web server until an exception is added to the firewall rule set.

Allow incoming HTTP/HTTPS traffic

In our example, we have a web server running on our host, but no one can access this webserver because a network firewall and a host-based firewall are filtering all incoming traffic. Let's change that now.

Because we are root on our web server, we can configure the host-based firewall by ourselves. If you are running RHEL or CentOS, you can allow HTTP and HTTPS traffic to pass the host-based firewall with the following commands:

$ sudo firewall-cmd --add-service=http
$ sudo firewall-cmd --add-service=https
$ sudo firewall-cmd --runtime-to-permanent

For more information on how to use the firewall-cmd command see the  firewall-cmd(1) documentation:

RHEL 7 -- Using Firewalls
RHEL 8 -- Using and configuring firewalls

For the network firewall, we have to ask our network security administrator kindly to add an exception to the firewall ruleset.

We are lucky and get an exception for our client in Image1 to access our web server. Table 1 shows a simple example of a rule set, allowing our client to access the ports 80 (HTTP) and 443 (HTTPS) on our web server and forbids all other traffic.

How to troubleshoot firewall issues

Sometimes you think that everything has been configured on all related firewalls to allow access to a particular service, but the clients are still not able to connect to it. Here is what I do in such a situation.

First, I'll check the webserver to confirm whether the service is running or whether it has stopped for some reason. I do so by running the following commands on my webserver:

$ sudo systemctl is-active nginx.service
active
$ ss -tln
State     Recv-Q    Send-Q       Local Address:Port        Peer Address:Port
LISTEN    0         128                0.0.0.0:80               0.0.0.0:*
LISTEN    0         128                0.0.0.0:443              0.0.0.0:*

Now I know that my web server is running and listening on the expected ports. Next, I'll check the host-based firewall to see whether both ports/services are allowed or not.

$ sudo firewall-cmd --list-services
dhcpv6-client http https ssh

If either HTTP or HTTPS was missing here, I would know that I have forgotten something, and the host-based firewall is blocking the client requests. The next step would be to try to access the webserver from a host on the same subnet. This way, I could confirm whether there is still some problem with my host or not. When everything is fine, it's time to probe the network firewall.

For this, I send a request from my client to my webserver and take note of the timestamp, source IP address, destination IP address, and destination port of my request. With this information, I kindly ask my network security administrator to check the logs of the network firewall for any access denied entries. Maybe the guy just forgot to commit the new rule to the running config.

That makes up the following troubleshooting order:

1. Check that service is up and running
2. Check that the host is listening on the appropriate ports
3. Check the host-based firewall for allowed services and ports
4. Check whether service is accessible from a host on the same subnet
5. Try to connect to service from another network host and make a note of timestamp and IP addresses and ports that are involved.
6. Ask you network security administrator kindly to check the logs of the network firewall(s)

Wrapping up

The list of steps above should help you diagnose firewall configurations that are preventing connectivity. Remember that the host-based firewalls protect the server and the network firewalls separate network segments.

[ Want more for your network? Download a free ebook on network automation with Ansible. ]