Ask Shadowman: Red Hat Linux Security


June 2002

He's smart. He's mysterious. He's got only one name like Prince or Michaelangelo. And every month Shadowman answers the toughest technical questions anyone has ever dared ask a two-dimensional logo. This month's mission: To respond to your questions about Red Hat Linux security. A remarkable feat considering Shadowman's face is missing a mouth...

Join us again in July when Shadowman shows the patience of a saint, the judgement of a Supreme Court Justice, and the communication skills of a boll weevil, answering your questions about application support for Red Hat Linux that you didn't know you asked.

Got a question about Red Hat Linux applications?

Got a question that you'd like Shadowman to answer? Ask him.



Ron M. wrote:
Is there a way to see what open connections are on iptables kinda like netstat?

Shadowman says:
This question reminds Shadowman of that old saying about how everything looks like a nail when all one has is a hammer.

However, in this case we have more than just a hammer -- we actually have a pretty good set of tools at our disposal. So why not use both -- netstat for tracking connections, and iptables for controlling access.

That said, you can get some information from iptables by using the -L and the -v options to get packet counts for the various chains and rules. But this is nowhere near the level of information that netstat can give you.

Use the right tool for the job, and you won't be forced to hammer a screwdriver through a Studebaker oil filter like some logo model whose name Shadowman won't mention...


David K. scribbled:
What steps does Red Hat take to secure their network from the outside world? If someone were able to gain access to any part of Red Hat's internal network, couldn't they use a keystroke monitor or sniffer to gain the critical passphrase for generating RPMS used on RHN, gain access to the CVS archive, and credit card information? By now you're probably thinking I'm a pessimist, but here's one more: suppose an eletrical fire completely destroys the Red Hat building. Do you have offsite backups of code and critical servers?

Shadowman says:
Shadowman doesn't think you're a pessimist; Shadowman thinks you're paranoid. But when it comes to security, a little paranoia never hurt anyone.

As far as Red Hat's network security is concerned, we haven't done anything:

  • We've left all our ports open
  • All our root passwords are one character long (and Shadowman will give you a hint -- it's a letter!)
  • All customer information (including credit card numbers) are kept on our main web server
So feel free to drop by, and...

Ok.

Now that all the nasty people have gone and are trying to wreak havoc, the rest of us can talk about Red Hat's security.

Shadowman asked the gurus in Red Hat's IS/IT department, and learned the following:

  • We have a firewall with outward-facing services originated from within a DMZ
  • Packet sniffers/keystroke monitors wouldn't do much good -- we use OpenSSH to encrypt all traffic on our internal networks, and you'd have to figure out who signs the RPMs (and from which system) in order to sniff the right keyboard (and more information about this is one thing that Shadowman won't share -- not even with non-nasty people such as Shadowman's readers)
  • Critical customer data is on a different network from the rest of the company (the basic premise is that the various internal networks are considered hostile, and configured appropriately)
  • Punching your way into one system will not get you access to everything, and major business functions are distributed between multiple systems, making overall access that much more diffi...

Uh oh, they're back!

Shadowman also looked into physical security and found that the beautiful Japanese rice paper decorations in the single company-wide server room are a bit of a fire hazard, especially since we have no fire suppression equipment. However, the fires we've had have been small, and centered mainly in the stack of backup tapes we keep in the corner. But feel free to come over, and have a smoke while touring our server room...

Whew.

Here's the real skinny:
  • Red Hat is located in more than just one building -- significant parts of the company are located around the world, not just at Red Hat's corporate headquarters
  • The systems are geographically dispersed, residing in various Red Hat facilities and cololocation sites
  • Backups are done regularly, with both onsite and offsite storage procedures in place
  • Disaster recovery plans have been a key focus of our IS/IT department after hurricane Fran passed directly over the Red Hat office in 1996...


Nat G. typed:
Last August I put in place a Red Hat Linux 6.2 system, running on a 1997 Gateway 2000 P133, to serve as my home firewall/gateway. I have Internet service via our local cable TV provider, and I wanted the Linux system to function as a masquerading firewall.

I never told anyone the system's root password, nor did I write it down anywhere. I closed down all network services except for ftp and ssh (yes, I did shut off telnet). And those services I restricted using the following /etc/hosts.deny:

ALL: ALL
offset with the following /etc/hosts.allow:
ALL: 127.0.0.1
ALL: 192.168.2.
# Want to allow ssh connections from the central office
sshd: mangrove
where 192.168.2.xxx are the local machines, and 'mangrove' is an alias defined in my local /etc/hosts file to refer to the numeric IP address of my employer's office.

I strongly doubt that my wife or my children even know how to get to a login prompt at the Linux machine, or what to do with it if they got one.

Despite these precautions, in February I discovered that the system had been broken into. /etc/inetd.conf had been altered to open up the telnet and the POP3 ports. Moreover, I got a mail message from our ISP advising us to perform a virus scan because they had detected "virus-like activity" from our IP address. I assumed that this was probably referring to outbound spam. And upon login, I got the following greeting:

Last login: Tue Oct 19 02:44:18 2021 from
which I took to mean that the intruder had also stomped on whatever file is used to record the last-login information.

The most distressing thing about this episode is that I have no plausible theory as to how the intruder got in. Therefore, I CANNOT guarantee that it couldn't happen again.

So I bought a commercial router and "retired" my Linux system from firewall duty. I had been preparing another Red Hat Linux system to serve a similar role in my wife's office; but I have abandoned that effort. Another commercial router now functions as their firewall.

My original inspiration to use a Linux machine for this purpose was the Red Hat 6.0 system my employer has been running. That, too, has now been moved behind a commercial router.

The bottom line is that until I know what hole the intruder exploited-- and have reason to believe that all such holes have been plugged -- I no longer dare to expose any Red Hat Linux system directly to the Internet. I'm greatly disappointed by this, because there are many reasons to like the idea of using Linux for the job!

I would very much like to receive such assurances, and to feel comfortable once again. But -- "Fool me once, shame on you. Fool me twice, shame on me!"

Shadowman says:
Shadowman has a confession to make -- he also used Red Hat Linux as a masquerading firewall. And when he first put that system up on the Shadow Lair's DSL connection, said system was hacked within 24 hours.

Does that mean that Red Hat Linux is incapable of securely connecting a private network to the Internet?

No, it means that Shadowman was an idiot.

When Shadowman configured that box, Shadowman did several things right:

  • He practiced good control of sensitive passwords
  • He shut down services that were not needed

However, Shadowman really blew it in other areas:

  • He left FTP (a very easy service to misconfigure) open
  • He didn't bother with firewall rules restricting access to active services, leaving things such as ssh open to every IP address in the world
  • He felt that the internal network was 100% trustworthy
  • He thought that security was something you worry about once and never again, once you "get it right"
  • He installed an older version of Red Hat Linux, and neglected to apply security-related updates as they became available

So Shadowman was an idiot. Fortunately, the bigger idiot in this tale of woe was the person that hacked Shadowman's system, because this person failed to cover their tracks:

  • They broke DNS on the system, making it very obvious to Shadowman that something was amiss
  • They left a suspicious entry in /etc/passwd:
    Name: b0rk
    UID/GID: 0/0
    
  • They left all the system logfiles in place, including a beautiful example of the exploit that actually did in the system
  • They left the .bash_history files for both root and b0rk in place, making it very obvious what they did once they got in

After pulling both internal and external Ethernet cables, cracking a cold Tsingtao, and reviewing everything in /var/log/, Shadowman started by questioning the users of the Shadow LAN. A few moments with Shadowboy and Shadowwoman quickly confirmed that if it was an inside job, they weren't involved.

That left Shadowdog, but since she has no account on the system (and, being a toy poodle, is not tall enough to effectively shoulder-surf for passwords, much less reach the keyboard to enter them), Shadowman felt sure that the attack came from outside. Why such paranoia? According to a recent CSI/FBI survey, even though the percentage of attacks originating from within organizations has now dropped to only 33%, that's still more than reason enough to consider internal networks as hostile, and to be ever vigilant for employees probing your network for weaknesses. Or packet-capturing poodles.

Next, Shadowman studied the other systems on the Shadow LAN. Thankfully, there was no evidence of untoward activity on these systems. This was backed up by the .bash_history files' lack of evidence that ssh was used by the perp (All systems on the Shadow LAN only have ssh access in place -- at least Shadowman got that part right). So the damage done was confined to the one system. Shadowman cheered, but since it was now 3am, the cheering awoke Shadowwoman, who strongly suggested that sleep, not computer forensics, were more appropriate for the hour.

The next day, Shadowman awoke to assess the overall damage, and to determine whether or not the authorities should be involved. In Shadowman's case, it didn't seem necessary -- the damage was limited to the time required to set things right. And given Shadowdog's checkered past as a stray, it didn't seem smart to invite The Man in for tea -- no sense in reminding Shadowdog of her days on the run, and risk her going on another rampage.

As far as setting things right, here's a thumbnail sketch of Shadowman's "Steps to Internet Recovery":

  • Completely reinstalled Red Hat Linux (selecting a more recent release)
  • Registered the system with Red Hat Network
  • Using Red Hat Network, applied all available security updates and bugfixes (sure, Shadowman could do all this manually, but he's lazy, and Red Hat Network does a better job keeping track of this stuff than Shadowman ever did)
  • Shut down all unneeded services (including FTP -- OpenSSH includes scp, which copies files just as effectively as FTP, and more securely)
  • Implemented a more aggressive firewall that only allowed outside access from known and trustworthy systems, while silently dropping (not rejecting) packets from everyone else.

After all this was done, Shadowman did some testing to make sure the firewall allowed the necessary functionality while still preventing script kiddies from turning the Shadow LAN into their playground. Shadowman had heard about a nifty firewall testing site made available by Gibson Research Corporation, so he pointed Mozilla at http://grc.com/default.htm and selected the "Shields UP!" link. Thinking back to movie shown at a bachelor party years ago, Shadowman hesitatingly clicked on the "Test My Shields!" and "Probe My Ports!" buttons.

After a delay, a somewhat breathy page confirmed that the Shadow LAN's firewall essentially made the Shadow LAN invisible to the outside world. After all, hackers can't hack what doesn't exist. Sure, Shadowman could have used nmap from an outside system to do the same thing, but it wouldn't be a complete test, as the firewall has rules that open various ports for all the external systems Shadowman uses. But the real reason is that Shadowman is a sucker for pages that say things like:

"Stealth Mode! There is NO EVIDENCE WHATSOEVER that a port (or even any computer) exists at this IP address!"

Finally, Shadowman resolved to do two things on a regular basis:

  • Check for evidence of breakin attempts
  • Check for new bugfixes and security patches, and apply them immediately (Red Hat Network makes this part easy)

It's been over a year, and there's been nary a problem with baddies attacking the Shadow LAN. But to paraphrase the old saying:

The price of security is eternal vigilance.

Security is never done, and there are never any guarantees -- what is secure today might not be secure tomorrow.

Shadowman is not sure if his situation is similar to yours, but Shadowman is confident that he is not alone when it comes to the mistakes he made, the process he used to recover from the break in, and the concern about waking up significant others at 3am.


Jon M. penned:
My question is how do you maintain security if the following is the setup: We are a Network Integration company and we support several Linux servers on various sites. What we want to do is have the logs either copied to our server or a realtime viewing of the logs (we are implementing a frame relay network to most of our clients to be able to monitor their servers). What I would like to be able to receive from those both in and out of the FR is a status of their server without opening too many ports in their firewall. Also which product(s) offers the best monitoring of security breaches or server status changes (DOS attacks, hack attacks, etc)? I need to be able to view the information is both Windows and Linux KDE desktop.

Currently our firewalls are ipchains and only allow in certain traffic (e-mail, http, dns, ssh access, icmp (i believe (only 2 type) and webmin access).

What is the best way to setup a DMZ with web servers and mail servers in the zone?

Shadowman says:
Shadowman marvels at the number of questions you've managed to put into one email! You might have a rewarding career as a member of the White House press corps.

However, Shadowman will take a page from the President's playbook, and choose to only answer those questions that interest him. And unlike the President, Shadowman is not going to allow you a follow-up.

Shadowman notes that syslogd does support remote system logging, so getting logs to show up on another system can be as simple as adding "-r" to the SYSLOGD_OPTIONS line in /etc/sysconfig/syslog on your "log server", and changing /etc/syslog.conf on your clients' machines to something like:

*.* @loghost

(where "loghost" is the log server's hostname)

You'll still need to have some sort of connectivity between your logging system and your clients' machines. A VPN could take care of this (just be sure to firewall the VPN to prevent h4X0rz from tunneling through the VPN and 0wNz0ring j00).

A quick search for "log analysis" on freshmeat.net (http://freshmeat.net) gave no fewer than 193 hits, including one that parses logs from Half Life: Counter-Strike. Not that Shadowman would ever have a use for such a thing...


Nathan with-no-last-name scrawled:
There are so many resources on the web regarding Linux and security, but not all are current, and changes happen so quickly that it can be difficult to keep up with current "best practices". Could Shadowman put together a list of the best current methods of using Red Hat to:

Set up a firewall
Send secure e-mail
Secure an e-mail server
Secure a web server
Perform secure transactions on the web
Set up a VPN (will there ever be a Freeswan rpm for Red Hat?)
and Anything else Shadowman deems appropriate?

Links to current reference material would also be great.

Shadowman says:
Shadowman wonders what you will do if Shadowman answers all your questions; your job for the next six months will already be done for you!

Fortunately, you've given Shadowman an out; Shadowman deems it appropriate that he give you a list of security-related sites that are the favorite of Red Hat's most security-minded security weenies.

Besides, just one of your "how to setup Red Hat Linux" questions could be the subject of an entire book. So without further ado, the list:

http://www.redhat.com/security/

http://lwn.net/ (the Security page in particular)

http://online.securityfocus.com/unix

http://packetstormsecurity.packetstorm.org/

http://rr.sans.org/

http://secinf.net/iunixe.html

http://securitygeeks.shmoo.com/

http://www.blackhat.com/

http://www.gocsi.com/

http://www.insecure.org/tools.html

http://www.intersectalliance.com/projects/LinuxConfig/index.html

http://www.linux-sec.net/

http://www.linuxsecurity.com/

http://www.nessus.org/

http://www.nmrc.org/

http://www.securiteam.com/

http://www.securityfocus.com/

http://www.shmoo.com/

http://www.snort.org/

http://www.whitehats.com/

http://www.wiretrip.net/rfp/


Ronald R. emailed:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html;
charset=3Diso-8859-=
1">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server 
version 6.0.5762.3">
<TITLE>VPN's</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->
<BR>

<P><FONT SIZE=3D2 FACE=3D"Arial">Is it going 
to support VPN's</FONT>
</P>

</BODY>
</HTML>

Shadowman says:
Yes. Unfortunately, it does not eliminate HTML mail. Or incompletely worded questions.


Got a question about Red Hat Linux applications for the July edition? Send it right now.