Ask The Expert: Linux Security Questions Answered

Mark Cox
Senior Director of Engineering

The man behind Apache Week, founding member of the OpenSSL group, the designer of the first non-US version of Stronghold Secure Web Server, and a member of the core development team for Apache since 1995, Mark Cox, Red Hat Senior Director of Engineering, is one of our heroes. We recently sat down to chat with him on Linux security, home ownership, and the advantages of open source code.

Q:  Hey Mark. Justify your existence.
A:  What a difficult question. To justify my existence I'd have to answer questions like "Why am I here?" and "Why do I exist?". If the answers to these questions are ever discovered the Universe would have to replace me with something new and even more bizarre. As Douglas Adams would have said, this has probably already happened.
Q:  Tell us a bit about your background. What made you interested in security in the first place?
A:  That's an easier question; security is really fun. Back in the early 90's during my PhD at the University of Bradford we had developed one of the first fully automated robotic telescopes. The telescope was at a very remote site high on the Oxenhope moors deep in Bronte country. During the day it would queue up requests for viewing time on the telescope then at night, if all the conditions were right the telescope would open its roof, schedule and perform jobs, process and return the results all by itself. I saw the potential for switching away from a dedicated modem BBS-type system and using the Internet for accessing the telescope. So we started out with an experimental interactive gopher site, but that was too difficult to make useful so just as the web started out we developed systems to allow internet users to queue jobs and see results using the web.

We had some unique security challenges to solve; firstly we've got hundreds of thousands of pounds of equipment at a remote site so physical security is important - especially when you have an open roof! But most interesting to me was the security systems for the network we needed to develop. Intruders getting into the system through the Internet had the ability to do real harm - the ability to damage the equipment by fooling the weather sensors or keeping the roof open for example. Also, the building housing the telescope was quite small and the telescope moved very fast, so when working in the building during the day we'd slow the telescope down so that we could get out of the way of it in an emergency and not get pinned to a wall or hit on the head. We went around identifying the risks - in this case the potential ability for an attacker to change these speed controls and hurt somebody.

You have to assume that a sophisticated attacker will be able to get into your system, and so the design was built around various security zones making it hard to pass between these zones. The planning and implementation worked out well; whilst the telescope was operational the only issues we saw was with denial of service as the telescope was so popular. We would get hundreds of jobs submitted every day, but there was no way we could perform them all -- in the north of England there are not many hours of darkness when it isn't raining.

Q:  How did you get involved with Apache and C2Net?
A:  At the time I was developing the telescope systems there were a number of things that the web server (NCSA httpd) just wasn't able to do. We wanted to do more with CGI scripting, mass user authentication against databases and so on. I had developed a collection of patches to the server we were using, and in discussing them met up with Brian Behlendorf. Brian had found a number of people who all had patches to the NCSA httpd server and he told me about the Apache project that was just forming.

The PhD was going well until the days when the commercial world wanted people who knew about the Internet and were willing to pay nearly ten times what I was earning as a student. It was a difficult decision, but I didn't like being poor so the PhD got put on hold and I started working for a small startup in Leeds, UK Web. UK Web was the first company to offer commercial support for Apache, and we started the Apache Week newsletter soon after. One of my Apache colleagues, Sameer Parekh, had a company in California and was heavily into crypto and privacy issues - he was actually named as one of the 50 people who matter most on the Internet for this. He had developed a packaged version of Apache with SSL support for the US market but due to the ridiculous crypto munitions regulations couldn't export his work. So I started out from scratch and developed a new version of the Stronghold server in the UK. That led to us founding C2Net Europe Ltd which later was part of the acquisition by Red Hat.

Actually before we were acquired by Red Hat we'd often tell people that our packaging of Apache was like the Red Hat packaging of Linux. We'd put everything you needed to run a secure Apache web server into one package that could be up and running quickly without having to spend ages downloading, compiling and patching lots of different programs.

Q:  What does a "Senior Director of Engineering" do at Red Hat? And when we say that, we mostly mean what do you do?
A:  My girlfriend has been pestering me for months for me to answer the same question - but I don't think it really matters what the job title is. Since joining Red Hat I've been doing all sorts of things. I see my job being about finding the things that need doing and getting them done no matter what they are. So at the moment I'm running a security response team at Red Hat - this involves keeping track of security issues that affect our users, analyzing and responding to them, making our advisories sane, talking to the press and generally making sure we're giving our customers what they want.

When I took a look at Linux security I was overwhelmed by the number of newsletters, mailing lists, and sites that have an excess of security information. Most of it is redundant or copied from other sources and a lot of it is copied badly with inaccuracies. As a user of Red Hat Linux I'd never have the time to keep up with and parse all this stuff. I want Red Hat to do that work for me - to act as a filter and give me clear, concise and accurate information about vulnerability and its consequences. So that is where I'm adding value at the moment.

As one part of that I've been working with some folks at Mitre on the CVE project. We've gone through the last couple of years of Red Hat security advisories checking them for accuracy and mapping them to common vulnerability descriptions. All the data from this will be published soon and will help everyone who is using our advisories or researching security metrics.

I also attend a lot of phone meetings and play buzzword bingo.

Q:  There's been a lot of talk lately about the benefits of Open Source code when it comes to keeping systems secure-- what are the biggest advantages?
A:  Open source security gives you some great values: Honesty, Openness, Transparency, Integrity. Closed source vendors can pick and choose what flaws they want to talk about; perhaps picking on one or two that they fixed quickly that they can hype in the media with a positive press spin, whilst keeping quiet and trying to hide others. Take a look at some of the large closed source vendors and you'll see this technique being used.

But take Linux for example, that sort of spin just won't happen - it's hard to hide flaws in open source software. Linux users are empowered with the ability to choose - there is more than one Linux distribution out there and so competition will ensure that security fixes are dealt with in a timely and professional manner and will keep everyone honest.

Q:  Security disadvantages?
A:  I'm having a hard time thinking of any real disadvantages, but there are some risks. One risk is that you'll immediately think open source software is more secure than closed source - but it's just as easy to write flawed open source software and just because you can read and audit the source doesn't mean anyone actually has. Some people argue that because you can see the code it's easier to write an exploit and that is true. But a determined attacker is still going to be able to write an exploit for the closed source software, it'll just take longer, and you only need the exploit to be written once.
Q:  What security advice would you give to someone setting up their first Linux web server? How should they get started, and what should they use?
A:  Vulnerabilities in software are found all the time so the critical piece of advice is to make sure that your servers are kept up to date with security fixes all the time. That means keeping track of all those cool utilities you download, install, and forget about like a PHP photo album software I found on my server recently that was a couple of years old and full of security holes. There are still Windows servers being infected with Nimda and Code Red worms because they've not been patched yet.

Part of this is knowing what you've got installed - the IIS worms took advantage of a part of the server that people didn't even know they had. With Red Hat Linux you get this toolbox approach, you get great control and choice over what you install. If you don't want a FTP server then don't install the package, or it's just one command to remove it. It's the same thing when you come to setup a firewall, block any ports you're not using.

After that you can look at intrusion detection systems or things that make sense of all the information going into your firewall logs, there are some great online resources for all this stuff. My final bit of advice is to keep good backups - if your system does get compromised you want to be able to recreate it from scratch.

Q:  Is that the same software you use? And what sort of monster cool, daddy-mac hardware do you run this all on?
A:  I used to roll-my-own versions of Apache and all the things like SSL, PHP, Perl I needed for my web site, it was fun to do so and I learned a lot. But when I started looking after more than one server this got tedious and time consuming and I never kept track of the security issues. Now I make sure all my servers are running some packaged version of Apache, either what comes with Red Hat Linux or our Stronghold product. Since I deal with security all day it's the last thing I want to worry about in my spare time.

As far as hardware goes my dedicated web and email server is a 800MHz PIII in a 1U configuration with 512Mb of ram. Nothing particularly exciting, but then Apache can easily saturate my Internet link way before it starts making an impact on the resources of the machine. My firewall uses the standard iptables netfilter code and a neat Perl utility psad ( which filters and makes sense of all the scans and attempted attacks.

At my house I've recently got into home automation. Everything from my alarm system, my heating, lighting, telephone logging, and more is controlled. I can log in remotely and schedule my TiVo to tape a show, turn on the lights and have a peek who is in the living room - much to my girlfriend's annoyance. Everything is run from a single old PIII box running Red Hat Linux. Home automation is definitely an area I want to do more work on in the future - we can already see the closed source vendors trying to position themselves in this space. The thought of having every appliance in my house running on an embedded Microsoft OS scares the willies out of me.

Q:  And speaking of your new house, I hear that the bug report is rather lengthy. So what gives--is it just built on faulty code, or is the support and update process broken?
A:  Well you expect bugs in new code just as you expect problems with a new house. But there are some other similarities too. My first mistake was not sticking to my specification. When I was looking for a new house up in Scotland I made a huge list of things I wanted and things I wanted to avoid. At the top of my list was "never buy a brand new house" and "keep within the budget". I found a very unique house that was just being built and reminded me of the castle we used to have on the Stronghold box. But it was new, it was outside of the budget, and I threw away my list and invented lots of new reasons to justify it - the lure of being able to run CAT5 cables to every room whilst it was being built was one. Some of the lessons I've learned...

Fixing things takes longer than doing it right in the first place. Taking some shortcuts when laying the laminate flooring may save a few hours work and help you meet your deadline, but it'll end up taking more time when you later have to lift and replace it to fit it correctly.

Beware of off-the-cuff estimates. When the builder keeps telling you your bath will be installed in 2 weeks you shouldn't just accept it - do you believe your engineers who tell you that every new feature you want coded will take 10 days? We've been waiting 5 months for our bath but it'll be with us in 2 weeks.

Make sure your builder is keeping good documentation. For the last month there has been a hole over 9 feet deep in my front drive where we're trying to find a leak in the main water distribution to the house. Unfortunately we've not yet found the main water pipe: it's plastic so we can't detect it from above, and the builder didn't wrap any conductive tape around it and actually can't remember where he laid it. This is just like trying to find a bug in uncommented code you wrote a year ago.

Faults are not Features no matter how much you spin them. Our house is quite noisy at night and the weight of traffic causes strange vibrations-- but to the builder this is a feature as it means we are close to a main road with great access to the motorway network. The electricity board even called the constant overvoltage at our house a feature because it meant our electric kettle would boil water quicker. Of course it's also causing all our light bulbs to blow on a weekly basis and stressing my UPS system-- but at least I can relax and forget about it with a nice, fast, cup of tea.

Q:  On your personal website (, which we'll plug here because it has all sorts of mildly embarrassing information about you that we want 500,000 + readers to go look at (editor's note: especially go check out the information about Mark's performance group, SONIK), you have a page full of favorite quotes that you like to use when speaking. What's your favorite security quote, and why?
A:  Jim Allchin of Microsoft last month told the federal court that "the protocol, which is part of Message Queuing, contains a coding mistake that would threaten the security of enterprise systems using it if it were disclosed". Say no more.
Q:  Red Hat sometimes gets criticized for coming out with advisories later than other vendors. Why do we do that?
A:  Before we release an updated package we have to make sure that the update is going to work and have minimal impact on our users, especially Red Hat Network users who can choose to have security updates automatically applied. This means we have to do extensive amounts of quality assurance to make sure the update isn't going to have any bad side effects. We also like to analyze security issues in depth, to allow us to prioritize it based on the severity and impact to our customers, but also to make sure we can give our customers the info they need to make an informed decision.

Actually these days we're not doing a bad job of dealing with critical issues in a timely fashion. One thing you will start seeing us do in a few weeks is providing status updates - so if there is an issue with a low priority that isn't critical we'll be telling you about it, when we expect a fix, and any workarounds in the meantime.

Q:  Other than fixing your house, what do you do in your spare time?
A:  I'm a workaholic, in my spare time I sleep, eat, and avoid telling my girlfriend what I've done all day.
Q:  If you could leave our readers with only one security recommendation that they should go put in place immediately after they finish reading this interview, what would it be?
A:  Sign your systems up to the Red Hat Network and use the 'up2date' tool to keep them updated - minimize the risk of a new security issue affecting your systems and let us worry about the details.
Q:  Thanks for the time. And good luck with the house.