Issue #19 May 2006

Red Hat Speaks

Ben Levenson, Quality Engineering Team Lead

How large is the QA team at Red Hat? Are you all in one place or scattered across the globe?
We call it Quality Engineering (QE) at Red Hat since the group does everything from software verification, to test and test tool development, to process improvement. QE at Red Hat is smaller than most people might expect considering the size and number of Red Hat's offerings. We're a global group spread from sea to shining sea in the USA, with outposts in Beijing, China; Brno, Czech Republic; Brisbane, Australia; and Pune, India.
What do QA folks do?
A day in the life of a Red Hat Quality Engineer:
7:00 Rise and shine
7:01 SSH to work to see if the system running the over-night kernel stress test still has a heartbeat
7:10 Brush teeth, eat, drink 1st cup of coffee
7:30 Head to the office
8:00 Triage Bugzilla emails and automated test result
9:00 Grab 2nd cup of coffee and head to the lab for a few rounds of installation testing (but don't let anyone see you walk into the lab with mug in hand)
13:00 Break for lunch. Typical crunch-time menu: coffee, cereal bar, and a pack of cheesy crackers
13:30 Check email and begin wild goose chase for obscure piece of hardware required to reproduce a bug. Secure the hardware and begin following the steps to reproduce as described by the reporter -- no luck. Try to reproduce again -- still can't get it to misbehave. Change a few things and try one more time -- still can't break it. Toggle the bug's state to NEEDINFO with comment reading "works for me" and move on.
14:30 Finish up installation testing, create a few new test cases, rack a server in the test lab, verify some bugs, and meet with the product managers and developers to review a few bugfixes which are being considered for the next update release.
18:00 Change IRC nick to "ben_brb" and head home for the evening.
18:30 Grab dinner and sign back on from home. Commit a patch to the test harness, qualify a couple of new tests, update some of the process documentation, and begin verification of the latest packages slated to be released in an erratum.
20:00 Tally up the weekly metrics. Check Bugzilla email and read the follow-up to the "obscure hardware" bug that was tested earlier in the day. Reporter notes that they forgot to tell you about the obscure network configuration that is also required to reproduce the bug.
20:01 ... Sign off.
What do you do? Describe a typical day.
A slight variation of the day described above, but with a few more meetings and a little less technical stuff.
You're the last step in the engineering process before our software reaches the outside world. Do you sleep easily at night?
I started sleeping a lot better once I accepted the fact that it is simply impossible to find every last bug. When we do let one slip through the cracks, we revisit the part of the process where the breakdown occurred and tighten things up for the next round.
What constitutes 'crunch time' for your group? Do you have any strategies to cope? Caffeine IV? Pizza delivery on speed dial?
As I said above, we're a pretty lean operation so it might be easier to tally up the non-crunch time. When we really need to get something done, we just buckle down and do it--weekends, late nights, early mornings, all-nighters.... We've done them all. Caffeine is part of the equation for most of us and the three caffeinated-beverage brewing devices in my cube are a testament to that fact: an electric kettle for tea, a drip coffee pot, and an electric moka for those times when we need a little extra boost.
What impact does the open source philosophy have on your work?
Though the editors at RHM may find this hard to believe, I am doing my best to apply the "release early, release often" strategy to my work within Red Hat. (My responses to these interview questions weren't exactly "early". [ Yeah, but by the standards set by other contributers, they weren't late either -- ed.] ) By starting the feedback loop early in the process, I can pretty much eliminate any chance of misdirected effort.
If I discover a bug in your software, should I tell you about it? How?
"Bugzilla is your friend." If I had a nickel for everytime I heard that in my 5 years at Red Hat.... Anyway, go to and follow the directions.
I want to get more involved then just sending you a bug report. What can I do?
Patches are always welcome -- send 'em if you've got 'em. And then there's Fedora -- lot's of opportunities to contribute there.
We hear you're pretty good with a turntable. Where did you learn to DJ? What have you been listening to lately?
"Pretty good" is a bit of a stretch. Someone was just being nice. I picked up a set of turntables a couple of years ago because I was looking for other ways to engage with the music I like.
"The Other Side of Mt. Heart Attack" by Liars is infectious. I can't get it out of my head. You've been warned. Other than that I've rediscovered a couple of good ones from a few years back while digitizing my music collection:
Richard D. James Album by Aphex Twin
Stolen and Contaminated Songs and Love's Secret Domain by Coil
Lunatic Harness by ?-Ziq