Skip to main content

Sysadmin tales: My worst mistake as a sysadmin

There are mistakes and then there are big mistakes. Thankfully, some have an easy fix.
My worst sysadmin mistake
Image by Tumisu from Pixabay

My first job as a systems administrator was working in the network engineering department at the University of Connecticut. Attending classes and working part-time, I was primarily responsible for some basic DNS and DHCP management tasks, in addition to maintaining a Cacti instance for our network monitoring. This was an excellent opportunity as I was able to access tools, technologies, and platforms that I would not have otherwise been able to work with until after I finished my degree and got a job in the "real world."

This was also where I created my worst production outage ever.

A few months into this position, I was given a small project—configure a bunch of DNS zones within Infoblox. Eventually, I got to the listing for Unbeknownst to me, was an Active Directory zone that we delegated to our Windows team. However, once this was configured as a zone within Infoblox, we suddenly became authoritative for that zone and had no records.

Of course, I did this while a meeting of all the top-level managers was going on. Suddenly, a large portion of our IT executive staff came into our small network engineering office and huddled around my boss' desk to debug.

After overhearing what was going on, I had to step up and pop my head around the corner to suggest that my creation of this zone might have caused the issue. I deleted the zone from Infoblox, and, sure enough, our delegation started working correctly again.

While bringing down all of Active Directory—and therefore, email, calendaring, and many other vital functions—for a large university was certainly scary, this was a positive learning experience. I had no reason to suspect my change would have this impact, and I did not face any negative repercussions. This made me much more confident in the organization I was working for and helped me understand that honest mistakes happen all the time in the industry.

While I have certainly still caused my fair share of outages, I have always thought back to that event to remind myself that I've done worse.

[ Free download: Advanced Linux commands cheat sheet. ]

Check out these related articles on Enable Sysadmin

Topics:   Sysadmin culture   Career  
Author’s photo

Jonathan Roemer

Jonathan Roemer is a Linux systems engineer with an interest in containerization, automation, and the human side of IT. He is currently an associate manager of containers and hybrid cloud at Datto, and can usually be found reading a book on his porch. More about me

Related Content