Skip to main content

Sysadmin tales: My worst mistake as a sysadmin

There are mistakes and then there are big mistakes. Thankfully, some have an easy fix.
Image
My worst sysadmin mistake
Image by Tumisu from Pixabay

My first job as a systems administrator was working in the network engineering department at the University of Connecticut. Attending classes and working part-time, I was primarily responsible for some basic DNS and DHCP management tasks, in addition to maintaining a Cacti instance for our network monitoring. This was an excellent opportunity as I was able to access tools, technologies, and platforms that I would not have otherwise been able to work with until after I finished my degree and got a job in the "real world."

This was also where I created my worst production outage ever.

A few months into this position, I was given a small project—configure a bunch of DNS zones within Infoblox. Eventually, I got to the listing for ad.uconn.edu. Unbeknownst to me, ad.uconn.edu was an Active Directory zone that we delegated to our Windows team. However, once this was configured as a zone within Infoblox, we suddenly became authoritative for that zone and had no records.

Of course, I did this while a meeting of all the top-level managers was going on. Suddenly, a large portion of our IT executive staff came into our small network engineering office and huddled around my boss' desk to debug.

After overhearing what was going on, I had to step up and pop my head around the corner to suggest that my creation of this zone might have caused the issue. I deleted the zone from Infoblox, and, sure enough, our delegation started working correctly again.

While bringing down all of Active Directory—and therefore, email, calendaring, and many other vital functions—for a large university was certainly scary, this was a positive learning experience. I had no reason to suspect my change would have this impact, and I did not face any negative repercussions. This made me much more confident in the organization I was working for and helped me understand that honest mistakes happen all the time in the industry.

While I have certainly still caused my fair share of outages, I have always thought back to that event to remind myself that I've done worse.

[ Free download: Advanced Linux commands cheat sheet. ]

Check out these related articles on Enable Sysadmin

Topics:   Sysadmin culture   Career  
Author’s photo

Jonathan Roemer

Jonathan Roemer is a senior DevOps engineer at Drizly with an interest in security, automation, and the human side of IT. He can usually be found hiking or reading a book on his porch. More about me

On Demand: Red Hat Summit 2021 Virtual Experience

Relive our April event with demos, keynotes, and technical sessions from
experts, and sign up to attend breakout sessions June 15–16.

Related Content

OUR BEST CONTENT, DELIVERED TO YOUR INBOX