My first job as a systems administrator was working in the network engineering department at the University of Connecticut. Attending classes and working part-time, I was primarily responsible for some basic DNS and DHCP management tasks, in addition to maintaining a Cacti instance for our network monitoring. This was an excellent opportunity as I was able to access tools, technologies, and platforms that I would not have otherwise been able to work with until after I finished my degree and got a job in the "real world."
This was also where I created my worst production outage ever.
A few months into this position, I was given a small project—configure a bunch of DNS zones within Infoblox. Eventually, I got to the listing for ad.uconn.edu. Unbeknownst to me, ad.uconn.edu was an Active Directory zone that we delegated to our Windows team. However, once this was configured as a zone within Infoblox, we suddenly became authoritative for that zone and had no records.
Of course, I did this while a meeting of all the top-level managers was going on. Suddenly, a large portion of our IT executive staff came into our small network engineering office and huddled around my boss' desk to debug.
After overhearing what was going on, I had to step up and pop my head around the corner to suggest that my creation of this zone might have caused the issue. I deleted the zone from Infoblox, and, sure enough, our delegation started working correctly again.
While bringing down all of Active Directory—and therefore, email, calendaring, and many other vital functions—for a large university was certainly scary, this was a positive learning experience. I had no reason to suspect my change would have this impact, and I did not face any negative repercussions. This made me much more confident in the organization I was working for and helped me understand that honest mistakes happen all the time in the industry.
While I have certainly still caused my fair share of outages, I have always thought back to that event to remind myself that I've done worse.
[ Free download: Advanced Linux commands cheat sheet. ]
