How Do We Make Updates Less Annoying?
Updates. They take time out of your day and your devices out of commission. That’s about the extent of it for an end user. But for organizations? Updating their systems is a big deal. And forgoing regular updates is a recipe for disaster.
"If it ain’t broke, don’t fix it" isn’t a good option for enterprise IT. Your system may keep running. but if you haven’t updated in awhile, you may be missing some stability and performance improvements—and probably some security patches too. Regularly updating is a good habit. How do we make it less annoying?
00:02 - Johan Philippine
Angela, Brent, how do you feel about updating your software?
00:08 - Angela Andrews
Hmm, well, it depends. Usually it's like, I'll do it as soon as I'm done for the day so I'm not really impacted, and I try to stay on top of it because I know it's important. I do my best. I think I'm okay with it.
00:24 - Johan Philippine
What about you, Brent?
00:26 - Brent Simoneaux
I am what I would describe as a smug updater.
00:30 - Johan Philippine
A smug updater? Oh, do tell me what that means.
00:33 - Angela Andrews
Tell me more.
00:35 - Brent Simoneaux
I think I understand the importance of updating and so I just feel really good about doing it. But not only that, for me, it feels like I'm being productive, but I don't really have to do much of anything. So I feel like I'm getting this like really important task done, but how much effort does it really take?
01:00 - Johan Philippine
I am a... I guess I'll call it a reluctant updater where, again, I know that it's super important to keep things up to date, but I get... I just get so annoyed by those notification messages saying like, "you've got a new update" and it's like, "ah, geez, I just did this. Do I have to do this again?" And then sometimes you roll out an update and it causes more problems than it fixes.
01:22 - Angela Andrews
All heck breaks loose. I know about those updates too.
01:30 - Johan Philippine
And this is all great for us as end users. But for businesses, updates are a much bigger deal. There are security implications, not just for your business, but for all of your customers as well. Right?
01:45 - Angela Andrews
01:47 - Johan Philippine
That's right. And doing it right takes planning, doing it wrong has consequences, and not doing it at all is just a recipe for disaster. So. If updating is something that we have to do all the time, how do we make updates less annoying?
02:06 - Brent Simoneaux
All right. Let's dig in. This is Compiler, an original podcast from Red Hat.
02:16 - Angela Andrews
We're your hosts.
02:17 - Brent Simoneaux
I'm Brent Simoneaux.
02:19 - Angela Andrews
And I'm Angela Andrews.
02:20 - Brent Simoneaux
We're here to break down questions from the tech industry, big, small, and sometimes strange.
02:27 - Angela Andrews
Each episode, we go out in search of answers from Red Hatters and people they're connected to.
02:33 - Brent Simoneaux
Today's question: How do we make updates less annoying?
02:37 - Angela Andrews
Producer Johan Philippine is here with the latest update.
02:44 - Johan Philippine
So we're going to be talking to two people today, Ben Pritchett and Vania Vergara, they're both going to help us out a little bit with the update landscape. First up, Ben Pritchett. He just started at IBM, but up until very recently, he worked on upgrading some of Red Hat's internal infrastructure. And he helped me out at the top with some definitions. Now, Brent?
03:09 - Brent Simoneaux
03:10 - Johan Philippine
We're going to do your favorite thing. We're going to get out the whiteboard real quick.
03:14 - Brent Simoneaux
I feel like every time we talk to you, Johan, we have a whiteboard in front of us.
03:20 - Angela Andrews
I always have a whiteboard in front of me.
03:23 - Brent Simoneaux
I know... I know you do.
03:23 - Johan Philippine
I like whiteboards. What can I say? They're useful.
03:28 - Brent Simoneaux
All right, let's do this.
03:30 - Johan Philippine
We're going to write X-dot-Y-dot-Z on the whiteboard.
03:36 - Brent Simoneaux
03:36 - Johan Philippine
Now these letters together, they represent a software's version number, and how it's broken-
03:41 - Brent Simoneaux
03:43 - Johan Philippine
That's right. Now, "X" all the way on the left. That's the big version number, major version. This one doesn't change very often. Think RHEL [Red Hat Enterprise Linux] 8 or RHEL 9, Windows 10 or Windows 11. These are big years-long updates that change a lot of the features and kind of completely change the architecture of, say, an operating system or the software and how it works. And often enough it's a big enough change that you kind of want a clean break from one installation to another. Angela, you can correct me if I'm wrong. There are ways to update from one to the other, but it can get pretty messy. Is that right?
04:26 - Angela Andrews
It can. Yes. The clean break option is always recommended.
04:31 - Brent Simoneaux
Would you say that it's an extreme update?
04:38 - Angela Andrews
I would, and so would you, and so would Ben, actually.
04:45 - Ben Pritchett
I would generally summarize these as major re-architectures in the offering.
04:50 - Johan Philippine
Angela was right. Ben agrees. Now "Y" stream updates are that middle number. They're smaller updates, less of a big deal than the major version updates, and they're more frequent.
05:05 - Ben Pritchett
Generally newer features. You could even see some deprecation of features as you kind of go through different "Y" stream versions.
05:13 - Johan Philippine
That's still going to be a pretty big change that you have to really plan for, right? Because if, if you want to use those new features, you’ve got to plan for integrating them to the rest of your software, the rest of your system. And if something is being deprecated that you're actively using, you have to figure out, "Well, okay, how am I going to replace this feature? How am I going to disconnect everything that previously was relying on it?" It can be a pretty big deal.
05:43 - Brent Simoneaux
So as a user, you should expect to see some difference here.
05:47 - Johan Philippine
05:49 - Angela Andrews
And you should also read the release notes for every update that comes out because you don't want to be caught by surprise.
05:57 - Johan Philippine
And as a user, you want to know what's coming on. But as a business, as an enterprise where your own infrastructure is running your application, that's running your website, you want to be absolutely sure that you know what's being added or released before you hit that update. Because if some, if something changes and you weren't aware of it, then your system could go down, then that's not good.
06:23 - Brent Simoneaux
All right, we've got our "X" stream. We've got our "Y" stream. What about the "Z" stream?
06:28 - Johan Philippine
The "Z" stream. So those are at the very end. They tend to be the most frequent updates. These are small updates that typically don't change the functionality of a system, but they're very important for other reasons.
06:40 - Ben Pritchett
These are kind of the general, like bug fix or smaller feature updates.
06:45 - Johan Philippine
Much smaller in terms of the scale, but much more frequent and still very important.
06:50 - Angela Andrews
Right. So, for some of our listeners, maybe this X, Y, and Z stream may sound a little bit new to you, but if you're used to hearing major, minor and patch, that's basically the same thing we're talking about here. Just a different nomenclature.
07:11 - Johan Philippine
So we have those in mind, X stream, Y stream, Z stream, but rolling them out is still the issue. Vania Vergara, she's a technical account manager here at Red Hat. She gave me, let's go with the dream scenario for updating your infrastructure.
07:27 - Vania Vergara
Oh, the dream is to click a button. That's the dream.
07:33 - Johan Philippine
You push a button and then the update just kind of rolls out, and you don't have to think about it. And it's all easy.
07:40 - Angela Andrews
It just works easy-peasy.
07:42 - Brent Simoneaux
But that's not how this happens in real life though.
07:46 - Johan Philippine
07:48 - Vania Vergara
It's an issue that is most common because we are always updating our customer's versions. So, we have a lot of problems there because sometimes the systems are really complex and to upgrade is not an easy task.
08:06 - Johan Philippine
So, it's easy for us as consumers to just push a button and then the iPhone update rolls out or your computer update rolls out and you don't worry about it. But on the other side, getting that update ready, writing it, rolling it out. That can be really complex because it's, again, in a lot of situations, it's not as easy as just pushing a button.
08:31 - Brent Simoneaux
Where does that complexity derive from?
08:33 - Angela Andrews
Well, because usually you are the person who's responsible for approving the updates to be rolled out to the infrastructure. You have to make sure that they do no harm. There's a lot of interoperability between systems and functions. So you have to test them and make sure that you don't break anything. Enterprise systems have a lot of components that really need to work together. So, yes, there is a lot that goes into it, research, understanding... It is a really complex operation and once you're okay, then you go ahead and approve it and you push that button. And now it gets dispersed through your environment.
09:19 - Johan Philippine
So Vania explained to me just how necessary regular updates are.
09:25 - Vania Vergara
Well, it's really important to update because, we have a lot of issues with security nowadays. And not only security issues, but if you want to have a system that is stable and secure, if you want to have a consistent infrastructure, you should have this updated.
09:48 - Johan Philippine
We talk a lot about security, but that's only really one aspect of keeping things up to date.
09:54 - Angela Andrews
That's true. And she talked about stability. So, they're not necessarily one and the same.
10:02 - Angela Andrews
Say if there is a security vulnerability, that's out there, it's out in the wild. You want to make sure that you've protected the systems in your environment from that vulnerability. Stability could be, like we said, like a bug fix where something really wonky is going on in a particular OS [operating system] or software. You want to make sure that the end users have a stable product that they're using. So, someone's gone through the work to try to fix something to make sure things work as they should. If you're that end user and you're using said product, and you're having all of these symptoms, you really do want your system to be stable. So, making sure your systems are both stable and secure is really important, if you're managing an infrastructure.
10:49 - Brent Simoneaux
I kind of have this feeling that maybe... a lot of organizations, maybe it is stable for them, or it feels stable, or it feels like everything's still working. And so the cost of updating might be worse than the perceived stability of the current system.
11:10 - Johan Philippine
Let's dive into that a little bit more all right. I asked Vania exactly about that same thing, about the "if it ain't broke, don't fix it" and how bad can that get?
11:19 - Angela Andrews
Famous last words.
11:20 - Johan Philippine
It turns out it can get pretty bad.
11:23 - Vania Vergara
They have some companies that have system 20 years old, 22 years old, they just don't touch it because it works and they don't have the people to interact with the system. They don't have the knowledge. So, if it works, leave it be. But you have a lot of issues of vulnerability with this system. And you have to take care of this.
11:52 - Angela Andrews
Boy, I felt this in my spirit because this is more common than you think.
11:59 - Johan Philippine
11:59 - Brent Simoneaux
11:59 - Angela Andrews
Yeah. This is really an issue that a lot of companies and organizations have to deal with. They're running these ancient operating systems and bespoke software that someone wrote and have since moved on in one form or another, and it works. It's running something that's critical to the organization, but there's no knowledge on, on how to fix it if it does break. So what do you do? You leave it alone and cross your fingers and your toes.
12:34 - Brent Simoneaux
And you're saying this is not uncommon, right?
12:37 - Angela Andrews
Not in my experience.
12:38 - Johan Philippine
Yeah. Not in Vania's experience either.
12:41 - Vania Vergara
The more you wait... Difficult, it will be to upgrade. And sometimes it will be just, you can't do it. It will be just unadvisable to upgrade. You have to build a new system entirely. It will take more time, more effort, and more money.
12:59 - Johan Philippine
It takes engineering time. It takes money to buy the software licenses to buy the hardware upgrade. It can get very expensive to update these systems and even more so when you have to rebuild it from scratch.
13:20 - Angela Andrews
So, they have to make that decision and it's not an easy one to make. And sometimes that can get kicked down the road to say another manager, or there's been some change in an organization and you've come in and all you hear are the stories, right? Don't touch this system. There's no support for this system. No, it hasn't been updating, but it's still running X application. So, this isn't... I'm sure companies don't do this lightly, but it's a lot holding them back from making that decision.
13:59 - Brent Simoneaux
There's also all of the other work that your team is responsible for as well. All of that stuff doesn't go away.
14:09 - Johan Philippine
Ben was telling me that even when you know that he and his team were working on a major update, they still had their other job to do.
14:18 - Ben Pritchett
There's never one thing going on in an IT Org. If I had to estimate it was roughly 50% of the engineering team's time focused on build-out of this new platform. Obviously we had to continue to support apps.
14:35 - Johan Philippine
So he had to split his time evenly between supporting existing infrastructure and building out the major update. Vania reiterates, why that 50% can take a while.
14:46 - Vania Vergara
So we don't have people to go through with the process of upgrading. Because some people believe or some specialists or professionals believe that an upgrade is just to click a button. So, I will upgrade my Windows. I will click a button and everything's okay. It's not quite the same. When you have complex systems or systems that are integrated with several platforms, different system- operational systems. If they have this kind of complexity, you have to test all the integrations. And if you operate one part of the system, maybe the other parts won't communicate themselves. And it's really annoying, and you don't want to discover it on production.
15:43 - Angela Andrews
She said annoying. Okay. Annoying. That's a nice way to put it.
15:51 - Johan Philippine
She was being a little diplomatic there, but yeah...
15:54 - Brent Simoneaux
She's not wrong.
15:55 - Angela Andrews
She is not wrong. I have another word for that, but annoying will do.
16:03 - Johan Philippine
It's annoying if you discover it before it rolls out into production, but if you discover it after it rolls out into production, then annoying is not the word that I would use either. But there's all these integrations that you have to test. Angela, you were telling us about this earlier. You have to make sure that all the different parts of your system still work and that they're communicating with each other. And if you don't get all that testing right, you could be in a lot of trouble.
16:35 - Vania Vergara
So many things that could go wrong. If you look from the application side, maybe you upgrade your system in the database wasn't updated. If you are updating a software, the driver that your operational system requires change it and you don't have it. So it won't work. Sometimes you require a kernel specification that you don't have so you have to recompile kernel or choose another line of version. So it's really a lot of things that could go wrong. And as I said, I have a certificate, I used TOS 1.1 and now I want to use 1.2. Well, you have to change something in your proxy or in your file. And that's a lot of things that you have to be aware of.
17:36 - Johan Philippine
She was just listing a few of the things that you need to keep track of but there are many more things.
17:45 - Angela Andrews
So, I have a customer who just ran some yum updates and it broke their (censored). I'm not going to say what customer it is, but they did a yum rollback, which is not supported, but they could not get this system up to save their lives. So now that the system's up support was like, "oh, we need the sos report." And he was like, "Bruh, I'm not reapplying these patches just to get you a sos report. And I gave you this log. I gave you that log. Y'all need to figure this out. I will not reapply. You have to tell me what broke it. I need to know what broke it." So a patch literally brought a production system down now to their credit they didn't... or discredit, they didn't test it, but that's a whole ‘nother story. Yeah. We're not going to go into that.
18:35 - Johan Philippine
So there's a lot of things to keep track of. Well, Vania and Ben, they've got a little bit of advice for us on how to avoid things going wrong.
18:42 - Vania Vergara
Nowadays nothing is really good for a long time. We encourage that every six months you have visit the release notes to see upgrades that are needed and to plan and put some time for your team to do the planning necessary to upgrade the systems, because it's not an easy task. You have to deal with system upgrade as you deal with a project of a software. So you have to have planning, you have to have timeline. You have to have people and you have to have, of course the material necessary for the upgrade.
19:30 - Johan Philippine
So the first thing she is suggesting that people do—that companies do—is to treat and update with the seriousness and the respect that it needs. Deal with it as you would any other software project that your company might be working on. Right? Allocate resources, put together a timeline, allocate engineering work to make sure that it gets rolled out properly.
19:51 - Brent Simoneaux
You need to plan for it like you would any other important task that your team needs to do.
19:59 - Johan Philippine
Now, part of that planning work is working with the affected teams, because it's not just your sysadmins that are rolling out these updates or upgrades by themselves. There are, especially for feature changes and stuff like that, you're going to be working with the teams that are integrating software into your system.
20:21 - Angela Andrews
This is a team effort. You are so right about this. You can't do it in a vacuum.
20:27 - Johan Philippine
And Ben had a little bit of advice on making sure that gets done properly.
20:33 - Ben Pritchett
We worked early and often with them to notify that, "Hey, at some point we need to be planning around migration. Like we will need the apps that exist here to end up over here." And so that was part of the effort as well, is kind of preparing app teams for platform adoption, understanding their migration pain points. And, and what were some of the features we could build into the platform to help with that.
21:00 - Brent Simoneaux
It's not just your team. That's going to be affected by this, that a lot of teams are dependent on this as well.
21:09 - Angela Andrews
So, Johan, how do we make this easier on everyone?
21:13 - Johan Philippine
Well, lucky for us, Ben and Vania have a lot of experience with this and they've got some advice to share. Let's hear Vania's top recommendation for making updates a little less annoying.
21:24 - Vania Vergara
One of the things that will be crucial for your success is automation because automation, you diminish the human error and you guarantee the process is done equally all the time. So I'm really a defender of automation. I really enjoy automation all the process, because when you automate, you have like a contract of what you have to do, and it won't be able to forget, "oh, I forgot this step" no, it's automated. Let the automation to do this job.
22:04 - Johan Philippine
So, automating your testing process really makes sure that when you're rolling out an upgrade or an update that all the pieces of your software are still communicating with each other the way that they should. That you're not breaking anything with this update. It's something that a lot of people do manually, which sounds like a nightmare, especially for very large systems, but automating it really seems to make it much more of a breeze.
22:32 - Ben Pritchett
You can imagine as you start to serialize and validate each and every kind of cluster that you offer, there's more and more kind of overhead for how you do that. So yeah, in terms of scaling, you definitely have to be on top of, I would say you have to be very good at automating the process because you certainly don't want to have a human button clicking and running scripts in order to execute and validate these kind of things. So anything you can do around automated validation is certainly helpful in this regard.
23:06 - Johan Philippine
So, you'll remember earlier in the episode, Vania recommended that companies upgrade their systems, update their systems every six months or so. Now for enterprise systems. These are big, they're complex. We've been talking about how there's a lot of moving parts that seems like quite a lot, quite frequent to do it every six months. But the best way to stay on top of that is to build out processes that systematize your updates. That means documenting the systems. Making sure that everything that needs to be checked is going to be checked. Building out that checklist and building out that automation and all of that takes a lot of time and effort, but you're front loading that work so that you can spend less time on each individual update without also worrying that update's the one that's going to be the one to break your production system. Now, she also has some advice that seemed a little counterintuitive because we've been talking about making sure that you get updates as soon as you can get them, but that's actually not something that she always suggests that you do.
24:19 - Vania Vergara
If you have a new bug or new issues, what advice to my customers is wait for the others to discover the problems. You don't have to do it on the first. Let's wait one month, two months and then we can upgrade your system.
24:35 - Johan Philippine
So she's talking about upgrades here, something that's not absolutely necessary like a security vulnerability or a bug fix or something that's affecting your user experience, do get those as soon as possible. But if it's an optional upgrade, go ahead and wait a little bit. Now, the final piece of advice that she has is to have like a developer or staging environment to test everything before it goes live so that if something does go wrong, it doesn't affect your business.
25:10 - Vania Vergara
They have this side environment, that's just a clone for production. They just clone everything that they have with difference because it's not a production really. It's not on the production so they can test all the integrations.
25:29 - Angela Andrews
That's the importance of having that test environment, but think about the cost that comes for having a duplicate of everything that needs to be updated. So there are actual costs and resources involved with that. So again, when we run up against those legacy systems that are sitting back there in the data center that don't have a clone, they don't have the ability to be cloned or duplicated and reproduced easily. We live in a time with containerization and virtualization. That really makes that easier, but that doesn't mean that there is still not costs involved with having these disparate environments to do this in.
26:21 - Brent Simoneaux
All right, Angela, Johan, let's come back to our question for this episode. How do we make updates, less annoying? Johan, I'm wondering, after talking with so many people about this topic, what have you learned?
26:39 - Johan Philippine
Updates are annoying, but there are things that you can do to make them easier and you want to make them easier because you want to make sure that you do them regularly? That if you keep them difficult and you give yourself a reason not to do them, then you're just going to end up in trouble. So it's really important to get those updates done for security reasons. It's really important to get them done for stability reasons and just to remain competitive in the overall tech landscape anyway. Because everything moves so quickly that if you leave your infrastructure behind to the point where you can't really upgrade it very much anymore, then you're leaving things on the table that could help you out in the long run.
27:26 - Brent Simoneaux
Angela, I'm curious, where's your head at, in all of this?
27:31 - Angela Andrews
I have a reinvigorated respect for people who have to go through these activities and these behaviors because we as—and I consider myself an end user at this point—we as end users don't understand just how much goes into it, how much planning, how much preparation to get it right, or as right as it could possibly be and the testing and the cross team communication. And there's so much that goes into that. I just have this respect because I was on that other side for a while, but listening to Vania and Ben, I really do have a greater appreciation for the things that go on that we don't see as end users and that we get kind of flustered by because, "This isn't working how I expect it. What the..." No one did it to you personally. Trust me when I tell you. So I think all of us as end users really needed to hear what they had to say. We wouldn't have the innovation that we have without folks like them.
28:45 - Angela Andrews
So listeners, after hearing what you just heard from our amazing guests, you have to tell us. What are some of your update, horror stories? I know if you're listening to Compiler, you probably have a couple of them up your sleeve and we want you to tell us all about them. So hit us up on Twitter or Instagram @RedHat always use the hashtag #CompilerPodcast. We really need to hear: What are some of your update horror stories? Let us know.
29:20 - Angela Andrews
And that does it for this episode of Compiler.
29:23 - Brent Simoneaux
Today's episode was produced by Johan Philippine and Caroline Craighead. Victoria Lawton pushes all of our buttons and makes sure we're all up to date.
29:35 - Angela Andrews
Our audio engineer is Elisabeth Hart. Special thanks to Shawn Cole. Our theme song was composed by Mary Ancheta.
29:43 - Brent Simoneaux
A big thank you to our guests, Ben Pritchett and Vania Vergara.
29:48 - Angela Andrews
Our audio team includes Leigh Day, Stephanie Wonderlick, Mike Esser, Laura Barnes, Claire Allison, Nick Burns, Aaron Williamson, Karen King, Boo Boo Howse, Rachel Ertel, Mike Compton, Ocean Matthews and Laura Walters.
30:07 - Brent Simoneaux
If you like today's episode, please follow us, rate the show and even leave a review. It really does help the show.
30:15 - Angela Andrews
And we would love to see it, and hear from you. So thanks for listening. See you next time.
30:20 - Brent Simoneaux