Legacies | Ossified Operating Systems

Operating systems are everywhere. They’re likely also all over the place. There are unsupported operating systems running everything from old machinery to everyday devices. And because of the foundational role they play, any change can have cascading effects on the rest of their systems.

How do you handle legacy operating systems? What do you need to be aware of? And how different are operating systems from each other?

00:04 — Scott McCarty
Let me think of how to answer this the right way. So, I think any systems administrator would be familiar with this intrinsically, what is a legacy operating system? Almost like porn, I can't explain it, but I know it when I see it. It's a bit of a pejorative word.

00:18 — Johan Philippine
Legacy operating systems, we don't really need to define them at this point, but we do need to talk about them because as with much of the legacy technology we've talked about in this series, it's everywhere.

00:33 — Scott McCarty
The latest greatest is what we love to talk about and it's what the media gets excited about, but in reality, the vast majority of the economic activity is happening on these legacy operating systems, these things that were deployed months, years ago, and are still running ad infinitum essentially for years and years and years. And they exist in datacenters. They exist on the edge. They exist in embedded devices all over our lives. There's old little devices that we have in our closets, or ones that maybe we still use. And they're no longer supported, no longer getting security updates.

01:07 — Johan Philippine
We are surrounded by operating systems, legacy operating systems. And in today's episode, we excavate the ossified operating systems and hear how to best tackle working with them.

01:20 — Angela Andrews
Yeah, let's do it. This is Compiler, an original podcast from Red Hat. I'm Angela Andrews.

01:32 — Brent Simoneaux
And I'm Brent Simoneaux.

01:34 — Angela Andrews
We go beyond the jargon and simplify tech topics.

01:37 — Brent Simoneaux
We're taking a few episodes to consider the value of legacy technologies.

01:43 — Angela Andrews
If you want to hear this series from the beginning, you can start from our episode, In Defense of Legacy. Today's episode, Ossified Operating Systems.

01:53 — Brent Simoneaux
Producer, Johan Philippine, is here to unearth some fossils.

02:02 — Johan Philippine
Alright, we are revisiting operating systems. We talked about them in the Stack/Unstuck series, and we touched on them in the Updates episode. Today we're going to cover why you might run into those older systems and how to handle them when you do. When we spoke to Aaron Lee a couple episodes ago about hardware, he also had a few things to say about legacy operating systems.

02:27 — Aaron Lee
In the case of academia, we had a device that was running Windows XP in era where the university was running Windows 10. From the most basic, you have the fact that that's a security issue. You cannot have a device running Windows XP because it puts everything at risk, and that's a really good example as to how failing in older technology can put everything at risk because you have something that is vulnerable because it's not being supported anymore.

02:51 — Johan Philippine
Now, I don't know if I'd call Windows XP failing at this point, but it has been around for quite a while. So, why is that such a security issue? We know XP has been around for a while and has had years of patches that must have plugged all of the security holes, right? Angela, help us out here.

03:09 — Angela Andrews
There is no way you should be running XP on your network. It is the weakest link—just because it was patched when it was supported, security vulnerabilities pop up all the time. People discover exploits that you didn't know existed. Think about it, the continuum of every operating system, it's good, it's patched until somebody finds something to exploit, right? That is literally the weakest link on your network. It's one of those things that it is such a huge risk for everything else on the environment. You really should work hard to move anything off of a legacy system. But as we've discussed, sometimes that's not as easy as it seems.

03:58 — Johan Philippine
Now, even when there's nothing valuable on that vulnerable machine, it still puts everything else on the same network at risk, because if someone can gain trusted access to a machine on your network, they'll likely be able to use that access to get to the other machines that had better outer defenses, let's call them. To hear more about how that plays out, you can check out our episode on memes and security. That was a fantastic example of exactly how they did that kind of thing.

04:25 — Brent Simoneaux
Johan's plugging all the old episodes.

04:28 — Angela Andrews
I love that. They all mesh together.

04:30 — Johan Philippine
Now, in Aaron's case, in academia, they were using this machine running XP to run a laser engraver. And, long story short, it was a very expensive piece of machinery that was still functioning properly, but there was no way to have that software be installed on a newer operating system, so they had to keep it running on Windows XP. There was no upgrade path for them that way.

04:53 — Johan Philippine
And that's not an outlier. So that was the case in academia, it's not ideal, but remember what we heard in the open. There are countless devices out there that are running on operating systems that haven't been, and no one plans on updating, anytime soon. A lot of us have experience being forced to work on an older operating system. It feels sluggish in comparison to what we have today. It can be kind of clunky, and on top of the security concerns, there's the issue of forwards compatibility, right? It's rare and without it you're really limited for options. You're not going to be able to install newer software on older operating systems, because there's resource limitations, just the way things are coded these days. You just wouldn't be able to run a lot of the newer software. But there's some good news that our next guest has to share.

05:41 — Angela Andrews
Oh, thank goodness.

05:43 — Johan Philippine
Right. Now Scott McCarty is a Senior Principal Product Manager here at Red Hat.

05:51 — Angela Andrews
Father Linux?

05:53 — Johan Philippine
Yeah, it's Father Linux. Would you mind telling us a little bit more about—

05:56 — Angela Andrews
Literally, that is his Twitter handle.

05:58 — Brent Simoneaux
That's his Twitter. Yeah.

06:00 — Angela Andrews
He is all things Linux. It's amazing. And he's on our podcast. How cool is that?

06:08 — Johan Philippine
Well, Father Linux, he has a history of working with operating systems of all ages, and like he alluded to in the opener, when it comes to operating systems, he's seen it all.

06:23 — Scott McCarty
When you're talking to a customer, they'll often tell you what they're trying to do. They're like, "Hey, we have these thousands of systems, we're trying to upgrade them." And then they pause for a second and they kind of show this face of embarrassment and they're like, "I mean, our stuff's really old, so don't judge me." And I think there's a bit of insecurity on the rest of the economy that's using this legacy technology, if you will. And then there's a bit of, I would say, the very good technology providers understand that well, and they respect that.

06:52 — Johan Philippine
Has this been your experience too, Angela?

06:57 — Angela Andrews
Yes, that has been my experience. When you're super proud of what you're running, what type of new services and things that you provide, and then you say, "Yeah, but we're running a couple of, insert deprecated operating system here," it makes you feel kind of embarrassed because it's like, "Wow, we're doing such great things here, but we're still running some of this old stuff that we have no business running." It really does make you feel bad because you know it shouldn't be this way, but for some reason you're still running it and it's probably out of your hands.

07:34 — Johan Philippine
He was talking about this sense of guilt and shame from the buying side of things when... Again, we've been talking about this, there are a lot of reasons for this to be the case. And like Scott said, very good technology providers, they understand that and they respect it, and if it's needed, help you work beyond that and get to the newer stuff that's needed. But there's also a little work that can be done on the selling side, and we're a little guilty of this too, right? We had a whole episode about the importance of updating your systems and kind of tongue in cheek made some allusions that it's not the greatest thing to keep running on the older stuff.

08:09 — Johan Philippine
The other distinction that Scott made is that there's the two economies that he's talking about. There's the technology industry. We're obviously trying to sell the latest stuff all the time, and then there's the rest of the economy, which they're just trying to make sure that their stuff is running, and if it's running fine and nothing's breaking, then why spend that money on getting the latest and greatest stuff if what they have running is just okay or it's doing the job that it needs to? Now, next we're going to talk about what to actually do when you run into those older operating systems and how to best handle them.

08:58 — Johan Philippine
Like Scott has been telling me, a lot of the times, you don't need to be on the latest and greatest operating system. Windows XP, probably a little bit too old at this point, probably want to look at getting beyond that. But between there and today, there's a sweet spot where there's a lot of advantages for security, reliability, and performance. Scott judges operating systems on a few criteria.

09:25 — Scott McCarty
So, I've always used these 3 dimensions to analyze operating systems. Security is one thing, performance is another. I might make a change, and now the performance is better overall for like a thousand use cases, but one use case it somehow plummets 10% and now my app is broken in some weird, strange way, and now I've affected availability again through performance. And then the third one is I may make a one-liner change and some config file changes or the way it handles a config—some little edge case scenario—next thing you know, a sysadmin logs in at 2:00 AM when there's a problem and they don't understand how the config file works anymore, and now it takes 4 hours to restore the system instead of 2 minutes.

10:04 — Johan Philippine
Those three categories, again, they're performance, security, and risk.

10:10 — Brent Simoneaux
Performance, security, and risk.

10:12 — Johan Philippine
That's right.

10:13 — Brent Simoneaux
Okay.

10:14 — Johan Philippine
Angela, have you ever run into those situations where you've updated to a new operating system and it just breaks everything?

10:20 — Angela Andrews
Yes sir. Not everything is for everybody. You think about if you're running an application and—

10:25 — Angela Andrews
Well, let me back up. If you are going to be updating, upgrading, migrating to an operating system and it is running some sort of software, you better make sure that the vendor of said software says this should run or can run on this operating system. Don't go installing software on new operating systems because you think it's a great idea and the vendor hasn't started supporting it yet. No bueno, don't do that.

10:57 — Angela Andrews
Another thing, if your systems are running, you're on supported OSs, you don't need to be bleeding edge. The features that are coming out in this new operating system, they might not even be applicable to you in certain circumstances. Maybe your system runs just fine and that extra performance that you think you're going to get from this new operating system might not be worth the trouble of migrating to another system and you might break something.

11:28 — Angela Andrews
So, I say all that to say be cautious, read the release notes, understand what's going on, what the underpinnings are, because you don't have to be—

11:41 — Angela Andrews
95% of the time, you don't have to be bleeding edge. You're fine where you are. Security is important, but if you're supported, you're already on a secure operating system that's getting its patches. If your performance isn't suffering, you're fine. And, if it ain't broke, don't fix it. Why introduce risk into your environment if you don't have to?

12:03 — Brent Simoneaux
I mean, that's really helpful criteria.

12:05 — Angela Andrews
It is though. It really is.

12:07 — Johan Philippine
That's the sweet spot we were talking about, right? You don't want to be too soon because there's some rough edges that sometimes need to be filed down and you don't want to be too late because then at that point, there's a lot of things that you're missing out on.

12:18 — Angela Andrews
It is. It is

12:19 — Johan Philippine
You get in that middle zone, you let other people who absolutely need to be on the latest and greatest update first, and they figure out what the bugs are and the edge cases and one of those edge cases might apply to you. And then those other patches come in that smooth the edges out and then you can update.

12:38 — Johan Philippine
Again, we had that whole episode about the operating system's role in the stack and how small changes in the foundation can ripple across the entire structure. Scott again made that same observation with me emphasizing that changes can be a benefit or a catastrophe. So again, when you're working with legacy operating systems, it's really important to keep that in mind when deciding whether or not it's worth making that change.

13:04 — Johan Philippine
What I found really interesting in my conversation with Scott is that we talked about how operating systems are in a way a solution to an even more entrenched problem of legacy hardware. Scott reminded us that if you go back far enough, we don't have any portability from one machine to another, and operating systems helps with that, and that trend has continued.

13:28 — Scott McCarty
If you look at the original computers that could literally only run that program on that specific piece of hardware all the way to modern containers where you can literally run a container almost anywhere, a virtual machine anywhere, you can definitely see a technology trend where portability is a theme and that's good because human beings need portability.

13:44 — Angela Andrews
We sure do.

13:46 — Brent Simoneaux
Wait, say more about that. I don't think I quite get that.

13:49 — Angela Andrews
Well, we live in a hybrid cloud world at this point, and if I'm running an operating system or a container in my internal network and I need to run it into one of the public clouds for some reason, I want to get closer to my customer. Well, maybe I want that container to run out in said cloud. I don't need to rejigger and re-architect just because I'm moving it from my hardware on-prem to some hardware in the cloud. I want it to run the same on any footprint, be it bare metal, be it virtual, be it containers, wherever. So portability is key. We cannot be locked in to the hardware when we're talking about running operating systems. It is just bad form. It takes us back to where we came from. We don't want to go back there again.

14:44 — Johan Philippine
We like consistency, right? If I can do a thing on this machine, it'd be great if I could do the same thing the same way on another machine.

14:51 — Angela Andrews
Tell them, Johan.

14:51 — Johan Philippine
Even if it's not the same, right? Now, the industry worked out some standards and miraculously for once adopted them.

15:03 — Scott McCarty
That compatibility or portability was enabled by something called POSIX, Portable Operating System Interface or something like that. I think they just added an X to the end to make it better. POSIX basically defined all the API calls that you would make into the operating system to list files or open a file, close a file, blah, blah, blah. It was all these basic things to start a process, kill a process, blah, blah, blah.

15:24 — Johan Philippine
Angela, I saw you making a face when he spelled out what POSIX stands for.

15:31 — Angela Andrews
I had no clue what POSIX stood for. I had no clue it was this thing. Everyone knew it existed. We never knew what the acronym meant, but thank you for shedding that light, Father Linux. I appreciate it.

15:41 — Johan Philippine
Now we know. Now, this walkthrough history also has a practical purpose. The POSIX standards haven't changed very much over the years. So if you're faced with a legacy operating system and you're already pretty familiar with the command line, chances are you'll be able to use that operating system without too much adaptation if they're both POSIX-based.

16:06 — Johan Philippine
It's something that he's taken advantage of in his career. As a young technologist in his 20s, he took on a consulting job to help out a printing company. They had a few servers running and some systems to control the printing machines.

16:19 — Scott McCarty
They would make paper products and they were doing all kinds of weird printing that I don't even fully understand it, but I know the machines looked big and kind of dangerous to me as a nerdy software guy. And so you go in the back room and there was a couple normal systems, maybe like a Sun system, which is POSIX, but then they had this OpenVMS system, which was installed by this specific software vendor that controlled this specific printing press. I'd never worked at this before in my life. I'd never even used it.

16:45 — Johan Philippine
Have you ever used OpenVMS or heard of it before, Angela?

16:49 — Angela Andrews
I've used it, but I've never heard it. But I have operated a printing press.

16:53 — Johan Philippine
Ah.

16:54 — Angela Andrews
I sure did.

16:55 — Johan Philippine
Interesting.

16:56 — Angela Andrews
It's a long story. I'll tell you about it.

16:58 — Johan Philippine
We'll get back to that in another episode. Now, he can't be sure that this OpenVMS system was POSIX compliant, but there was still enough overlap for him to work things out.

17:10 — Scott McCarty
I found that these skills translated pretty well. I had worked on a bit of SGIs, IRIX, I had worked on some Solaris, I'd worked on Linux. So I had 3 or 4 Unixes I had messed with and worked on, I guess, at work I should say. So when I went to OpenVMS, it wasn't that big of a jump. I was like, "Okay, you type commands into this weird terminal thing, it outputs stuff once you type these commands, that's good." You kind of get the gist of this. At least it's I'm not typing in numbers or something. It was just like there were commands. So I had to look up the commands, start to understand them.

17:42 — Johan Philippine
Solaris, SGI, those aren't names you hear very much anymore.

17:48 — Angela Andrews
I was going to say, I hear those names.

17:51 — Johan Philippine
Oh, really?

17:52 — Angela Andrews
But you're right. Your caveat of not much anymore.

17:57 — Johan Philippine
So you still do run into them, just not all the time, right?

18:01 — Angela Andrews
Exactly.

18:02 — Johan Philippine
Right. But when you do run into them, I'm assuming you have some familiarity with them because there's a lot of overlap in the syntax and how they're used. Is that right?

18:12 — Angela Andrews
Indeed, a lot of these Linux operating systems, a lot of the core commands are the same. So if you can putt your way around to try to figure out how to make things work, that's that thing where we can kind of translate what we know about this operating system and we can kind of try to use it on this operating system and hopefully, we're successful. If not, we read the documentation. Again, putts around until we kind of figure it out and then you know just enough to be dangerous or just enough to do whatever it is you set out to do on this system. Just enough.

18:49 — Johan Philippine
So you have the fundamentals down of working with operating systems and it helps you go from one to another, and you're not going to be a power user right away for those older systems. But there's that solid base that you can start from.

19:06 — Scott McCarty
You start to realize you can kind of use any computer and kind of at least 60 to 80% of your skills will transfer. Even when you get advanced on Windows, you'd be surprised how fast they can actually learn Linux once they know how to use a computer. They understand what a driver is, they understand what a file is, they understand what network connections are and performance, and security, basic security concepts. So yeah, I would say the concepts are still the most important.

19:32 — Johan Philippine
Yeah, there's a couple things I'd like to bring up here. He says, learning Linux like it's a monolith, and to some degree thanks to POSIX, it is. But beyond the fundamentals, there's a lot of variability between distributions and there are a lot of distributions out there.

19:48 — Angela Andrews
Yes, there are. Everyone has their favorite flavor of Linux. And you head to Reddit, you're going to see some really interesting conversations about which one is the best flavor. So yeah, when you're learning Linux, you're learning the basics, the things that kind of make things Linux.

20:10 — Scott McCarty
There are literally thousands and thousands of Linux distributions, and they all use different versions of kernels and different versions of [inaudible 00:20:17] and different programs and different package managers and all these different—

20:21 — Scott McCarty
It is a flora and fauna. It's like a Cambrian explosion of different pieces of technology. And so yeah, Linux, they're all POSIX. They all kind of look and feel the same, but there's definitely some differences

20:32 — Johan Philippine
With Linux, the operating system opened up and everyone wanted to put their own spin on it. It's the blessing of open source, right? Take it and make it your own. But that can also be a detriment to that portability we've been talking about. From what Scott was telling me, there were a scary few years where it looked like that divergence might keep going, but there's been a successful move to establishing and keeping those fundamental standards.

21:00 — Johan Philippine
Now when so many things depend on operating systems to be stable, it's not a huge surprise to learn that there's a lot of overlap of function over time and across operating systems. That doesn't mean that there are no changes over time, however, and when they do come, they can be quite controversial.

21:18 — Scott McCarty
Almost on one hand, you can count the things over a 25-year career that really rattled me, where I was like, "Alright, I got to go learn this new thing. It's pretty changing." And that's why I think systemd was so big to people. They got mad because they're used to not having to change too much and kind of graft on technology at a speed that they were pretty comfortable with.

21:36 — Johan Philippine
So I'm not very familiar with what the uproar was about over systemd and what systemd does. Angela, you look like you're kind of on the edge of your seat trying to tell us a little bit more. So why don't you go ahead.

21:48 — Angela Andrews
So, when we switched to systemd, which is how we use services and unit files and things like that, we were so used to—

21:59 — Angela Andrews
It's just that thing, you're so used to doing one thing one way, and it is embedded in your DNA. And something comes along and it's like, you know what? We're going to rip all that out and we are going to do this. This is how we're going to run with systemd. We're going to have all these different unit files. They do all these different things. You're going to have to start/stop services with this whole new syntax.

22:23 — Angela Andrews
And I remember some of my colleagues being like, "Oh, this is so dumb. Everything worked the way it was. Why are we doing this?" And it didn't bother me one way or the other. It's like, "I'll just put a sticky note. Just remember you have to run this command." And then it just kind of opened up. Once you understood more of systemd and what it was actually doing, it was a move in the right direction. But that knee jerk, boy, people were really up in arms. But again, we're not comfortable with change too much in technology.

23:00 — Brent Simoneaux
So I think from this episode, we're kind of learning that one, operating systems are everywhere.

23:08 — Angela Andrews
And everything.

23:09 — Brent Simoneaux
Everywhere. And that you're going to run into a lot of legacy operating systems in your career as a technologist. And I think what I'm hearing from what Scott was saying is that maybe that's not as scary as it might seem at the beginning. And there are a lot of, I don't know what to say, valid reasons for it, or a lot of logical reasons for why that is. Is that right, Johan?

23:41 — Johan Philippine
Yeah, absolutely. So he mentioned at the very top of the episode that there are a lot of devices out there running on operating systems that don't get updates. And it could be an internet of things (IoT) device that's in your house. It can be your car is running on an operating system that doesn't often get updated. It can be an old computer, it can be a cell phone, a tablet, and that's just in consumer devices as well. You go into commercial, into manufacturing, there's just going to be devices everywhere that—

24:11 — Brent Simoneaux
Enterprises.

24:12 — Johan Philippine
... because they don't need to update that much, they don't, right? And so if you start working for the companies that make and update these devices that have the older operating systems on them, you might run into being the person that has to finally put that update in place.

24:35 — Johan Philippine
Now, that being said, let's keep going, let's take that a little further, right? Recap of the episode. It's going to be everywhere. It might be part of your job to run into it. It's not that big of a deal, right? Because—

24:48 — Angela Andrews
Not anymore.

24:48 — Johan Philippine
Not anymore.

24:49 — Angela Andrews
With the internet making all of this information so accessible, with generative AI and going in and asking it questions and it providing you with these really in-depth answers, you don't have to sweat it. I think gone are the days where you have to worry about this old operating system and I have to move over to this new one, and I'm not really sure and I'm not really comfortable. I think those problems and those issues are starting to go away slowly but surely. I think we're more comfortable with change more than we used to be because I think things are changing so quickly. We've gotten used to things not staying the same for very long. And thanks to the internet, we can learn those new things at the push of a button.

25:41 — Johan Philippine
And thanks to standards like POSIX, there aren't—

25:42 — Angela Andrews
Yay, POSIX.

25:44 — Johan Philippine
... that many new things to learn when you're going from one operating system to the next. Or at least you can get started with the fundamentals and you can build on from there, right. On our next episode of Compiler Legacies, Kim Huang is going to call on the API experts.

26:05 — Angela Andrews
Ooh, I can't wait for that. What did you think about today's episode? We are talking about the ossified operating system. I'm sure you're running some somewhere in a closet at your job that no one wants to talk about because yes, we are embarrassed. We would love to hear what version? What version of said operating system are you running? Hit us up on our socials @RedHat using the hashtag #CompilerPodcast. We would love to hear it.

26:38 — Angela Andrews
And that does it for the operating systems episode of Compiler Legacies.

26:43 — Brent Simoneaux
Today's episode was produced by Johan Philippine, Kim Huang and Caroline Creaghead.

26:49 — Angela Andrews
A big, huge thanks to our guests, Scott McCarty and Aaron Lee.

26:54 — Brent Simoneaux
Victoria Lawton keeps our operating system elitism in check.

26:59 — Angela Andrews
Our audio engineer is Chris Clarke. Special thanks to Shawn Cole. Our theme song was composed by Mary Ancheta.

27:08 — Brent Simoneaux
Our audio team includes Leigh Day, Stephanie Wonderlick, Mike Esser, Nick Burns, Aaron Williamson, Karen King, Jared Oates, Rachel Ertel, Devin Pope, Matias Faundez, Mike Compton, Ocean Matthews, Paige Johnson, and Alex Traboulsi.

27:24 — Angela Andrews
If you like today's episode, please follow the show, rate the show, leave us a review, share it with someone that you know. It really helps us out.

27:32 — Brent Simoneaux
Alright, we will see you next time.

27:35 — Angela Andrews
Take care everybody.

27:37 — Brent Simoneaux
Bye.