Untangling Networks

  |  Compiler Team   自動化と管理 ネットワークの自動化 専門能力の開発

Compiler • • Untangling Networks | Compiler

Untangling Networks | Compiler

About the episode

At home, connecting your devices to the internet is a pretty simple process. But setting up and running an enterprise network is an exercise in extreme organization and constant vigilance.

Elle Universal, Senior Product Marketing Manager at Red Hat, outlines all the considerations that come with large-scale networks that keep thousands of devices online—but also keep them protected from intrusion.

Compiler team Red Hat original show

購読する

Subscribe here:

Listen on Apple Podcasts Listen on Spotify Subscribe via RSS Feed

トランスクリプト

There are so many of those edge devices we talked about in previous episodes that never get updated, because who would've thought to update, what did you say? An aquarium? An aquarium thermometer. Wow. Yeah, for one of those giant fish tanks with hundreds of gallons of water in it. Oh, my God, that's great. This is Compiler, an original podcast from Red Hat. I'm Emily Bock. And I'm Jennifer Scalf. On this show, we go beyond the buzzwords and jargon and simplify tech topics. This season, we're covering the fundamentals of IT infrastructure. And on today's episode, we gaze deeply into everyone's favorite spaghetti monster, networking. Networks are all around us connecting our devices to each other and the greatest network of all, the internet. And while many home networks are as simple as plugging in your combo modem and router, enterprise networks are a little more complicated. So, today we're going to dive into why enterprise networks are so complex, what pitfalls to avoid, and hear a few tips to help you untangle the potential mess of ethernet cables in the server room. Jennifer, what's your experience been with setting up network infrastructure? I'm trying to think of a spot within the network I haven't touched at some point in my career or outside in my personal life. I think I've set up just about every kind of open source or proprietary router, switch, anything in my house in the last 20-some years and then out in the world at work, or I've watched other people do it. I mean, from taking things out of the box, running the cable. We used to crimp the cables ourselves back in the day. I don't know if anybody does that anymore, but we used to crimp the cables to plugging all that in and then obviously setting up the firewalls. And my last job, the last thing I was doing was actually going into the operating system and configuring the drivers and the kernel modules and all of that. So, yeah, I've really been all over on the "network" as ever anybody wants to define that. Oh, yeah. I think anyone who's been involved with networks has some kind of horror story. I know my previous company, we hired a network engineer and his first day was the day we moved offices and he had to move the entire server room from one building to another. I can't imagine. No, no. Well, that wasn't on purpose though, right? It was before my time there, so I can't be sure, but man, that is a next level cruelty. Threw them right into the fire. I mean, but that's what we used to have to do. Now everything is a little bit and we're going to talk about it segmented and there's lots of access control lists and all kinds of ways we separate things out. But back in the day, yeah, if you were in the network, you were running the cables, you were doing everything. And then if somebody had to move everything, it was a very manual physical process. Oh, yeah. Had a good amount of upper body strength for some of those old switches. For real. Sometimes you might not really know you need a specific network engineer until you need one. Oh, that too. Maybe that's what happened. Everybody has their horror stories and I'm sure we'll hear a few more as we go through this episode too. We spoke with Elle Universal, a senior product marketing manager here at Red Hat and she started us off by emphasizing the disparity between home and enterprise networks. So, essentially your home network is way less complex than what you'd be setting up for an enterprise just right off the bat. The complexity is a completely different scale. So, your home network serves as a single location with a handful of devices typically managed by a service provider whereas your enterprise network supports dozens of different use cases, devices across all different environments. And typically you have a team of people that is responsible for managing all of that. So, the best way that I've heard this explained, especially to someone like me who was new to all of this was if your home network is a bicycle, it's simple, it gets you where you need to go and it's easy to fix by yourself, but your enterprise network is your commercial jet. So, redundant systems, high capacity, and it requires a specialized crew and standard operation procedures. I think that illustrates it pretty well, actually, going from a bicycle to a commercial jet, because that's a huge jump that really illustrates the difference. What is behind that leap in complexity? I love all of our analogies and we certainly love our airplane analogies. That has come up pretty commonly actually. I don't know why. Yeah, but it is a good one. Again, it's a good one. You're going from a few people in a house that just need maybe their gaming, maybe they're working from home off and on now. They've got a few laptops, they've got a few tablets, maybe their phone connects at home. We don't always use your data plan at home. Two, an analogy of a jet, even if we just put the people on there and they're getting on the network on there, they then are exposing themselves to the other people on the network, the other devices on the network. You're getting perfect strangers coming and going. I mean, I don't know, maybe somebody throws a party at their house and they also have a number of strangers. So, maybe there is some overlap too there, but for the most part there's specialization. There's the amount of people joining on your home networks, you can get away with having maybe one router and a repeater if you have a two or three story house living in a condo. Yeah. If you're talking about across a single work site, you might have four or five different buildings and 100 different people with seven different devices just from the company alone. So, we're getting into a lot more complexity right there. Oh, yeah. I think you hit the nail on the head there too with there's also levels of complexity too like one building is going to have a much higher level of complexity than like your house, but also like a company like IBM, 100,000 people throughout the globe. That's going to be exponential step above in terms of complexity. Exponential is a great word for it. Exactly. That factorial, you put the little exclamation point at the end. It's all the combinations. And so, I think that level of complexity growing is true for a single network as well. A lot of companies have more than a single office or a storefront. When an office is expanding, whether it's five or 50 locations, whatever it might be, it becomes a wide area network. And without standardization, so without it kind of being a rinse and repeat of a setup and having essentially a golden standard of how you're configuring these networks, it's going to be impossible to kind of have compliance intact and then also being able to secure or fix these networks because if they're all operating differently, then the team that is responsible for kind of keeping everything up and running the way it needs to be is going to have to be sorting through all these different standards or best practices and know how each one runs versus saying, "Nope, everything should run exactly like this." And if it doesn't, then there's the problem. So, it kind of helps maintain those networks by having that essentially golden standard of setup and configuration. Standardization strikes again. We talked about this in the infrastructure episode too, but it seems like there's that golden standard of setup and configuration and we'll hear that more throughout this episode, but what would something like that look like? That would look like a dream come true for your network admins. Oh, my goodness. I'm thinking about the amount of times that I had to run something like TCPdump. There's all these wrappers for that underlying ancient troubleshooting script that we used to run and it still holds its weight. So, we're running TCPdump, right? We're checking to see are those packets actually moving between whichever devices they need to be moving between? And you could name anything, web servers, database server, anything. If there was a gold standard between the different switches, hubs, edge routers, oh my gosh, Emily, the edge routers. Everybody forgets the edge routers. They're all set up exactly the same, we promise. Yeah. Okay. Always the exception, right? Yep. So then we bust out our TCPdump and we start looking at our PCAPs and it looks like we're looking at the matrix. Yeah, I'm going to refer to the matrix guys. That's how far I'm going to go back. Always, of course. Everything's perfect. Don't worry about it. But joking aside, if we had a gold image, if every one of those pieces of hardware between the different... And right now we're just talking about some laptops or some phones or whatever, hopefully in an office building. But you could still have... Some of the data centers are still pretty spread out and so there could be situations where in the next room there is a database that needs to be connected to locally and then you're even thinking about the thickness of the walls, y'all. It's ridiculous. So, anyway, yes, going back to having a gold image, that would be delightful. It rarely happens, but it would be nice. Yeah. One of those perfect ideals that like on paper of course, but in reality it doesn't really happen. So, it sounds to me like maybe less than striving for perfection, although we should always try. It comes down to being able to find and solve a problem when one appears and standardization can help them stand out more. Is that about right? Yeah, I would say definitely. It'll make your lives easier. Everything will run smoother, literally. Yeah. All right. So, we're starting to get a handle on how challenging a large scale network can be to set up and manage. With so many parts to keep track of, it's really easy for something to go wrong. Elle shared a few of the most common mistakes people make when setting up networks. One of the really big ones is keeping everything on the same network. So, your guest Wi-Fi, account servers, employee laptops, et cetera, everything is all on one network and this opens you up for a pretty massive security risk because it can allow for an infected device to connect to the Wi-Fi and then make a lateral move. So, it can then jump to your account servers or anything else that's connected to that Wi-Fi or that network because it's all kind of on the same plane essentially. Yeah. Put it in the bucket. Okay. Number one, everyone lumped together on the same network. Why is this such a big deal and what do you do instead? So, it's what Elle said that if everybody is together on the same network and they can see all the other devices on the same network, there's a lot of chances for the security holes that are on the various devices. So, I think about our cell phones, I'm not going to name any vendors in particular, but there have been some recent security errata that have come up saying, "Hey, no, seriously, you do not want to connect to public networks with certain cell phones right now." I'm not trying to scare anybody. I already did a whole series on securities. You can go back and listen to that, but just from a security perspective right there, if you are on the same network as another device and that device can see you, it is a lot more likely for them to be able to exploit some known and unknown to the greater world of security exploits that are already out there. The other thing that I haven't seen brought up recently, but I want to bring it up because I think it's important for folks running the hosts is if you're not separating and putting firewalls and whatnot between your development environments and your production environments, you are going to open yourself up to a world of hurt. And now we're talking about Wi-Fi right now. We're talking about laptops, but I want to talk hardwired also for just a moment that if there isn't some kind of firewall between your Udev environment, your production environment, your straight-up development environment, I'm speaking from a deep pain inside of me remembering something that happened to me long ago in my career where a production website connected to the development dev server or vice versa and someone upgraded and it took down both because they meant to upgrade the development site, it connected to the production website. The production website thought it was connecting to the production... It was a whole mess. So, anyway, check your firewalls folks. That's what I'm saying. Check your firewalls. Yes. Always make sure your environments match for sure. It kind of goes both ways too, because I think what you were touching on was an insecure network can be a threat to you on your device, but also your network is only as secure as your least secure item on it. There was a story a couple of years ago about like a casino that got hacked via, I think it was an aquarium thermometer that was like smart and attached to their network and it was an attack venue that was exploited. That's amazing. So, yeah, be careful. That's amazing and I 100% could see that happening in today's world. There are so many of those edge devices we talked about in previous episodes that never get updated because who would've thought to update? What did you say? An aquarium? An aquarium thermometer for like one of those giant fish tanks with like hundreds of gallons of water in it. Oh, my God. That's great. I actually checked it before we started this episode just to make sure I didn't make it up. And it was, I think in 2018, I don't know if anything came from it, but a whole bunch of headlines were all talking about it as part of the internet of things and the potential downsides of some things being smart. That's amazing. Great story. So, think of that next time you tweet from your fridge, I suppose. That is maybe a good lead into the next thing to look out for, which is keeping track of everything connected to your network. And then the next thing that people can run into is device sprawl. So, going back to the analogy of your home network versus your enterprise, in your home network, you typically know every single device that's on there. And if you don't, it'll be pretty apparent, be like, "I have no idea what this is. Let's kick it off." In an enterprise setting, that is so hard to determine what's supposed to be on and what's not supposed to be on because you have employees bringing their own devices or you have company devices, but you also have all of the smart tech. I'm thinking like smart thermostats or badge scanners or things like that that have to connect to your network in order to work. So, yeah, see, thermostats, device sprawl. Employees bringing in their laptops, connecting their smartphones, all the internet of things, IoT devices that make your offices fancy fun houses or more functional. That sounds like a lot to manage from a networking aspect especially. And I imagine that would increase the risk of a malevolent device sneaking in unnoticed. Oh, yeah. Let's do one more. We've alluded to the Spaghetti Monster and Elle added it to her list too. Another kind of common mistake that you would typically see neglecting those physical components of your network. I think a lot of people think about all of the software components, but not the hardware, especially when you're setting them up. I think what you'll see is these physical components will get set up and you're either ignoring labeling cables or ignoring cooling and power redundancy, things like that where if you are just relying on one person to know, then you're going to set yourself up for trouble if that person leaves or if that person's on vacation and something happens. So, they're like, "Oh, my gosh, the network guru's gone." And you don't know what cable goes to what or anything like that. So, beyond just the devices connecting to the network, you also need to keep track of and properly manage all of the hardware that literally connects everything together physically. We've all seen those nightmare pictures of ethernet cables tangled up in a giant mess and I won't mention the state of my own desk and its cables with no indication of what goes where and you can't exactly try to organize everything by unplugging it to untangle it and replug it somewhere else. That would be disastrous. So, how do you handle that situation? Very carefully. And I wanted to note, I have actually seen the reverse one time long ago, legendarily I went into a server room that was a few stories tall that had the most beautiful waterfall of network cables. So they didn't have to do that. They had very good device management, but guys, it is out there and if you ever get a chance to see that, it's wonderful. So satisfying. It really is for technologists. It's just like, "Oh, satisfying is the word for it." That's true. I think of it like professional athletes like I admire but could never do. Right? Oh, my goodness. I do have a little color coding over here, but not as much as we should. Oh, mine would give you nightmares. I am a gremlin. I am sorry. Oh, well, I have to say also one other thing she made me think of, we've had plenty of, and I'm sure Emily, you've had the same where you've heard the horror stories or had it happen where the person leaves. So, either the DevOps, SysAdmin, the network guru, the whoever left. We had somebody leave long ago at an old job and they were running the server in a closet. It was that story. Yep. I lived that. It was actually that. So that is a real thing that really does happen. Make sure you guys have multiple ways to log in and you understand what those are doing, network devices or otherwise. That actually does happen. And I'm laughing, but I'm assuming it's happened to everybody, maybe not. No, it absolutely has. And anyone out there looking to start a startup or anything, yeah, you can do the closet route, but temperature matters. Keep that door open. So, we've covered a few of the many pitfalls people can fall into when navigating networks. Next up, we're going to cover some of the things you can do to make your life easier. We talked about just how many specialized devices are needed to put a network together and doing it once can be intimidating, but there's a good chance you'll need to do it multiple times. One of the most tedious parts of networking is the unboxing and setting up of new switches because you're essentially setting them all up to match these same configurations. So, you're doing the same thing 50, 100, 200 times depending on how many you're setting up. So, automation, set up essentially a playbook that you'll say, "Okay, this is how I need it configured here." Check all the devices that needs to configure, hit run, and then it sets that up for you so you're not sitting there manually doing each one yourself. Automation. It's my favorite. And it is so good for making use of that kind of golden configuration standard. And what a joy to know that you can just automate at least one part of that setup. Oh, come on. Everybody loves that new switch smell, don't they? I love pulling that thing out of the... Okay. The first time, the second time, definitely not the 50th time. Definitely. Yeah. And yeah, no, once you get real proud of yourself for being able to set up that playbook, let's be honest. And I will say it, Emily, because Ansible is amazing. Okay. I'm so biased, but yes. Not at all. But once you set it up and it works, it's perfect. It's lovely. Nobody touch anything. It's so satisfying. Again, we're going back to the word satisfying, all the kids love these days. But yeah, because it does. It gets very tedious. And there's a human error. You introduce all kinds of human error. So, it's fabulous. Having done many a very tedious task manually, it is not possible to completely avoid user error and automation is very good at avoiding human error. Even when everything is set up properly, it's pretty likely that your admins will need to make changes when problems come up or you need to move offices or upgrade your equipment. It's going to happen. So, in that large network that you have, it's very easy for configuration drift to happen. So, someone can change a password or open up a new port for a quick fix and they forget to close it or whatever it is. But I've talked to a lot of customers that find this super helpful and use this, but either every night or every morning a automated script will run and it will compare the switches current settings to the golden standard or the golden configuration that they have set up and it will flag any discrepancies that they see. And then from there, if they have it set up or implemented, you could also automate that remediation and automatically overwrite essentially what the error is and fix it to make it compliant before anyone logs back on and sees it. Erase the evidence. I mean, not really erase the evidence, but fix the problem automatically. Elle mentioned some of these systems even can use AI to identify and remediate deviations from that gold standard. How would someone set up something like this? Well, now I definitely know we're using more of a predictive, I would say predictive AI agents that are out there. I've seen some. I myself have not coded them yet. I've coded other AI agents. Yes, yes. We have to say that. We're beholden to say that, but we have. But they're more predictive, I would say, and some of the scripts and things that we've written in the past, I'm laughing. I just need to say just for a second before I get to the generative. I'm laughing because of how many times and one in particular I can think of where she said that they just compared the differences and told, maybe made a report like, "Hey, somebody made changes either on the servers themselves or on the switches. Here are the changes that were made." And people do that, by the way, for all kinds of reasons. Yeah. We talked a couple of episodes ago about if you introduce a new application, something's going to change. You're not using the standard gold image anymore. And so there's a lot of reasons it's legit. However, the place that I was at one time would just change it back. So if a machine was rebooted, it had to be virtual machines by this point. If it was rebooted, it would boot with one configuration and then about 10 minutes later would just change to whatever the golden image was. You see, there was a delay and things would work for the 10 minutes. The 10 minutes would go by, the script would change- Yes. ... the configuration. You see where I'm going this route? It would just do it. It wouldn't give a report or anything. Just do it. Yeah. So, anyway, I thought that was really funny and it just took me back again to yet another, "Oh, wow, she's really touching on history, bringing up sore spots, but also things to learn from." So, I could think of 1,000 ways that the predictive AI models that are out there now could help with that. And also some of the more creative things we could probably do with generative AI. We had to have humans going in there. We had to run, once again, we had to be running our TCPdump, my heart and reviewing all those PCAPs. I don't know. Have you ever had to do anything like that, Emily? No, thankfully not. Oh, it's so fun. Come with me sometime. I can't remember what the new interfaces are called, but we see something called Ethereal. It was so... What a name? But we could write, we could create agents that do that generative. I mean, these things are all based off of our brains from like 20 years ago, I swear. So, we could do it and really save ourselves a lot of time and a lot of, wait a minute, wasn't it set differently last week? Wait, there is a gold image. Why is it different now? And then why would someone change it? And because that's what we had to do. Well, and silently in the background just slowly driving you mad. I don't know what all these applications do, but we could have an AI agent that could, right? You could have all that information in there. I'm trying to think of the positive. We do tend to sometimes dwell on the negative in my world, but on the positive side, you could have agents that know a lot more about different types of applications than the humans running the network could fathom and they could understand why those changes might have been made instead of just, "It was changed. Oh, no, bad." Yeah. Or even know if those changes are potentially harmful versus necessary. There's a very fine line. Very, very fine line. And honestly, humans struggle to find it without all the context anyway, but as long as we're dreaming of impossible things, that would be a fun one to have. As with anything that's automated or handled with AI, Elle emphasized the importance of having both contingencies and limits. Those fixes that you put into place also have guardrails. So, you will make sure that whatever automated or remediated response also fits certain parameters. And a lot of times companies when they're implementing this, they'll make sure that the automated or the remediated change gets flagged with a team member. So, they actually go back and check and say, "Okay, yep, this was remediated. Everything looks good." Or if by an off chance something doesn't, it's still flat. So, there's still a human there checking it and making sure that it was done correctly. So, it's not just kind of letting everything go into the Wild Wild West. Yes. Automation and AI both are very, very powerful things and they can go very powerfully in the wrong direction without at least a little bit of oversight. Just like humans. Just like humans. Just like humans. Every once in a while, you got to make sure everybody's on the same page. So, those were the guardrails that we were talking about, right? We have this generative AI, we have predictive AI. We have all kinds of different scripts, automation from even without AI that are comparing between gold images and whatever changes might have been made. A human probably should be in there looking to see and deciding, should that change stick? If so, add it to the gold image. If not, have a conversation with whoever made that change. Or at least allow the exception, because there is such a thing as too strict of guardrails too like in that scenario we were talking about where you make that change and 10 minutes later it is just magically undone with no indication to you. There is a balance to be struck, I think. Those poor sysadmins. Oh, I remember that. They said, "We didn't know. I'm sorry." They brought me in to troubleshoot and work on it. "Are you sure you didn't change anything?" "We didn't change anything." "Are you sure you didn't change anything?" So, also log it all folks, please. Logs are- Yes. That's the other thing is all of that should be logged and audited, the change that you made and then the change it made it back. You should always be able to find why and how something happened and when. Otherwise, you have a really good story to tell about 15 years later. Pretty sure I would be able to convince myself that the computer had come to life and was actively making my life difficult. Well, now we kind of can to a certain... No, not yet. We're not quite there yet. Not yet for real, but it is becoming a very convincing fact Emily. And I think based on everything that we've said so far, there is no escaping entirely the network tangle in the enterprise space, but there are some great ways to make it work for you. We covered a fair bit about networking today, the complexity, the scale, and some of the biggest mistakes you can make when tangling with the Wi-Fi wizards. With our limited time, we only got a quick glance of what's out there, but there's a lot in your favor if you know where to look. Make sure to hit us up on social media at Red Hat and use the hashtag, #CompilerPodcast and that does it for this episode of Compiler. This episode was written by Johann Philippine. Thank you to our guest, Elle Universal. Compiler is produced by the team at Red Hat with technical support from Molly Brock. If you like today's episode, follow and review our show on your platform of choice. Until next time.

More about 自動化と管理

自動化はとくに製造業や自動車などの業界で、反復作業を削減するか、またはこれを不要にします。

ブログ投稿

Red Hat Ansible Automation Platform：2025 年を振り返る

2026 年 1 月 21 日
自動化と管理
ブログ投稿

ポリシーをコードとして捉える：なぜ今必要？どのように役立つのか？

2024 年 4 月 18 日
自動化と管理

About the show

Compiler

Do you want to stay on top of tech, but find you’re short on time? Compiler presents perspectives, topics, and insights from the industry—free from jargon and judgment. We want to discover where technology is headed beyond the headlines, and create a place for new IT professionals to learn, grow, and thrive. If you are enjoying the show, let us know, and use #CompilerPodcast to share our episodes.