“That’s what open source is all about”: A short history of collaboration, innovation and education in research

This article appeared in the May 2023 issue of Red Hat Research Quarterly.

In 2017, Red Hat Chairman Paul Cormier and Boston University (BU) professor Orran Krieger helped spearhead a collaborative partnership between the two institutions that would come to include expanding Red Hat’s participation in the MOC Alliance, the establishment of the Red Hat Collaboratory at BU for research incubation, and the creation of a Red Hat OpenShift Data Science environment at BU for open source education resources. The Red Hat Research team has been managing Red Hat’s relationship with BU since the team’s inception in 2018.

To celebrate the fifth year of Red Hat Research and the research quarterly, RHRQ editor Shaun Strohmer sat down with Paul and Orran to look back at how and why this partnership formed and its impact on education and the tech industry. They had plenty to say about changes in computing environments, what computer science education needs today, and why open source is the cornerstone of innovation.

Shaun Strohmer: What motivated each party to create the Red Hat-Boston University partnership? Let’s start with Paul. What was the appeal of starting a collaborative relationship with the computer science folks at BU?

Paul Cormier: We first thought it was a way to grow a community around OpenStack through collaboration and connect with people early in their careers. From there, it has significantly expanded. The Mass Open Cloud (MOC) played a big part in it, too. It was very enticing to have a place to run all of our products together as a portfolio in a real-world, large-scale environment so we could understand the system aspects of it more deeply.

One thing we’ve always tried to do at Red Hat—and I’ve been here 23 years—is look at our product set as a portfolio. In the technology world today, we don’t think enough about how things work together as a system. Joining BU in the MOC was a big opportunity to see how our combined products interact in a systems world.

Shaun Strohmer: From the BU side, Orran, what did you hope a Red Hat partnership could bring to the university or the MOC?

Orran Krieger: There were many reasons why building the cloud for academic users made sense, but I really wanted to base it on open source software so we could see everything going on. Red Hat brought a lot of individual technologies for the on-premises cloud, including Red Hat Enterprise Linux, OpenStack Ceph, and so on. In the public cloud realm, there’s increasingly this oligopoly of three or four players. Everything is closed, so you can’t see what’s going on. But Red Hat specifically is about openness. Openness eventually wins, and by having Red Hat as the anchor partner, when we bring in other partners who might be more focused on proprietary things, we can still impose openness on everything. 

The Red Hat Collaboratory has moved to the brand new BU Center for Computing and Data Sciences.

Shaun Strohmer: Did the partnership initially start with Red Hat providing certain resources?

Paul Cormier: I wouldn’t put it that way. Resources weren’t the first thought in our minds, and I don’t think they were in Orran’s mind either. For us, the beginning was about getting broader: a broader look, broader participation, a broader take on these technologies we knew were the underpinnings of what was emerging as cloud technology.

Shaun Strohmer: Then the relationship also broadened to include specific research projects.  How did you hope Red Hat’s engagement in individual projects would advance them?

Orran Krieger: I’ve had a foot in the academic research community and industry most of my career. I like doing research involving innovative things that lead to publications. But I also want the things I do to have a real impact, to be integrated into things that make a difference to people and society. As systems are getting more complicated, no individual faculty member has the expertise on the details of different projects that are out there. You can’t build the system from scratch. I also wanted to create a model where engineers engaged with students so the students could have a tangible impact on open source software that could host real users. It’s a very innovative thing—I don’t think there’s anything like this anywhere else.

When I learned how to program, a lot of it was essentially apprenticeship, where people who were better than me were beating me up. I can’t do that with a huge number of students, but I can involve engineers in projects and get real criticism and learn modern practices. That engagement has added a degree of expertise and professionalism that’s allowed students and projects to have an impact they couldn’t have otherwise. 

Shaun Strohmer: Paul, what were your initial thoughts on getting Red Hat engineers engaged in university research projects?

Paul Cormier: We didn’t want the university doing work to augment our products directly. Instead, we were interested in research that would feed into our products three, four, five years or more down the road. Open source is such a big part of everything in the software world today. If you look at what Red Hat did to be successful, we started with pure open source, which is really a kind of research. And as Orran said, it’s a place to get an apprenticeship. You put your ideas and code out there and get beat up on it. That’s what open source is all about. What Red Hat did was take that model and develop the ability to build commercially consumed enterprise software. We had a head start in the research world just because we were based in open source. 

Orran Krieger: One of the things we talked about then—and I’m super excited because it’s actually happened—is accelerating the impact or the transition to product. So much in the open source community still goes through a release cycle. What we started talking about, even in that first meeting, was whether we could take things from research and expose them at a very early stage, in a CD (continuous delivery) kind of way.

There are three ways of doing radical innovation. There’s the research community, which has been doing radical, out-of-the-box thinking for a long time. There’s the open source community, which radically changed all the software we’re using today. And then there’s that cloud world, where they often innovate even before releasing, testing a change at scale and allowing for rapid evolution. The open source community hasn’t had that, and collaboration seemed like a way to get all three things—the cutting-edge thinking of research, the impact of open source, and accelerated innovation—tied together.

Paul Cormier: I agree. The open source community is big, but it’s also very individualistic. They don’t have that big testbed. But I also remember we talked about the skill sets needed to operate a cloud at scale: we realized early on that it would be a hybrid, multicloud world. One cloud by itself was never going to be sufficient. So what are the skill sets? The only people that knew how to operate clouds at this scale were the Big Three cloud guys. 

Orran Krieger: That part is sometimes forgotten. We have this whole high-performance computing (HPC) community, and we have the Massachusetts Green High-Performance Computing Center (MGHPCC). They’re now transitioning to operations on the MOC for much of what they do. We have 500,000 cores of compute, 500 petabytes of storage, and tens of thousands of users just on the HPC clusters, and increasingly that demand is shifting to cloud. We have people here doing operations that are used to scale, so we already have a community of operators that are in a position to say, “This is what you’re going to need to support production operations.” 

So we looked at research from that perspective. What skills, tools, and processes do people need to operate clouds at scale? In a hybrid cloud world, that’s what many companies have to figure out—and that was a really attractive thing too. That led to building almost a whole new profession of site reliability engineers (SREs). That doesn’t sound new today, but it was back then. 

Shaun Strohmer: I want to discuss a factor you mentioned earlier: the ability to bring in other industry partners. What are the benefits of that? And what are the challenges?

Paul Cormier: It is a huge benefit. Sometimes people get confused about what Red Hat is. One thing I always say is that Red Hat is not an open source company: Red Hat is an enterprise software company with an open source development model. It’s a different thing, right? We do all of our development in open source because we think that open source is a better way to get to a better outcome.

Some of our commercial competitors look at us as a software company and not necessarily as an open source company. Some may not know what open source means. But working in collaboration with universities has been a powerful demonstration of how we do open source. Now this open source-based technology around the cloud is driving where the whole industry’s going. 

We’re not doing this just to make Red Hat products better, though certainly that is an outcome we’d love to see. The first order of business was to solve problems in a completely open world with anyone, especially from the academic community, that wanted to participate. That really helped other commercial software companies and even some of our competitors better understand what Red Hat is all about. Before, they just saw us as a competitor, winning or losing deals with each other. This was a great way for us to illustrate what open collaborative development truly means. When you’ve got university partners, by their nature, what they do is open, collaborative development.

Orran Krieger: I can’t expand on what Paul said except to say that I violently agree. Also, no one company has the power to do everything. As for the challenges, I get nervous about some of the benefits being lost if we aren’t mindful of them as we grow. One of our tasks initially was finding a model for aligning the incentives of engineers and researchers, so they want to work together. If people’s incentives are purely publications or purely how many lines of code they commit to open source or get into a product, it doesn’t work. It’s a lot to [Red Hat Research director] Hugh Brock’s credit that we’ve been so successful in bringing them together. 

Shaun Strohmer: Paul, how has this partnership benefitted Red Hat over time, whether in terms of talent or innovation or something else?

Paul Cormier: Red Hat is a software vendor by heart— we like to code things. But we realized we would need people with specific skills as we got into the work. Now we have our own managed services, where we manage our products and our partner’s clouds for our customers. It’s one of our fastest-growing segments. And it grew from the knowledge and tools we got from working with BU. One of the reasons why managed services are one of our fastest-growing segments is that the industry doesn’t have enough talent for customers to do that themselves. This collaboration was the beginning of identifying and meeting that need from both a skills perspective and a tools perspective.

Orran Krieger: That’s really exciting. We’ve experienced the flip side of that. We brought loads of students down to MGHPCC when we racked 500 servers that had arrived, which was so unfamiliar to them. The computer is sort of this magic box for a lot of students. This partnership engages students in something real, which they then get to use.

Paul Cormier: I totally agree with you. People who come up through the Linux world often play with this independently. But when we recruit from universities, we see some people, even in computer science, who don’t understand how a computer works underneath. It’s fantastic that you get that benefit on the student side.

BU PhD Ali Raza presents his thesis work on Unikernel Linux. His related paper on the subject will appear at top systems conference EuroSys 2023 in Italy in July.

Shaun Strohmer: Let’s move from getting students to understand systems to the value of systems research more broadly. 

Paul Cormier: One of the questions I often ask is, “What problem are we trying to solve?” It’s so valuable to have that big-picture understanding of the problem and have the skills to drill down into the piece you’re trying to solve at the moment.

I see many people who only understand the big picture. And I see another set of people trying to solve their little piece of the problem without understanding where it fits in the bigger problem. This is what systems are: if you can traverse that system up and down, you’re a much better problem solver. And systems are just getting bigger every day. 

Orran Krieger: The assumptions are changing. Before, Moore’s Law essentially said, “Don’t worry about adding more stuff, because things are going to get faster anyhow,” so we—as an industry—added layers and layers of crud. And the networks have gotten hugely faster, solid-state drives have changed, and persistent RAM accelerators have changed things. As we see the end of Moore’s Law and Dennard scaling, systems are more critical, and they’re critical at scale, because problems occur at scale that don’t occur when it’s not at scale.

Shaun Strohmer: Let’s talk about computing environments. We’ve talked a lot about cloud, but what about on-premises computing? Will that continue to be relevant?

Paul Cormier: It’s as relevant as cloud computing. It’s a hybrid world. On-premises is still relevant because there’s a whole set of applications that, for a hundred reasons, aren’t going to move to the public cloud, whether it’s security, data, privacy, location, or what have you. 

The problem is I don’t think much of the IT world understood that until they got into moving applications to the cloud. Big businesses, back when Amazon Web Services (AWS) was just taking off in the mid-2000s, were planning to have 90% of their apps in Amazon within five years. Now we’re helping them build private clouds and hybrid environments, and they’re only about 25% in the cloud. They’re probably going to grow that, but the point is they didn’t understand why they had to be a hybrid shop, or what it meant, until they started getting into the work of cataloging their applications and understanding what they do, how they interact with a thousand other applications, the cost of where they’re going to run it and where they’re not going to run it, and what it was going to take from a software development perspective to get out to the cloud. 

Once they understood that, they became hybrid believers. Over the last three or four years, we have run into that in spades every day. It’s a very different world than six or seven years ago when people were thinking, “We’re just going to the cloud because it’s easy.”

Shaun Strohmer: Orran, what about from a university or research perspective?

Orran Krieger: A decade from now, I don’t believe this question will even be relevant. The question is, how do we make a system so it’s elastic and we can shift computing between them? What we’ve been doing here at BU is moving stuff between HPC clusters and OpenStack and OpenShift with different compliance regimes. Ultimately, it’s about clusters of computers–datacenters–shifting computing around and building services that can span all that.

BU professors Vaisa Kalavri and Jonathan Appavoo present their Red Hat Collaboratory project “Towards high performance and energy efficiency in open source stream processing” at the MOC Alliance Workshop.

Paul Cormier: I couldn’t agree more. From a system perspective, we just expanded that big picture to include whether it will run in multiple clouds, on-premise, and so on. That’s now the new big design point to design down from. Look at microservices. That’s about breaking up a monolithic application into a set of services that can potentially run everywhere. That’s just good software design. 

Orran Krieger: University research is a good example. You have multiple universities, multiple clusters deploying all kinds of things. If you’ve got data sets for gene sequencing or data sets from NASA, they will be used by thousands of universities that want to access the data set in many different places. We want those places to collaborate in terms of hosting the data efficiently, not having it redundantly stored across many different things. It’s actually systems of systems.

Shaun Strohmer: Before we finish, I’d like to hear from you about the research work you think will be important in the next few years and how the Red Hat-BU partnership has unique opportunities to engage in it.

Paul Cormier: It might not sound new, but the multi- or hybrid-cloud world is still new in every aspect. Ninety-nine percent of the software I see does not run in a hybrid cloud world. We have security issues across that, and data, privacy, and management issues. For a lot of software today, the clouds and the on-premises are just a bunch of islands. We have a ton of work to do before we’re truly running that as one system. Before we can expand, for example, to use AI and machine learning and automation to do hybrid better, we need to understand this new world better.

Don’t get me wrong: I’m not saying we don’t understand hybrid. I’m saying that there is still so much to learn there. We will have to ask exactly the questions we discussed earlier. Where does it fit in the big picture? And what are the things I need to do with the lower picture? How does what I’m doing affect the layer above me or three layers above me? 

Orran Krieger: So systems matter—they really matter. On one side, you’re taking things to a massive scale; on the other, you’re hyper-optimizing things for what they’re actually doing. Take Kubernetes. It says, we’re going to make the kernel standardized, and it doesn’t matter what the application runs on.

We’re figuring out how to take systems not just as a pool of things but actually optimize them for what they’re doing, all across the stack. There’s a very cool project at the Collaboratory now [see BU PhD Han Dong’s article, “Tuning Linux kernel policies for energy efficiency with machine learning,” in this issue] using AI to analyze and optimize a specific workload, then cut its energy use by one half. I think the role of the operating system and the hardware—especially as we’ve gotten away from a world where we’re getting performance improvements just by the clock cycle doubling every few years—will be increasingly important. 

Paul Cormier: That goes back to what we were talking about earlier. We do those optimizations in all layer levels, but now how do they fit back into the big picture? And what are the tools we now need to develop to manage and run it?

Orran Krieger: We’ve got our work cut out for us—we’ll have job security in the system space for a while!

Paul Cormier: It’s a great time to be a systems engineer.

Orran Krieger:  When I entered the field and built my first system, it was this dismal world where operating systems were proprietary, OS innovation could only happen within companies, and performance didn’t matter anymore. And today, open source and systems are at the center. 

Shaun Strohmer: That’s a perfect summary. Thanks so much to both of you for taking the time to talk with us today and share these insights on where we’ve been and where we need to go.