MultiChoice modernizes video streaming with Red Hat

Hitesh Govind, Video-on-Demand Engineering Manager, MultiChoice Group: MultiChoice Group is South Africa’s leading entertainment platform. We provide live TV, streaming, and subscription services to millions of households. There are a number of businesses under the MultiChoice Group banner. My team is responsible for maintaining the infrastructure behind our video-on-demand offering.

Govind: Our video-on-demand platform needs to be highly available, and that means accommodating peaks in demand and millions of concurrent viewers without a hitch. We can’t afford any downtime— if the platform had gone down during the Olympics, for example, we would have had a lot of unhappy customers.

The platform was built on hyperconverged infrastructure hosting virtual machines running Linux CentOS 7, which was reaching end of support. We also had two big issues with our datacenters. The staging and production environments were out of alignment, so we couldn’t predict how code would behave when it went into production. The two sites were also running different infrastructures, so there was no parity between them.

Rather than replacing our infrastructure with a like-for-like platform, we took the opportunity to modernize and standardize our environment. If we wanted to become more efficient and cloud-native in the future, we needed to replace manual processes with automation, adopt microservices, and introduce a container orchestration platform.

Govind: Red Hat OpenShift Virtualization met our technical requirements and complemented the other technologies in our stack. We’re 99% Linux-based and have been working with Red Hat technologies for a while. My team are big fans of Red Hat OpenShift, so it was the natural place to start our search. We evaluated OpenShift Virtualization against traditional virtualization solutions from other vendors. Red Hat supports virtualization, automation, and allows us to containerize applications from one platform, so it was the best fit.

Govind: We set an ambitious timeline and couldn’t have achieved it without support from Red Hat and LSD Open. You need partners who will go on the journey with you. We ran a request for proposal (RFP) and LSD Open gave the best responses to our questions. LSD Open was able to differentiate its offering by sharing technical expertise and experience. They had the vision and drive to guide us to something new, plus they’re local to us here in South Africa.

Julian Gericke, CTO, LSD Open: If you always use the same approach, you always get the same results, and we wanted to give MultiChoice a modern, one-stop shop to support its video-on-demand offering.

Initially, we recommended a platform that was similar enough to the original tech stack to simplify change management, but Red Hat OpenShift Virtualization offered so much more. There was an opportunity to achieve environmental parity and to get started on the modernization roadmap. Our solution allows MultiChoice to go hybrid or move to a hyperscaler in the future.

Gericke: We worked with a Red Hat Solution Architect to design a scalable, fit-for-purpose solution. The first step was to replace CentOS 7 with Red Hat Enterprise Linux®. We then deployed OpenShift on bare metal and made that operational.

Using the Red Hat Migration Toolkit for Virtualization, porting virtual machines from VMware into OpenShift was seamless. We took a full image of physical machines and mounted them on OpenShift Virtualization.

To create alignment, we moved the staging and production environments to the same cluster. Images can be cloned in the staging environment, modified, and moved into production, and Multichoice is already seeing benefits in terms of how quickly it can deploy things. The specs between the main datacenter and disaster recovery are identical.

Now, MultiChoice has a mixed workload across the modernized and heritage layer, with workflows deployed on OpenShift and containers running concurrently. We also established CI/CD pipelines to shift towards DevOps and provided training for the team.

Govind: The implementation took 8 months, and we had daily check-ins with LSD Open.

There were a few small challenges during the project. We used the platform’s image builder to convert Docker images to a Kubernetes compatible version. Most translated well, but we needed to fine-tune the Java heap sizes to get all containers functional. Some still didn’t work as expected because they were running unsupported components. LSD Open helped to debug those to get things moving again.

We also had some issues with firewalls. The Red Hat OpenShift implementation was greenfield, so we had to recreate subnet rules, and some documentation we needed to do that was outdated or missing. That was a big learning curve – when we wrote those documents 8 years ago, no one thought we’d been moving to a different subnet!

The first week we went live we also had a couple of issues. Some containers that had been migrated were out of scope for the project and there were insufficient soak tests done to confirm if the memory was adequately optimized. Once we realized they were crash looping, it was easy to fix. We also had to fix some SSL certificates, but since then, the platform has been stable.

Govind: LSD Open wrote some playbooks for us. We’re using it to automate patch management for Tenable. It scans for vulnerabilities and triggers a workflow to patch any issues it detects, which simplifies maintenance. We also have playbooks to check mounts, devices, capacity, and storage connectivity. We’re looking for more things to automate. We want playbooks to be reusable across the organization. They’re building blocks we can use to create more complex automations in the future.

Govind: It was a big change and we’re still learning every day. We’ve invested in training to upskill developers and engineers, and they’re working on Red Hat certifications. Kubernetes was completely new to us, so we leaned on Red Hat and LSD Open to help us adapt. We use Red Hat Learning Subscriptions rather than face-to-face training, as it’s more convenient for our team. We can work on getting certifications in our own time.

Govind: People are naturally resistant to change, and there was a lot of pushback. Modernization goes against the grain of the way things have always been done at MultiChoice. Our stakeholders wanted to stick with what they knew—using Docker on a VM—but I was really invested in migrating to OpenShift Virtualization. I attended a lot of OpenShift events to gain knowledge and iron out any stakeholder concerns. The Red Hat team was also really responsive to any queries they had, and together we won them over. I’m confident that OpenShift Virtualization is the right choice for us now and in the future. We’ve seen great results really quickly, which impressed people.

Govind: We’re still in the middle of our modernization journey and have only been live on OpenShift Virtualization for a few weeks, so we’ve got more work ahead of us, more OpenShift accreditations to earn, and want to automate more workflows with Ansible Automation Platform.

In the long term, we’re considering a wider OpenShift adoption across the company and will keep on developing our CI/CD pipelines and microservices deployments with support from LSD Open.

Gericke: MultiChoice is the first South African Media and Entertainment company to adopt OpenShift Virtualization; it’s a pioneer and we’re proud of what we’ve achieved together. Modernization can be difficult and complex. There’s the technological side and the cultural side to consider, and we navigate that by developing strong relationships, providing training, and enablement. We triaged issues together and MultiChoice picked up new skills and adapted quickly. I’ve never seen such a fast Kubernetes implementation in 10 years of working on these projects.