Lessons learned from a large-scale telco OpenStack and SDN deployment.

The obvious cannot be overstated: implementing OpenStack and software defined networking (SDN) at a large telco is a complex undertaking. That’s why it’s just plain good advice to listen in on the experiences of those who’ve already forged such an implementation, and here’s your chance. At the recent OpenStack Summit in Barcelona, Red Hat shared its lessons learned from a Red Hat OpenStack deployment with an SDN provider in the telecommunications market in the Europe, Middle East, and Africa (EMEA) region.

Red Hat’s Guil Barros, an OpenStack product manager focused on partner enablement, Cyril Lopez, senior technical OpenStack consultant, and Vicken Krissian, a senior technical product manager, outlined Red Hat’s work with an SDN partner on the project. The initial request for proposal (RFP) was published in mid-2015, and Red Hat had a proof-of-concept (POC) ready for the client by the end of the year. The deal was signed in February 2016, an integration hackfest was held to explore and develop potential solutions, and by July there was a fully working platform. User acceptance tests were started in the fall, and a pre-production platform was completed. The implementation is set to go live soon – a target the customer established at the onset.

The fourth-quarter 2016 deadline has been a challenge, especially given that the Red Hat OpenStack Platform 10, with SDN integrated, is only now launching. Even more challenging – the SDN partner’s product won’t get certified to the Red Hat platform until first-quarter 2017.

These types of challenges illustrate why it is critical to keep communications open. As Barros told attendees at the summit in Barcelona, “It is not that we don’t want to line up the roadmaps we’ve each got – call them products – in mind. The customer has a product they are delivering to their customers and we have an OpenStack product that is going to have to meet those needs, and the SDN vendor may have an SDN product that will integrate with the OpenStack product to create the deliverable.”

The project also encountered integration challenges. Early on, the teams had to work with the first version of Red Hat OpenStack Platform director, a toolset for installing and managing a complete OpenStack environment, and there was no SDN integration. The teams also had to break integration down into two parts. Initially, they used Red Hat OpenStack Platform version 8 and 9, because the new version 10 was not available.

Cultural issues also bubbled up. The customer wanted to be autonomous from the beginning and wanted to learn from doing rather than rely on too much support from Red Hat. There was only one dedicated engineer assigned onsite. Krissian told attendees Red Hat usually kicks projects off with workshop to bring all the stakeholders together; in this instance that was not done. Without clear communication and synchronization among different entities, there can be slower pace for learning and even deployment.

“This is one of the big lessons we’ve learned. We need much deeper integration between us and the partner, with a lot more people either onsite or in some partner engineer exchange,” Barros said.

There were also implementation challenges. Telcos can be sensitive to change, and security requirements are tantamount, so these factors have to be addressed upfront. Lifecycle management must be part of the implementation, especially when using open source. OpenStack goes through a new release cycle every six months, Lopez stressed.

The Red Hat OpenStack deployment has given us a real opportunity to roll up our sleeves, tackle the challenges, get to work, and learn along the way. Here’s our lessons learned:

  • Institute continuous integration throughout the project. This allows for automated testing of every pre-release system, and make sure customers have access to them.
  • Communication is key. Understand the functionalities the customer wants, and where, identify key stakeholders and establish formal points of contact; and be consistent.
  • Hold a discovery design workshop at the beginning of the project. Use this to share a detailed architecture review, develop requirements and features targets, and create roadmaps and assign ownership of them.
  • Test, test, test! Hackfests are your friend, and don’t just limit them to the project’s onset. Use them whenever a major new version is integrated into the implementation.
  • Align all four work streams — upstream development, Red Hat, partners, and the customer – to lock in features, architecture and timelines (and avoid scope creep).

Red Hat is uniquely poised to address challenges encountered in any OpenStack implementation. There are large numbers of Red Hat engineers and developers directly involved in OpenStack development and we are one of the leading contributors to the upstream community. We also have a lot of experience in large-scale deployments, and we don’t just offer individual products, we have an entire ecosystem of partners on which to draw expertise and complementary products and services.




Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.