Performance and Scaling your Red Hat Enterprise Linux OpenStack Platform Cloud

As OpenStack continues to grow into a mainstream Infrastructure-as-a-service (IaaS) platform, the industry seeks to learn more about its performance and scalability for use in production environments. As recently captured in this blog, common questions that typically arise are: “Is my hardware vendor working with my software vendor?”, “How much hardware would I actually need?”, and “What are the best practices for scaling out my OpenStack environment?”  

These common questions are often difficult to answer because they rely on environment specifics. With every environment being different, often composed of products from multiple vendors, how does one go about finding answers to these generic questions?

To aid in this process, Red Hat Engineering has developed a reference architecture capturing  Guidelines and Considerations for Performance and Scaling of Red Hat Enterprise Linux OpenStack Platform 6-based cloud. The reference architecture utilizes common benchmarks to generate a load on a RHEL OpenStack Platform environment to answer these exact questions.

Where do I start?

With the vast amount of features that OpenStack provides, it also brings a lot of complexities to the table. The first place to start is not by trying to find performance & scaling results on an already running OpenStack environment, but to step back and take a look at the underlying hardware that is in-place to potentially run this OpenStack environment. This allows one to answer the questions “How much hardware do I need?” and “Is my hardware working as intended?” all while avoiding the complexities that can affect performance such as file systems, software configurations, and changes in the OS. A tool to answer these questions is the Automatic Health Check (AHC). AHC is a framework developed by eNovance to capture, measure and report a system’s overall performance by stress testing its CPU, memory, storage, and network. AHC’s main objective is to provide an estimation of a server’s capabilities and ensure its basic subsystems are running as intended. AHC uses tools such as sysbench, fio, and netperf and provides a series of benchmark tests that are fully automated to provide consistent results across multiple test runs. The test results are then captured and stored at a specified central location. AHC is useful when doing an initial evaluation of a potential OpenStack environments as well as post-deployment.  If a specific server causes problems, the same AHC non-destructive benchmark tests can be run on that server and the outcome could be compared with the initial results captured prior to deploying OpenStack. AHC is publically available open source project on GitHub via https://github.com/enovance/edeploy.

My hardware is optimal and ready, what’s next?

Deploy OpenStack! Once it is determined that the underlying hardware meets the specified requirements to drive an OpenStack environment, the next step is to go off and deploy OpenStack. While the installation of OpenStack itself can be complex,one of the keys to providing performance and scalability of the entire environment is to isolate network traffic to a specific NIC  for maximum bandwidth. The more NICs available within a system, the better. If you have questions on how to deploy RHEL OpenStack Platform 6, please take a look at Deploying Highly Available Red Hat Enterprise Linux OpenStack Platform 6 with Ceph Storage reference architecture.

Hardware optimal? Check. OpenStack installed? Check.

With hardware running optimally and OpenStack deployed, the focus turns towards validating the OpenStack environment using the open source tool Tempest.

Tempest is the tool of choice for this task as it contains a list of design principles for validating the OpenStack cloud by explicitly testing a number of scenarios to determine whether the OpenStack cloud is running as intended. The specifics on setting up Tempest can be found in this reference architecture.

Upon validating the OpenStack environment, the focus shifts to answering scalability and performance questions.  The two benchmarking tools used to do that are Rally and

Cloudbench (cbtool). Rally offers an assortment of actions to stress any OpenStack installation and the aforementioned  reference architecture has the details on how to use the benchmarking tools to test specific scenarios.

Cloudbench, cbtool, is a framework that automates IaaS cloud benchmarking by running a series of controlled experiments. An experiment is executed by the virtue of deploying and running a set of Virtual Applications (VApps). Within  our reference architecture, the workload VApp consists of two critical roles used for benchmarking, the orchestrator role and workload role.

Rally and CloudBench complement each other by providing the ability to benchmark different aspects of the OpenStack cloud thus offering different views on what to expect once the OpenStack cloud goes into production.

Conclusion

To recap, when trying to determine the performance and scalability of a Red Hat Enterprise Linux OpenStack Platform installation make sure to follow these simple steps:

1. Validate the underlying hardware performance using AHC
2. Deploy Red Hat Enterprise Linux OpenStack Platform
3. Validate the newly deployed infrastructure using Tempest
4. Run Rally with specific scenarios that stress the control plane of OpenStack environment
5. Run CloudBench (cbtool) experiments that stress applications running in virtual machines within OpenStack environment

In our next blog, we will take a look at specific Rally scenario and discuss how tweaking the OpenStack environment based upon Rally results  could allow us to achieve better performance. Stay tuned and check out our blog site often!