Red Hat Satellite 6.9 with Puma Web Server

table { border: #ddd solid 1px; } td, th { padding: 8px; border: #ddd solid 1px; } td p { font-size: 15px !important; vertical-align: top; }

Until Red Hat Satellite 6.8, the Passenger web/app server was a core component of Red Hat Satellite. Satellite used Passenger to run Ruby applications such as Foreman. Satellite 6.9 is no longer using the Passenger web server. The Foreman application (main UI and API server) was ported to use the Puma project. In this post, we’ll provide an overview of Puma and see how the application server switch affected performance of the Satellite server.

Puma tunables

Puma is a Ruby application server which is used for serving Foreman related requests to the clients. For any Satellite configuration that is supposed to handle a large number of clients or frequent operations, it is important for Puma to be tuned appropriately. In order to tune Puma, always consider following parameters: 

• foreman-service-puma-threads-min 

• foreman-service-puma-threads-max

Puma uses a thread pool. You can set the minimum and maximum number of threads (per worker) that are available in the pool which we call `foreman-service-puma-threads-min` and `foreman-service-puma-threads-max`. For example, you can tune threads as: 

satellite-installer --foreman-foreman-service-puma-threads-min=0 
--foreman-foreman-service-puma-threads-max=16 

Note: Puma will automatically scale the number of threads, from the minimum until it caps out at the maximum, based on how much traffic is present.

Puma offers one mode related to workers called "clustered mode." Clustered mode forks workers from a master process. Each child process still has its own thread pool. You can tune the number of workers using the `foreman-service-puma-workers parameter`. For example:

satellite-installer --foreman-foreman-service-puma-workers=2

Note: With workers count set to 2 (which is by default) and maximum number of threads set to 16, Puma will be able to spawn up to 32 threads in total.

As of now, Puma is not part of tuning profiles, but it is being tracked in bug 1959691. Moreover, when changing the tunings, users should be aware of the number of connections PostgreSQL is able to handle. 

How we measured

To test how the change from Passenger to Puma affected performance, one of the experiments we did was comparing concurrent registration speeds on Satellite 6.8 and 6.9. We have measured the speed of concurrent content host registrations up to 560 in increments of 70 registrations. (So, 70, 140, 210, etc.) Besides the duration of the registration process, we kept an eye on registration failure rate and Satellite server system metrics like consumed memory, system load and many others. We also retried measurements with Satellite servers with different numbers of CPUs to be able to estimate best parameters.

Performance Findings

• In our testing, Satellite 6.8 (with Passenger) in its default config was able to handle about 270 concurrent registrations. However,  Satellite 6.9 (with Puma) in its default config was able to handle about 480 concurrent registrations.

• Moreover, registrations with Passenger in our testing are 28% faster at lower concurrency levels, but Puma is much faster and reliable at higher concurrency levels.

• Fewer threads will lead to more memory usage for different scales on the Satellite server. For example, when we tune the Puma server with maximum 16 and  minimum 16 threads, Puma will consume about 12% less memory as compared to the same setup but with minimum threads set to 0.

• More workers will allow for lower time to register hosts in parallel. If you have enough CPUs, adding more workers adds more performance.
  For example, we have compared Satellite setup with 16 CPUs and different numbers of workers. Performance improvement for setup with two and four workers was 34%, four to eight improved by another 17%, but eight to 16 workers only improved by another 4%.

• Some more details are provided in the performance tuning guide of Red Hat Satellite 6.9. 

Takeaways

In this post, we discussed the newly introduced Puma application server for Red Hat Satellite 6.9, as well as how this impacted the performance of the Satellite Server. Three parameters are used to tune Puma:

--foreman-foreman-service-puma-workers 

--foreman-foreman-service-puma-threads-min 

--foreman-foreman-service-puma-threads-max 

Each of these parameters can have a significant impact on performance. If you're interested in detailed information on Satellite, then the Red Hat Satellite documentation is a must read.