Low letancy/High performance RHEL

Recently I worked with a customer who requires high performance servers and near real time response for their business application. They were very concerned about the power saving options in the CPU and other server components and were advised by their application vendor to disable these feature.

The customer is running Red Hat Enterprise Linux 6 on a 28 x (128 GB RAM, 2 x 8-core Intel Xeon CPU, SSD harddisks) – quite huge specs!

Since the application vendor have milliseconds-based SLA with the customer, they urged the customer to use the following values to disable power saving at CPU level:

  • Background

CPU’s use a transition multiple levels of power-saving states (called C-states), for example it changed from full functioning state (called C0) to a lesser state where some CPU components are shut down, if it remains idle it will transit to deeper C-State and shut down more components. Until CPU is assigned a new task/application it starts to wake up into higher C-States it fully functioning again.

Unfortunately, transitioning from deeper power saving states back to a fully powered up state takes time and can introduce undesired application delays when powering on components.

  • How to really do it in RHEL?

The correct way to do this is to pass the processor.max_cstate=1 and idle=poll parameters to the kernel on server startup, this will disable CPU power saving, lower latency and increase performance. The impact to this is much more power consumed, much heat generated, and the server has to be monitored to make sure it’s OK, and has to be cooled down all the time, hence even more power consumption.

However, Red Hat provides the interface /dev/cpu_dma_latency which control processor C-State and can prevent CPU from transitioning to deeper power saving modes. Simply any process can open the file /dev/cpu_dma_latency and write 0 to it to accomplish the same effect (preventing CPU from entering deeper C-State), and this effect remains as long as the process keeps the file open.

So /dev/cpu_dma_latency can control the above settings in much better way, this interface is controlled with the package tuned, and controlled with the command tuned-adm. The idea about it is it defines power saving profiles that you can change based on your needs, so for example to disable CPU power saving to decrease latency and increase performance we would use:

# tuned-adm profile latency-performance

Which will do the exact effect required by application vendor, as well as preventing other power saving parameters in the system like HD.

Returning back to normal powersaving state can be achieved with:

# tuned-adm profile default

or even

# Shutdown tuned
# tuned-adm off

So, to still save power and help the server be in reasonable temperature and not to emit too much heat, enabling latency-performance profile before business hours and return it back to normal server behavior after business hours (via crontab) would be much better solution in my opinion since it will give the desired application performance and yet conserve power in non business time and avoid high power consumption.

Tuned can be installed from Red Hat repository or from RHEL CD.