High %LAT_C values in ESXTOP

ESXTOP is a great tool for troubleshooting performance issue in VMware vSphere. In the past I written a post about CPU troubleshooting. One of the main values I mention in this blog post was %RDY.Want to know what it means? Read the post 🙂

While checking a vSphere 5.1 environment of a customer I got the following ESXTOP results

As you can see the %USED and %RUN differer a lot. Meaning the virtual machine want more CPU resources (%RUN) than it is actually getting (%USED). A small difference is normal but not up till 50%. %RDY was normal but as you can see %LAT_C is very high. But what does the value %LAT_C means? In the man pages %lat_c is explained as:

%LAT_C Percentage of time the resource pool or world was ready to run but was not scheduled to run because of CPU resource contention.

As you can read, there is a high CPU resource contention. But where does it come from? This is a DELL PowerEdge R620 with 2 Intel E2650 8 cores processors who is running 4 virtual machines. All these virtual machine have 8 vCPUs configured. So you may think that the amount of vCPUs is to high for the amount of pCPUs. But I wouldn’t expect these values. Then I looked a the P state off the CPU, seeing P states of 4, 5 even 11 or 12 explained to me that power saving was enabled in the BIOS for the CPUs. After I disabled power saving (in this case set the performance to maximum) I got the following results in ESXTOP.

As you can see %RUN and %USED are quit the same and %LAT_C is low. This is what we want to see.