Got latency on your VM’s? It might be RSC
Let’s say we have two VM’s with the following specs:
- Windows 2012r2 (Latest patches)
- Same VLAN
- Same version of ESXI 3029758
- Both HW version 11
- Both have the latest VMware tools for the ESXI Build 3029758
Any new latency between these two machines wouldn’t be expected considering they are both within the same physical location and all within the same topology that has been in place for months almost untouched.
After Vmotioning both to the same host, the latency goes away. That would be expected considering VM’s will use the internal networking for VM to VM networking located on the same host. Bypassing the need to go out to the VDS or vSwitch As shown below.
So let’s go back to the title of the post. What is RSC and why does it matter to VMware?
RSC ( Receive Segment Coalescing) is a technology used to help CPU utilization on a server. It achieves this task by taking the payload off of the CPU and giving it to the network adapter, in our case the VMxNet3 VMware adapter. RSC Strips headers from the packet’s, combines those packets, then sends those packets to the right destination. Without RSC the receiver would get 4-5 packets, but with RSC enabled the receiver only has to process the single packet sent with the 5 packets of information stuffed inside.
With VM Hardware version 11 there was a bug introduced that caused the ESXI to not keep the data properly is the PSH flag (PSH Flag explanation ) was not written in the first packet but was written to the packets following. If you read the article below it gives an awesome example of why this is useful.
Imagine you are walking in a line of 5 friends, friend 1 doesn’t have a pass to get in the gate, but friend 2-5 does. Being a gentleman friend 1 lets friend 2-5 go thru while he buys his ticket. But then friend 2-5 are waiting for him while they are already in the park. pretty similar concept to the PSH flags’.
While packet 2-5 have the PSH flag that grants them permission to go to the application, ESXi has a hiccup while waiting for the PSH flag on packet 1 causing it to wait before the packet can be sent and the full information is received. Here is the KB article that has the problem highlighted Vmware KB .
What is the fix?
To fix this problem the solution is pretty simple on the OS side you can disable RSC and stop the function, but doing this keep track of the memory use on that box and the box that received a bulk of those packets.
Running the command Will give you the output in below:
What is affected?
Currently, this affects people running ESXI 6.0 build 3568940 or below also running windows 2008 r2 and above. This problem can be solved 1 of two ways. By updating to ESXi 6.0 Update 2 Build 3620759 or above, or by running the above command on your machines affected by the problem.
KB Articles below for reference also for citing sources: