Fixing khugepaged CPU usage VMware Workstation

vmware-workstation-khugepaged-fix.md

If you run VMware Workstation 11 or above you may encounter high CPU usage from process khugepaged on Ubuntu 15.04+

The fix is to disable transparent hugepages. It seems Ubuntu has it enabled by default.

You can check the current status on your system by running:

cat /sys/kernel/mm/transparent_hugepage/enabled

cat /sys/kernel/mm/transparent_hugepage/defrag

Fedora outputs: always [madvise] never but Ubuntu outputs: [always] madvise never

Fedora seems to not be effected but I havn't tested it myself.

So I suggest not using madvise and just disable it totally.

To disable it run the following commands as root:

echo never > /sys/kernel/mm/transparent_hugepage/enabled

echo never > /sys/kernel/mm/transparent_hugepage/defrag

That will only disable it for the current session.

To have it persistant across reboots I suggest adding this to your rc.local:

# Fix for VMware Workstation 11+ khugepaged.
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

Ensure this goes above the line:

exit 0

More info and references:

• https://communities.vmware.com/thread/505923
• https://docs.mongodb.com/manual/tutorial/transparent-huge-pages/

I tried the above options by @msizanoen1. Indeed the VM is now usable. However, it is still slow and I see that kswapd0 now occasionally will peg 1 CPU at 100%, despite the fact that I have no swap enabled. This process is only running when I run a VMware VM, which suggests this is somehow linked...

I tried the above options by @msizanoen1. Indeed the VM is not usable. However, it is still slow and I see that kswapd0 now occasionally will peg 1 CPU at 100%, despite the fact that I have no swap enabled. This process is only running when I run a VMware VM, which suggests this is somehow linked...

It's likely that not using swap was the cause for kswapd consuming 100%. Generally it's not recommended to run a Linux system without some kind of swap, and this might be especially true when running with memory defragmentation disabled.

Try setting /proc/sys/vm/compaction_proactiveness to 1. The thing is, you need to compact eventually. What you want to avoid is a war between VMware and kernel. This will obvisouly happen if you compact too eagerly, which is the default. But if you compact too lazily, for example by not being proactive at all, you will run into a situation where you must do it reactively, with the same bad result. This is as if everyone has kept missing the proactiveness part. Setting it to 1 has now worked for me for some time, at least weeks, if not months.

Try setting /proc/sys/vm/compaction_proactiveness to 1. The thing is, you need to compact eventually. What you want to avoid is a war between VMware and kernel. This will obvisouly happen if you compact too eagerly, which is the default. But if you compact too lazily, for example by not being proactive at all, you will run into a situation where you must do it reactively, with the same bad result. This is as if everyone has kept missing the proactiveness part. Setting it to 1 has now worked for me for some time, at least weeks, if not months.

AFAIK (and through my own testing and reading of the kernel source) setting vm.extfrag_threshold=1000 and disabling transparent hugepage defrag will prevent the kernel from ever compacting memory, reactively or not, and will cause it to fall back to swapping pages out of memory and/or invoking the OOM killer instead.

vm.compaction_proactiveness and /sys/kernel/mm/transparent_hugepage/defrag controls different aspects of proactive memory compaction, while vm.extfrag_threshold controls reactive memory compaction (e.g. when the kernel needs to allocate a large chunk of continuous memory). Setting vm.extfrag_threshold to 1000 disables reactive memory compaction.