KVM - How does the VM return control back to the controlling VMM process on signals

kvm-qemu-signals.md

How to setup the controlling process to handle signals

1. Setup a signal handler for the vcpu thread for that signal. It does not have to do anything
2. When a signal is sent to the thread hosting the VM it will result in EINTR (not a reported kvm exit reason)

                signal(SIGUSR1,sig_func); // Register signal handler inside thread, the function is empty
                ret = ioctl(kvm->vcpus->vcpu_fd, KVM_RUN, 0);

                if (ret < 0) {
                        perror("KVM_RUN exited:");
                        if ((errno == EINTR) || (errno == EAGAIN)) {
                                continue;
                        }
                        exit(1);
                }

At this point that VPCU is no longer running.

What happens under the hood.

The code is hard to follow as it is a mix of kernel code and kvm code. Here is how you can do it without instrumenting the kernel or kvm.

1. Run the VM on a single vCPU on Physical CPU 14

taskset -c 14 ./kvmsample

2. Find the thread that is running that vCPU and send it USR1

ps auxw | grep kvm
mrcastel 16201 99.7  0.0 580228   692 pts/2    Sl+  12:07   3:26 ./kvmsample


ps -T 16201
  PID  SPID TTY      STAT   TIME COMMAND
16201 16201 pts/2    Sl+    0:00 ./kvmsample
16201 16203 pts/2    Rl+    4:54 ./kvmsample

kill –USR1 16203

3. If you see the kvm trace you will see something like

       kvmsample-12663 [013] .... 592491.823723: kvm_exit: reason EXTERNAL_INTERRUPT rip 0x3 info 0 800000fd
       kvmsample-12663 [013] .... 592491.823725: kvm_fpu: unload
       kvmsample-12663 [013] .... 592491.823727: kvm_userspace_exit: reason restart (4)

4. Above you see that it was an external interrupt that caused the exit. So which one was it. It could be true external interrupt like the Local APIC timer interrupt that happens periodically or something that the OS triggered to pull the CPU out of VMX Non root back into VMX Root.

If you see the KVM code it looks for pending signals each time it gets a VM exit. But the OS does not wait for a unrelated interrupt to yank the VM out of vmx non root. It does it explicitly

5. If you for example keep sending USR1 to the KVM thread you will see one and only one interrupt that happens in response to the signal which is the Linux “RES: Rescheduling interrupt”

Note: In this case the guest is a tight loop that consumes all CPU and does not exit out at all

So in summary

Signal -> Rescheduling interrupt -> VM Exit -> KVM finds signal pending on pthread -> errno=EINTR return to the VMM.

Ohhh nice!
I was using ioctl on the vcpu with KVM_INTERRUPT.
On an interrupt my KVM code would exit by issuing an IO-operation.
This usually worked in the xx-microsecond range, but sometimes the latency was spiking up to 2 seconds.
(intel core i7 3xxx,
only 1 vcpu thread running. other
applications running on the host firefox in the Background with youtube, Codeblocks. CPU ussage was below 30%)

I am really frustrated by the latency if KVM_INTERRUPT