Test vhost-user-net

vhost-user-net-testing.md

This document is a list of ways we can properly test the network is functional when using vhost-user-net. The document assumes we'll be using vhost-user-bridge program from qemu/tests as the vhost-user-net daemon.

1. Communication between 2 VMs

The point in this case is to validate communication between 2 virtual machines by connecting each VM to its own vhost-user-net daemon. Each daemon read/write to each other ports, based on the socat port forwarding.

          +-----------+                      +-----------+
          |           |                      |           |
          |           |                      |           |
          |    VM1    |                      |    VM2    |
          |           |                      |           |
          |           |                      |           |
          +-----------+                      +-----------+
                |                                  |
                | /tmp/vubr1.sock                  | /tmp/vubr2.sock
                |                                  |
                |                                  |
       +----------------+                 +----------------+
       | vhost-user-net |                 | vhost-user-net |
       |    daemon 1    |                 |    daemon 2    |
       +----------------+                 +----------------+
         |            |                      |           |
 listen  |            | send          listen |           | send
  4444   |            | 5555           6666  |           | 7777
         |            |                      |           |
         |          +--------------------------+         |
         |          |   forward 5555 => 6666   |         |
         |          +--------------------------+         |
         |                                               |
         |          +--------------------------+         |
         +----------+   forward 7777 => 4444   +---------+
                    +--------------------------+

One thing to note, socat is not needed in theory, but this allows to start each VM with their own vhost-user daemon before to connect them together. By not using socat and connecting the daemon together directly, if one of the two VMs tries to send some data to the other VM while the VM has not completely booted, the daemon simply crashes because it does not know what to do with this.

In practice, here is how we can run this whole setup:

Start the vhost-user-net daemons

# From one terminal
./vubridge -u /tmp/vubr1.sock -l 127.0.0.1:4444 -r 127.0.0.1:5555

# From another terminal
./vubridge -u /tmp/vubr2.sock -l 127.0.0.1:6666 -r 127.0.0.1:7777

Launch the VMs

# From one terminal
./target/debug/cloud-hypervisor \
          --kernel vmlinux \
          --disk clear-kvm.img \
          --cmdline "console=hvc0 root=/dev/vda3" \
          --cpus 1 \
          --memory "size=1G,file=/dev/shm" \
          --vhost-user-net mac=52:54:00:02:d9:01,sock=/tmp/vubr1.sock
          
# From another terminal
./target/debug/cloud-hypervisor \
          --kernel vmlinux-2 \
          --disk clear-kvm-2.img \
          --cmdline "console=hvc0 root=/dev/vda3" \
          --cpus 1 \
          --memory "size=1G,file=/dev/shm" \
          --vhost-user-net mac=52:54:00:02:d9:01,sock=/tmp/vubr2.sock

Connect the daemons with socat

# From one terminal
socat udp-listen:5555,reuseaddr,fork udp:localhost:6666

# From another terminal
socat udp-listen:7777,reuseaddr,fork udp:localhost:4444

Setup VM1

# From inside the guest
sudo ip addr add 192.168.222.1/24 dev enp0s4
nc -u -l 192.168.222.1 1234

Setup VM2

# From inside the guest
sudo ip addr add 192.168.222.2/24 dev enp0s4
nc -u 192.168.222.1 1234

Result: At this point, anything you type from VM2's terminal should appear on VM1's terminal.

2. Communication between host and guest

The point in this case is to validate the communication between host and guest is functional.

    +-----------+
    |           |
    |           |
    |    VM1    |
    |           |
    |           |
    +-----+-----+
          |
          | /tmp/vubr1.sock
          |
          |
 +--------+-------+
 | vhost-user-net |
 |    daemon 1    |
 +--------+-------+
          |
  listen  |
   4444   |
          |
          |
+---------+--------+
|                  |
| Send packet with |
|                  |
|    IP + data     |
|                  |
+------------------+

This is a reduced way of testing, in order to avoid the complexity of running two virtual machines.

That being said, there is one important trick here, which is about injecting a packet containing IP address information. The reason is that we need to make sure the guest network interface will receive the packet because it will have the IP associated with the packet.

Here is how to proceed:

Start the vhost-user-net daemon

# From one terminal
./vubridge -u /tmp/vubr1.sock -l 127.0.0.1:4444 -r 127.0.0.1:5555

Launch VM

# From one terminal
./target/debug/cloud-hypervisor \
          --kernel vmlinux \
          --disk clear-kvm.img \
          --cmdline "console=hvc0 root=/dev/vda3" \
          --cpus 1 \
          --memory "size=1G,file=/dev/shm" \
          --vhost-user-net mac=52:54:00:02:d9:01,sock=/tmp/vubr1.sock

Setup VM

# From inside the guest
sudo ip addr add 192.168.222.1/24 dev enp0s4
nc -u -l 192.168.222.1 4444

Now the VM is waiting for some packet to land on the network interface enp0s4. Here is the way to send a packet including the IP address of the guest network interface:

From the host

echo -e "\x52\x54\x00\x02\xd9\x01\xec\xb1\xd7\x98\x3a\xc0\x08\x00\x45\x00\x00\x40\x00\x00\x00\x00\x40\x11\x3d\x58\xc0\xa8\xde\x02\xc0\xa8\xde\x01\x11\x5c\x11\x5c\x00\x2c\xa0\x5e\x48\x65\x6c\x6c\x6f\x20\x57\x6f\x72\x6c\x64\x20\x74\x68\x72\x6f\x75\x67\x68\x20\x76\x68\x6f\x73\x74\x2d\x75\x73\x65\x72\x2d\x6e\x65\x74\x21\x0a" | nc -u localhost 4444

You can rely on packetor to verify/update the content of this frame:

52 54 00 02 d9 01 ec b1 d7 98 3a c0 08 00 45 00
00 40 00 00 00 00 40 11 3d 58 c0 a8 de 02 c0 a8
de 01 11 5c 11 5c 00 2c a0 5e 48 65 6c 6c 6f 20
57 6f 72 6c 64 20 74 68 72 6f 75 67 68 20 76 68
6f 73 74 2d 75 73 65 72 2d 6e 65 74 21 0a

The reason why we need to inject the IP ourselves is because from a host perspective, the port 4444 is listened by vhost-user-net daemon on address localhost. But once the packet reaches the guest, we still need the guest kernel to route it to the right network interface based on the IP. That's why we send this UDP frame to localhost, but writing ourselves the destination IP as being 192.168.222.1.

Here is the packet captured from guest OS. I have also verified the content with wireshark.

@bjzhjing
Great, can you extend the integration test with the Ubuntu image to make use of nc? This will be a nice test to validate vhost-user-net is functioning.

Sure, I will!