Test vhost-user-net

vhost-user-net-testing.md

This document is a list of ways we can properly test the network is functional when using vhost-user-net. The document assumes we'll be using vhost-user-bridge program from qemu/tests as the vhost-user-net daemon.

1. Communication between 2 VMs

The point in this case is to validate communication between 2 virtual machines by connecting each VM to its own vhost-user-net daemon. Each daemon read/write to each other ports, based on the socat port forwarding.

          +-----------+                      +-----------+
          |           |                      |           |
          |           |                      |           |
          |    VM1    |                      |    VM2    |
          |           |                      |           |
          |           |                      |           |
          +-----------+                      +-----------+
                |                                  |
                | /tmp/vubr1.sock                  | /tmp/vubr2.sock
                |                                  |
                |                                  |
       +----------------+                 +----------------+
       | vhost-user-net |                 | vhost-user-net |
       |    daemon 1    |                 |    daemon 2    |
       +----------------+                 +----------------+
         |            |                      |           |
 listen  |            | send          listen |           | send
  4444   |            | 5555           6666  |           | 7777
         |            |                      |           |
         |          +--------------------------+         |
         |          |   forward 5555 => 6666   |         |
         |          +--------------------------+         |
         |                                               |
         |          +--------------------------+         |
         +----------+   forward 7777 => 4444   +---------+
                    +--------------------------+

One thing to note, socat is not needed in theory, but this allows to start each VM with their own vhost-user daemon before to connect them together. By not using socat and connecting the daemon together directly, if one of the two VMs tries to send some data to the other VM while the VM has not completely booted, the daemon simply crashes because it does not know what to do with this.

In practice, here is how we can run this whole setup:

Start the vhost-user-net daemons

# From one terminal
./vubridge -u /tmp/vubr1.sock -l 127.0.0.1:4444 -r 127.0.0.1:5555

# From another terminal
./vubridge -u /tmp/vubr2.sock -l 127.0.0.1:6666 -r 127.0.0.1:7777

Launch the VMs

# From one terminal
./target/debug/cloud-hypervisor \
          --kernel vmlinux \
          --disk clear-kvm.img \
          --cmdline "console=hvc0 root=/dev/vda3" \
          --cpus 1 \
          --memory "size=1G,file=/dev/shm" \
          --vhost-user-net mac=52:54:00:02:d9:01,sock=/tmp/vubr1.sock
          
# From another terminal
./target/debug/cloud-hypervisor \
          --kernel vmlinux-2 \
          --disk clear-kvm-2.img \
          --cmdline "console=hvc0 root=/dev/vda3" \
          --cpus 1 \
          --memory "size=1G,file=/dev/shm" \
          --vhost-user-net mac=52:54:00:02:d9:01,sock=/tmp/vubr2.sock

Connect the daemons with socat

# From one terminal
socat udp-listen:5555,reuseaddr,fork udp:localhost:6666

# From another terminal
socat udp-listen:7777,reuseaddr,fork udp:localhost:4444

Setup VM1

# From inside the guest
sudo ip addr add 192.168.222.1/24 dev enp0s4
nc -u -l 192.168.222.1 1234

Setup VM2

# From inside the guest
sudo ip addr add 192.168.222.2/24 dev enp0s4
nc -u 192.168.222.1 1234

Result: At this point, anything you type from VM2's terminal should appear on VM1's terminal.

2. Communication between host and guest

The point in this case is to validate the communication between host and guest is functional.

    +-----------+
    |           |
    |           |
    |    VM1    |
    |           |
    |           |
    +-----+-----+
          |
          | /tmp/vubr1.sock
          |
          |
 +--------+-------+
 | vhost-user-net |
 |    daemon 1    |
 +--------+-------+
          |
  listen  |
   4444   |
          |
          |
+---------+--------+
|                  |
| Send packet with |
|                  |
|    IP + data     |
|                  |
+------------------+

This is a reduced way of testing, in order to avoid the complexity of running two virtual machines.

That being said, there is one important trick here, which is about injecting a packet containing IP address information. The reason is that we need to make sure the guest network interface will receive the packet because it will have the IP associated with the packet.

Here is how to proceed:

Start the vhost-user-net daemon

# From one terminal
./vubridge -u /tmp/vubr1.sock -l 127.0.0.1:4444 -r 127.0.0.1:5555

Launch VM

# From one terminal
./target/debug/cloud-hypervisor \
          --kernel vmlinux \
          --disk clear-kvm.img \
          --cmdline "console=hvc0 root=/dev/vda3" \
          --cpus 1 \
          --memory "size=1G,file=/dev/shm" \
          --vhost-user-net mac=52:54:00:02:d9:01,sock=/tmp/vubr1.sock

Setup VM

# From inside the guest
sudo ip addr add 192.168.222.1/24 dev enp0s4
nc -u -l 192.168.222.1 4444

Now the VM is waiting for some packet to land on the network interface enp0s4. Here is the way to send a packet including the IP address of the guest network interface:

From the host

echo -e "\x52\x54\x00\x02\xd9\x01\xec\xb1\xd7\x98\x3a\xc0\x08\x00\x45\x00\x00\x40\x00\x00\x00\x00\x40\x11\x3d\x58\xc0\xa8\xde\x02\xc0\xa8\xde\x01\x11\x5c\x11\x5c\x00\x2c\xa0\x5e\x48\x65\x6c\x6c\x6f\x20\x57\x6f\x72\x6c\x64\x20\x74\x68\x72\x6f\x75\x67\x68\x20\x76\x68\x6f\x73\x74\x2d\x75\x73\x65\x72\x2d\x6e\x65\x74\x21\x0a" | nc -u localhost 4444

You can rely on packetor to verify/update the content of this frame:

52 54 00 02 d9 01 ec b1 d7 98 3a c0 08 00 45 00
00 40 00 00 00 00 40 11 3d 58 c0 a8 de 02 c0 a8
de 01 11 5c 11 5c 00 2c a0 5e 48 65 6c 6c 6f 20
57 6f 72 6c 64 20 74 68 72 6f 75 67 68 20 76 68
6f 73 74 2d 75 73 65 72 2d 6e 65 74 21 0a

The reason why we need to inject the IP ourselves is because from a host perspective, the port 4444 is listened by vhost-user-net daemon on address localhost. But once the packet reaches the guest, we still need the guest kernel to route it to the right network interface based on the IP. That's why we send this UDP frame to localhost, but writing ourselves the destination IP as being 192.168.222.1.

Run Communication between host and guest.

127.0.0.1:4444 is the port under listening, but it's from host, not from guest, once login guest, issue netstat -an, you can see 4444 is not listed there. I use tcpdump instead of nc -u -l 192.168.222.1 4444 to monitor the package sending from host to guest (actually, host->udp backend, vhost-user-bridge -> guest finally through virtqueues).

root@cloud-hypervisor ~ # tcpdump
[ 37.211422] device enp0s3 entered promiscuous mode
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp0s3, link-type EN10MB (Ethernet), capture size 262144 bytes
12:16:51.738032 IP 192.168.222.2.4444 > cloud-hypervisor.4444: UDP, length 36
12:16:51.738077 ARP, Request who-has 192.168.222.2 tell cloud-hypervisor, length 28

Try ifconfig you can see the Rx package count update accordingly.
root@cloud-hypervisor ~ # ifconfig
enp0s3 Link encap:Ethernet HWaddr 52:54:00:02:D9:01
inet addr:192.168.222.1 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: fe80::5054:ff:fe02:d901/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3 errors:0 dropped:0 overruns:0 frame:0
TX packets:29 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:237 (237.0 b) TX bytes:4266 (4.1 Kb)

In a short, the data sent from host could be received by guest. While, I'm looking at why ARP request sent out, I think the current implementation is not a fully one may cause that appears.

@bjzhjing Well I thought about tcpdump but you would need to decode what has been received and it sounds more complicated.
I assure you that listening from inside the guest works. You're mixing two things with the port 4444. One is about the host listening on locahost address port 4444, and the other one is about the guest listening on 192.168.222.1 port 4444. And this works because I constructed the packet frame with the right IP and port information inside.
The point of using nc is to be able to get the output Hello World through vhost-user-net! from what is being sent from the host. I really want to validate the content of what is being sent and not only the amount of packets.

root@cloud-hypervisor ~ # nc -u -l 192.168.222.1 4444
Hello World through vhost-user-net!

Here is the packet captured from guest OS. I have also verified the content with wireshark.

@bjzhjing
Great, can you extend the integration test with the Ubuntu image to make use of nc? This will be a nice test to validate vhost-user-net is functioning.

Sure, I will!