How I got networking to work in gem5

Authors:	Jason Lowe-Power

I think the right way to go is Open vSwitch. Below are some notes as I've gotten things to work on my office computer.

Installing Open vSwitch

sudo apt install uml-utilities
sudo apt install openvswitch-switch openvswitch-common

This is pretty straightforward on Ubuntu. I don't think it's necessary to build from source.

Setting up a bridge

For a first step, I want to set up a simple bridge interface from another interface to my hardware ethernet port.

I've created a script: create-bridge.sh

ovs-vsctl add-br br0
ovs-vsctl add-port br0 enp0s31f6

ip addr flush dev enp0s31f6
ifup br0

# Not sure about the next line...
ip route add default via <IP Addr>

Some of this is specific to my computer (e.g., the ip address and interface name). Additionally, since my computer uses a static IP, I edited the /etc/network/interfaces file. I copied the block for enp0s31f6 to br0 except for the auto line.

Two important gotchas that I found:

The second line ovs-vsctl add-port br0 enp0s31f6 will cause your networking to go down on the computer. Thus, you cannot do this from a remote computer, or you must ensure that all of the lines are in a single script.
The last line is very important. Although, I'm not sure why and sometimes it gives an error: "RTNETLINK answers: File exists".

See http://docs.openvswitch.org/en/latest/faq/issues/ for some more common issues.

My guess at next steps

I would like to get the tun/tap interface to qemu working. I think if that works then gem5 will work.

For how to set up Open vSwtich with qemu, see http://docs.openvswitch.org/en/latest/howto/kvm/.

I think I'm going to need to run a dhcp server on the host. I wonder if I can do this inside a docker image so I can make sure not to pollute all of my other configurations.

I think I might be able to just use iptables to make a NAT on the host between br0 and tap0. It's not going great, though.

Getting two qemu instances to talk

This information mostly came from here: http://wiki.flav.com/wiki/Open_vSwitch_Tutorial

Set the static IP of each instance. I put the following in /etc/network/interfaces

Now, I create a new bridge and add two tap devices to the bridge.

ovs-vsctl add-br br0
ip addr add 192.168.21.1/24  broadcast 192.168.21.255 dev br0
ip tuntap add mode tap vport1
ip tuntap add mode tap vport2
ifconfig vport1 up
ifconfig vport2 up
ovs-vsctl add-port br0 vport1 -- add-port br0 vport2

Next, I start the qemu instances.

sudo qemu-system-x86_64 -drive format=raw,file=../gem5/gem5_system_files/ubuntu.img -m 1024 -enable-kvm -nographic -append "console=ttyS0 earlyprintk=ttyS0 lpj=7999923 root=/dev/hda1" -kernel ../linux-stable/arch/x86/boot/bzImage -net nic,macaddr=00:11:22:EE:EE:EE -net tap,script=no,downscript=no,ifname=vport2

Note: you should change the ifname and the filename for the disk image in each of the qemu instances.

Finally, to get it so that the host can see the new network, I ran the following. I'm not sure why this works and doesn't kill the host's network.

NOTE: If you skip this the connection between qemu and gem5 seems to be more stable.

ovs-vsctl add-port br0 enp0s31f6
ip addr flush dev enp0s31f6
ifup br0

With the above, I can ping the two qemu instances from eachother and ping the host from the qemu instances. I can also ping the qemu instances from the host. However, the qemu instance cannot access the outside world.

Next step: Getting gem5 to talk to a qemu instance

Now, let's add another tap device for gem5 and add it to the bridge we created.

ip tuntap add mode tap gem5-tap
ifconfig gem5-tap up
ovs-vsctl add-port br0 gem5-tap

Now, kill one of the two qemu instances and boot gem5 (or boot gem5 with a different disk).

Note: The disk for gem5 needs a slightly different interfaces file. gem5 uses a different driver than qemu so you'll need to use a different interface name.

Now, when you boot gem5 you shuold be able to ping between the qemu instances and the gem5 instance. Cool!

A couple of caveats.

You cannot have time_sync_enable = True
You must directly connect the tap to the ethernet interface
You cannot run gem5 with multiple threads (I think the ethernet device needs to migrate the event queue or something like that)
The first time I tried this there were lots of errors. However, when I removed the host from the network, things worked better.

As it stands, for some reason qemu can talk to the host and vice versa, but the host and gem5 can't communicate.

Getting things talking to the outside world!

Use iptables.

iptables -t nat -A POSTROUTING -o enp0s31f6 -j MASQUERADE
iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i br0 -o enp0s31f6 -j ACCEPT

Now, of course, this breaks gem5 talking to anything. I seem to getting closer, though.

After rebooting the gem5 instance, things are working! I can ping the host and also talk to the outside world. The only thing not working is DNS, but I think that's a gem5 configuration issue.

After adding dns-nameservers 8.8.8.8 8.8.4.4 to /etc/network/interfaces everything is working in gem5 now.

Every once in a while it starts going really slow, but I can live with that.