VM IO Slowdown Culprits

tcpdump

13:45:06.096992 IP clamav-du.viaverio.com.http > mueller.43412: Flags [P.], seq 18349422:18350374, ack 143, win 33304, options [nop,nop,TS val 1092767572 ecr 126307304], length 952
13:45:06.097000 IP mueller.43412 > clamav-du.viaverio.com.http: Flags [.], ack 18350374, win 661, options [nop,nop,TS val 126307325 ecr 1092767572], length 0
13:45:06.179816 IP clamav-du.viaverio.com.http > mueller.43412: Flags [.], seq 18350374:18353270, ack 143, win 33304, options [nop,nop,TS val 1092767653 ecr 126307325], length 2896
13:45:06.179863 IP mueller.43412 > clamav-du.viaverio.com.http: Flags [.], ack 18353270, win 651, options [nop,nop,TS val 126307345 ecr 1092767653], length 0
13:45:06.179940 IP clamav-du.viaverio.com.http > mueller.43412: Flags [.], seq 18353270:18354718, ack 143, win 33304, options [nop,nop,TS val 1092767653 ecr 126307325], length 1448
13:45:06.180156 IP clamav-du.viaverio.com.http > mueller.43412: Flags [.], seq 18354718:18356166, ack 143, win 33304, options [nop,nop,TS val 1092767653 ecr 126307325], length 1448
13:45:06.180167 IP mueller.43412 > clamav-du.viaverio.com.http: Flags [.], ack 18356166, win 661, options [nop,nop,TS val 126307346 ecr 1092767653], length 0
13:45:06.180674 IP clamav-du.viaverio.com.http > mueller.43412: Flags [P.], seq 18356166:18358566, ack 143, win 33304, options [nop,nop,TS val 1092767653 ecr 126307325], length 2400
13:45:06.180716 IP mueller.43412 > clamav-du.viaverio.com.http: Flags [.], ack 18358566, win 661, options [nop,nop,TS val 126307346 ecr 1092767653], length 0
13:45:06.259029 IP clamav-du.viaverio.com.http > mueller.43412: Flags [.], seq 18358566:18361462, ack 143, win 33304, options [nop,nop,TS val 1092767734 ecr 126307346], length 2896
13:45:06.259073 IP mueller.43412 > clamav-du.viaverio.com.http: Flags [.], ack 18361462, win 651, options [nop,nop,TS val 126307365 ecr 1092767734], length 0
13:45:06.259094 IP clamav-du.viaverio.com.http > mueller.43412: Flags [.], seq 18361462:18362910, ack 143, win 33304, options [nop,nop,TS val 1092767734 ecr 126307346], length 1448
13:45:06.259345 IP clamav-du.viaverio.com.http > mueller.43412: Flags [.], seq 18362910:18364358, ack 143, win 33304, options [nop,nop,TS val 1092767734 ecr 126307346], length 1448
13:45:06.259355 IP mueller.43412 > clamav-du.viaverio.com.http: Flags [.], ack 18364358, win 661, options [nop,nop,TS val 126307365 ecr 1092767734], length 0
13:45:06.259671 IP clamav-du.viaverio.com.http > mueller.43412: Flags [P.], seq 18364358:18366758, ack 143, win 33304, options [nop,nop,TS val 1092767734 ecr 126307346], length 2400
13:45:06.259758 IP mueller.43412 > clamav-du.viaverio.com.http: Flags [.], ack 18366758, win 661, options [nop,nop,TS val 126307365 ecr 1092767734], length 0
13:45:06.338161 IP clamav-du.viaverio.com.http > mueller.43412: Flags [.], seq 18366758:18369654, ack 143, win 33304, options [nop,nop,TS val 1092767815 ecr 126307365], length 2896
13:45:06.338207 IP mueller.43412 > clamav-du.viaverio.com.http: Flags [.], ack 18369654, win 661, options [nop,nop,TS val 126307385 ecr 1092767815], length 0
13:45:06.338321 IP clamav-du.viaverio.com.http > mueller.43412: Flags [.], seq 18369654:18371102, ack 143, win 33304, options [nop,nop,TS val 1092767815 ecr 126307365], length 1448
13:45:06.338533 IP clamav-du.viaverio.com.http > mueller.43412: Flags [.], seq 18371102:18372550, ack 143, win 33304, options [nop,nop,TS val 1092767815 ecr 126307365], length 1448
13:45:06.338544 IP mueller.43412 > clamav-du.viaverio.com.http: Flags [.], ack 18372550, win 661, options [nop,nop,TS val 126307385 ecr 1092767815], length 0
13:45:06.338637 IP clamav-du.viaverio.com.http > mueller.43412: Flags [P.], seq 18372550:18374950, ack 143, win 33304, options [nop,nop,TS val 1092767815 ecr 126307365], length 2400
13:45:06.338737 IP mueller.43412 > clamav-du.viaverio.com.http: Flags [.], ack 18374950, win 661, options [nop,nop,TS val 126307385 ecr 1092767815], length 0
13:45:06.421968 IP clamav-du.viaverio.com.http > mueller.43412: Flags [.], seq 18374950:18377846, ack 143, win 33304, options [nop,nop,TS val 1092767896 ecr 126307385], length 2896
13:45:06.422046 IP mueller.43412 > clamav-du.viaverio.com.http: Flags [.], ack 18377846, win 661, options [nop,nop,TS val 126307406 ecr 1092767896], length 0
13:45:06.422066 IP clamav-du.viaverio.com.http > mueller.43412: Flags [.], seq 18377846:18379294, ack 143, win 33304, options [nop,nop,TS val 1092767896 ecr 126307385], length 1448
13:45:06.422244 IP clamav-du.viaverio.com.http > mueller.43412: Flags [.], seq 18379294:18380742, ack 143, win 33304, options [nop,nop,TS val 1092767896 ecr 126307385], length 1448
13:45:06.422256 IP mueller.43412 > clamav-du.viaverio.com.http: Flags [.], ack 18380742, win 661, options [nop,nop,TS val 126307406 ecr 1092767896], length 0
13:45:06.422374 IP clamav-du.viaverio.com.http > mueller.43412: Flags [.], seq 18380742:18382190, ack 143, win 33304, options [nop,nop,TS val 1092767896 ecr 126307385], length 1448

Note: ClamAV is not installed on the host system, only inside the guest does it get installed during sw_install benchmark.

I recommend fixing this and any future network problems with in-guest iptables rules:

iptables -F;
iptables -P INPUT DROP;
iptables -P OUTPUT DROP;
iptables -P FORWARD DROP;
iptables -A INPUT -i eth0 -p tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT;
iptables -A OUTPUT -o eth0 -p tcp --sport 22 -m state --state ESTABLISHED -j ACCEPT

Host-side it is a bit more difficult if you want to use user networking (simplest for benchmarking). There didn't seem to be an easy or even possible solution.

Thus, configuring the guest to disallow all network traffic except in-bound ssh traffic and outbound ssh-established traffic seems like the best option.