I hit this while running iperf3 across two VNICs on different NICs (no mac-loopback).
rpz@thunderhead:~$ iperf3 -c 192.168.99.101 -B 192.168.99.100 -t 10
Connecting to host 192.168.99.101, port 5201
[ 4] local 192.168.99.100 port 48040 connected to 192.168.99.101 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 52.4 MBytes 440 Mbits/sec
[ 4] 1.00-2.00 sec 56.6 MBytes 475 Mbits/sec
[ 4] 2.00-3.00 sec 56.6 MBytes 475 Mbits/sec
[ 4] 3.00-4.00 sec 56.6 MBytes 475 Mbits/sec
[ 4] 4.00-5.00 sec 56.6 MBytes 475 Mbits/sec
[ 4] 5.00-6.00 sec 56.6 MBytes 475 Mbits/sec
[ 4] 6.00-7.00 sec 53.5 MBytes 449 Mbits/sec
[ 4] 7.00-8.00 sec 56.6 MBytes 475 Mbits/sec
[ 4] 8.00-9.00 sec 56.6 MBytes 475 Mbits/sec
iperf3: error - unable to receive control message: Connection timed out
The sender was hung on the 8-9s line for a long time. Here's the ps and pstack.
rpz@thunderhead:~$ ps -l -p 101138
F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD
0 O 101 101138 101128 3 99 20 ? 729 pts/3 3:26 iperf3
rpz@thunderhead:~$ pstack 101138
101138: iperf3 -c 192.168.99.101 -B 192.168.99.100 -t 10
fe86f775 __pollsys (80363c0, 2, 8036478, 0, 3, 40) + 15
fe7fc102 pselect (5, 80364e0, 80384e0, fe8da280, 8036478, 0) + 272
fe7fc498 select (5, 80364e0, 80384e0, 0, feeee214) + 89
feed3369 iperf_run_client (8594008, 8598200, feecb34b, 8594008) + 109
08051663 run (8594008, 7, 803a5b0, 805172e) + f3
08051758 main (803a54c, fe8dd628, 803a588, 8051477, 7, 803a5b0) + 48
08051477 _start_crt (7, 803a5b0, f5ab2681, 0, 0, 0) + 96
0805134a _start (7, 803a6a0, 803a6a7, 803a6aa, 803a6b9, 803a6bc) + 1a
Here is some mdb before it went away.
> 0t101138::pid2proc |::ps -t
S PID PPID PGID SID UID FLAGS ADDR NAME
R 101138 101128 101138 101128 101 0x4a004000 fffffe5afaee7038 iperf3
T 0xfffffe59ea5947e0 <TS_ONPROC>
>
> 0xfffffe59ea5947e0::findstack
stack pointer for thread fffffe59ea5947e0 (iperf3/1): fffffe007dc18940
fffffe007dc18980 apix_intr_exit+0x24()
fffffe007dc189e0 apix_hilevel_intr_epilog+0xf1()
fffffe007dc18a50 apix_do_interrupt+0x4c0()
fffffe007dc18a60 _sys_rtt_ints_disabled+8()
fffffe007dc18b60 mutex_owned+0x2d()
fffffe007dc18bd0 socket_vop_poll+0x71()
fffffe5afaee7038 4()
fffffe5a2e0d8700 0xfffffe5a231eb618()
>
> 0xfffffe59ea5947e0::findstack
stack pointer for thread fffffe59ea5947e0 (iperf3/1): fffffe007dc18be0
fffffe007dc18c40 0xfffffe5afaee9126()
fffffe007dc18d30 pcache_poll+0x508()
fffffe007dc18e00 poll_common+0x49c()
fffffe007dc18eb0 pollsys+0xfb()
fffffe007dc18f00 _sys_sysenter_post_swapgs+0x24f()
> 0xfffffe59ea5947e0::findstack
stack pointer for thread fffffe59ea5947e0 (iperf3/1): fffffe007dc18910
fffffe007dc18950 apix_intr_exit+0x24()
fffffe007dc189b0 apix_hilevel_intr_epilog+0xf1()
fffffe007dc18a20 apix_do_interrupt+0x4c0()
fffffe007dc18a30 _sys_rtt_ints_disabled+8()
fffffe007dc18b60 so_poll()
fffffe007dc18bd0 socket_vop_poll+0x71()
fffffe5afaee7038 4()
fffffe5a2e0d8700 0xfffffe5a231eb618()