investigation.md

the program

it creates a tcp socket, connects to 127.0.0.1:8080, closes the socket, and repeats

the issue

every few iterations, connect blocks for just a little over a second

things that seem required for this to happen

• this only happens if the client and the server run on the same machine and the connection is on the loopback interface. if the client is running on a different machine, it's fast in all cases
• if the client forks for each iteration, this issue never happens

things i've considered/tried

• using a server not written by me (happens with every server afaict, in the examples above this was using python's http.server, but it also happens with anything else including things like netcat)
• running under strace can sometimes make the single process version go fast, sometimes it makes no difference. i can't tell why this happens. running under gdb doesn't seem to affect this
• investigating TIME_WAIT
  • setting net.ipv4.tcp_tw_reuse = 0 (disabled) or 1 (reuse always) or 2 (reuse only for loopback, which is the default) all had no effect
  • the default value net.ipv4.tcp_tw_reuse_delay is 1000, seemed like a smoking gun. i set it to 1, 200, 5000, it made no difference at all
  • there are thousands of available ports to pick
  • REUSEADDR/REUSEPORT made no difference
  • SO_LINGER made no difference
  • binding to 0 to somehow force a new ephemeral port made no difference
  • TIME_WAIT seemed like a promising avenue but it doesn't really make sense to me why it would cause this. why would the problem only happen in the single process case but not the fork case?
• investigating some kind of per process rate limiting (i don't know what it could be)
  • running the client with sudo made no difference
• checking in dmesg (there's nothing at the time this happens)
• checking selinux (it's disabled)
• checking if ip -s link show lo reports drops
  • says 0 drops
• checking tcpdump
  • says no packets were dropped
  • shows a retransmit (two consecutive packets with the same seq id) after 1 second
• adding a short sleep to work around something like PAWS that checks the tcp timestamps?
  • with the sleep the issue mostly disappears without forking (it apparently can still happen but much more rarely, like maybe once every 1000 iterations)
  • overall this made the single process version much faster, so we were never hitting any per-process rate limit
  • maybe the forking was only helping because it takes more than 0 time so the timestamp changes
  • maybe strace was helping for the same reason
  • there's a TCP_TIMESTAMP sockopt but apparently on linux it always returns EPERM even for root
  • disabling it globally via net.ipv4.tcp_timestamps without sleeping did not fix the issue

reqs.c

#include <arpa/inet.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/socket.h>
#include <sys/wait.h>
#include <time.h>
#include <sys/time.h>
#include <unistd.h>

struct {
    struct timespec times[100];
    const char *names[100];
    int i;
    struct sockaddr_in self;
} volatile *timedata;

static int wrap(const char *name, int val) {
    clock_gettime(CLOCK_MONOTONIC, (struct timespec *)timedata->times+timedata->i);
    timedata->names[timedata->i++] = name;
    return val;
}

int main(int argc, char **argv) {
    int limit = argc > 1 ? atoi(argv[1]) : 1000;

    timedata = mmap(0, 4096, PROT_READ|PROT_WRITE, MAP_ANON|MAP_SHARED, -1, 0);
    if (timedata == MAP_FAILED)
        return 1;

    struct sockaddr_in peer = {
        .sin_family = AF_INET,
        .sin_port = htons(8080),
        .sin_addr = { inet_addr("127.0.0.1") },
    };

    for (int i = 0; i < limit; i++) {
#define TIME(...) wrap(#__VA_ARGS__, __VA_ARGS__+0)

        timedata->i = 0;
#ifdef FORK
        int pid = fork();
        if (pid == 0) {
#endif

            TIME();
            int fd = TIME(socket(AF_INET, SOCK_STREAM, IPPROTO_TCP));
            if (fd == -1)
                return 1;

#if 0
            // these were tried and they didn't make any difference

            // add so_linger to avoid TIME_WAIT
            struct linger sl = { .l_onoff = 1, .l_linger = 0 };
            if (TIME(setsockopt(fd, SOL_SOCKET, SO_LINGER, &sl, sizeof sl)) == -1)
                return 1;

            // try to force a new ephemeral port by binding
            struct sockaddr_in b = {
                .sin_family = AF_INET,
                .sin_port = 0,
                .sin_addr.s_addr = INADDR_ANY
            };
            if (TIME(bind(fd, (struct sockaddr *)&b, sizeof b)) == -1)
                return 1;

            // set reuseaddr/port
            int one = 1;
            if (TIME(setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &one, sizeof one)) == -1)
                return 1;
            if (TIME(setsockopt(fd, SOL_SOCKET, SO_REUSEPORT, &one, sizeof one)) == -1)
                return 1;

            // always EPERM on linux
            // int zero = 1;
            // if (TIME(setsockopt(fd, SOL_TCP, TCP_TIMESTAMP, &zero, sizeof zero)) == -1)
            //     return 1;
#endif

            if (TIME(connect(fd, (void *)&peer, sizeof peer)) == -1)
                return 1;

            // save the ephemeral port we got after connect
            socklen_t len = sizeof timedata->self;
            if (TIME(getsockname(fd, (struct sockaddr *)&timedata->self, &len)) == -1)
                return 1;

            TIME(shutdown(fd, SHUT_RDWR));

            TIME(close(fd));
            TIME();
#ifdef FORK
            return 0;
        }
        int result;
        waitpid(pid, &result, 0);
        if (!(WIFEXITED(result) && WEXITSTATUS(result) == 0))
            return 1;
#endif

        //usleep(1000);

#define tdiff(a,b) (((b).tv_nsec - (a).tv_nsec)/1e9 + (b).tv_sec - (a).tv_sec)

        float elapsed_total = tdiff(timedata->times[0], timedata->times[timedata->i-1]);
        if (elapsed_total > 1)
            printf("\x1b[31m");
        printf("%6d/%d: port %d ", i+1, limit, ntohs(timedata->self.sin_port));
        for (int i = 1; i < timedata->i-1; i++) {
            const char *name = timedata->names[i];
            int namelen = strcspn(name, "(");
            float elapsed = tdiff(timedata->times[i-1], timedata->times[i]);
            if (elapsed < 1)
                printf("%.*s %f ", namelen, name, elapsed);
            else
                printf("\x1b[43m%.*s %f\x1b[49m ", namelen, name, elapsed);
        }
        printf("total %f\x1b[m\n", elapsed_total);
    }
}

tcpdump.log

reading from file lo.pcap, link-type EN10MB (Ethernet), snapshot length 262144
1759497762.419759 IP 127.0.0.1.51919 > 127.0.0.1.8080: Flags [S], seq 273255642, win 65495, options [mss 65495,sackOK,TS val 2869584766 ecr 0,nop,wscale 7], length 0
1759497762.419775 IP 127.0.0.1.8080 > 127.0.0.1.51919: Flags [S.], seq 1431860931, ack 273255643, win 65483, options [mss 65495,sackOK,TS val 2869584766 ecr 2869584766,nop,wscale 7], length 0
1759497762.419788 IP 127.0.0.1.51919 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869584766 ecr 2869584766], length 0
1759497762.419807 IP 127.0.0.1.51919 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869584766 ecr 2869584766], length 0
1759497762.419822 IP 127.0.0.1.51919 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869584766 ecr 2869584766], length 0
1759497762.419891 IP 127.0.0.1.51997 > 127.0.0.1.8080: Flags [S], seq 502195188, win 65495, options [mss 65495,sackOK,TS val 2869584766 ecr 0,nop,wscale 7], length 0
1759497762.419898 IP 127.0.0.1.8080 > 127.0.0.1.51997: Flags [S.], seq 897020019, ack 502195189, win 65483, options [mss 65495,sackOK,TS val 2869584766 ecr 2869584766,nop,wscale 7], length 0
1759497762.419908 IP 127.0.0.1.51997 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869584766 ecr 2869584766], length 0
1759497762.419917 IP 127.0.0.1.51997 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869584766 ecr 2869584766], length 0
1759497762.419924 IP 127.0.0.1.51997 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869584766 ecr 2869584766], length 0
1759497762.419950 IP 127.0.0.1.56593 > 127.0.0.1.8080: Flags [S], seq 894212216, win 65495, options [mss 65495,sackOK,TS val 2869584766 ecr 0,nop,wscale 7], length 0
1759497762.419955 IP 127.0.0.1.8080 > 127.0.0.1.56593: Flags [S.], seq 348042121, ack 894212217, win 65483, options [mss 65495,sackOK,TS val 2869584766 ecr 2869584766,nop,wscale 7], length 0
1759497762.419960 IP 127.0.0.1.56593 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869584766 ecr 2869584766], length 0
1759497762.419968 IP 127.0.0.1.56593 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869584766 ecr 2869584766], length 0
1759497762.419974 IP 127.0.0.1.56593 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869584766 ecr 2869584766], length 0
1759497762.419996 IP 127.0.0.1.58943 > 127.0.0.1.8080: Flags [S], seq 3157926384, win 65495, options [mss 65495,sackOK,TS val 2869584766 ecr 0,nop,wscale 7], length 0
1759497762.420002 IP 127.0.0.1.8080 > 127.0.0.1.58943: Flags [S.], seq 741725044, ack 3157926385, win 65483, options [mss 65495,sackOK,TS val 2869584766 ecr 2869584766,nop,wscale 7], length 0
1759497762.420007 IP 127.0.0.1.58943 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869584766 ecr 2869584766], length 0
1759497762.420014 IP 127.0.0.1.58943 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869584766 ecr 2869584766], length 0
1759497762.420020 IP 127.0.0.1.58943 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869584766 ecr 2869584766], length 0
1759497762.420044 IP 127.0.0.1.47611 > 127.0.0.1.8080: Flags [S], seq 2634298211, win 65495, options [mss 65495,sackOK,TS val 2869584766 ecr 0,nop,wscale 7], length 0
1759497762.420049 IP 127.0.0.1.8080 > 127.0.0.1.47611: Flags [S.], seq 853640823, ack 2634298212, win 65483, options [mss 65495,sackOK,TS val 2869584766 ecr 2869584766,nop,wscale 7], length 0
1759497762.420054 IP 127.0.0.1.47611 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869584766 ecr 2869584766], length 0
1759497762.420062 IP 127.0.0.1.47611 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869584766 ecr 2869584766], length 0
1759497762.420067 IP 127.0.0.1.47611 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869584766 ecr 2869584766], length 0
1759497762.420090 IP 127.0.0.1.39991 > 127.0.0.1.8080: Flags [S], seq 2934271239, win 65495, options [mss 65495,sackOK,TS val 2869584766 ecr 0,nop,wscale 7], length 0
1759497762.420095 IP 127.0.0.1.8080 > 127.0.0.1.39991: Flags [S.], seq 2550731976, ack 2934271240, win 65483, options [mss 65495,sackOK,TS val 2869584766 ecr 2869584766,nop,wscale 7], length 0
1759497762.420101 IP 127.0.0.1.39991 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869584766 ecr 2869584766], length 0
1759497762.420109 IP 127.0.0.1.39991 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869584766 ecr 2869584766], length 0
1759497762.420116 IP 127.0.0.1.39991 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869584766 ecr 2869584766], length 0
1759497762.420144 IP 127.0.0.1.58869 > 127.0.0.1.8080: Flags [S], seq 4283392454, win 65495, options [mss 65495,sackOK,TS val 2869584766 ecr 0,nop,wscale 7], length 0
1759497762.420150 IP 127.0.0.1.8080 > 127.0.0.1.58869: Flags [S.], seq 1645254242, ack 4283392455, win 65483, options [mss 65495,sackOK,TS val 2869584766 ecr 2869584766,nop,wscale 7], length 0
1759497762.420156 IP 127.0.0.1.58869 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869584766 ecr 2869584766], length 0
1759497762.420165 IP 127.0.0.1.58869 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869584766 ecr 2869584766], length 0
1759497762.420172 IP 127.0.0.1.58869 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869584766 ecr 2869584766], length 0
1759497762.420199 IP 127.0.0.1.49719 > 127.0.0.1.8080: Flags [S], seq 1172044506, win 65495, options [mss 65495,sackOK,TS val 2869584766 ecr 0,nop,wscale 7], length 0
1759497763.441633 IP 127.0.0.1.49719 > 127.0.0.1.8080: Flags [S], seq 1172044506, win 65495, options [mss 65495,sackOK,TS val 2869585788 ecr 0,nop,wscale 7], length 0
1759497763.441651 IP 127.0.0.1.8080 > 127.0.0.1.49719: Flags [S.], seq 2945869639, ack 1172044507, win 65483, options [mss 65495,sackOK,TS val 2869585788 ecr 2869585788,nop,wscale 7], length 0
1759497763.441665 IP 127.0.0.1.49719 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869585788 ecr 2869585788], length 0
1759497763.441776 IP 127.0.0.1.49719 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869585788 ecr 2869585788], length 0
1759497763.441796 IP 127.0.0.1.49719 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869585788 ecr 2869585788], length 0
1759497763.441868 IP 127.0.0.1.46017 > 127.0.0.1.8080: Flags [S], seq 3434880291, win 65495, options [mss 65495,sackOK,TS val 2869585788 ecr 0,nop,wscale 7], length 0
1759497763.441888 IP 127.0.0.1.8080 > 127.0.0.1.46017: Flags [S.], seq 1960592129, ack 3434880292, win 65483, options [mss 65495,sackOK,TS val 2869585788 ecr 2869585788,nop,wscale 7], length 0
1759497763.441895 IP 127.0.0.1.46017 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869585788 ecr 2869585788], length 0
1759497763.441906 IP 127.0.0.1.46017 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869585788 ecr 2869585788], length 0
1759497763.441918 IP 127.0.0.1.46017 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869585788 ecr 2869585788], length 0
1759497763.441949 IP 127.0.0.1.53155 > 127.0.0.1.8080: Flags [S], seq 1704213031, win 65495, options [mss 65495,sackOK,TS val 2869585788 ecr 0,nop,wscale 7], length 0
1759497763.441956 IP 127.0.0.1.8080 > 127.0.0.1.53155: Flags [S.], seq 1609667992, ack 1704213032, win 65483, options [mss 65495,sackOK,TS val 2869585788 ecr 2869585788,nop,wscale 7], length 0
1759497763.441965 IP 127.0.0.1.53155 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869585788 ecr 2869585788], length 0
1759497763.441974 IP 127.0.0.1.53155 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869585788 ecr 2869585788], length 0
1759497763.442023 IP 127.0.0.1.53155 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869585788 ecr 2869585788], length 0
1759497763.442086 IP 127.0.0.1.53427 > 127.0.0.1.8080: Flags [S], seq 1871135644, win 65495, options [mss 65495,sackOK,TS val 2869585788 ecr 0,nop,wscale 7], length 0
1759497763.442093 IP 127.0.0.1.8080 > 127.0.0.1.53427: Flags [S.], seq 3366612985, ack 1871135645, win 65483, options [mss 65495,sackOK,TS val 2869585788 ecr 2869585788,nop,wscale 7], length 0
1759497763.442101 IP 127.0.0.1.53427 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869585788 ecr 2869585788], length 0
1759497763.442121 IP 127.0.0.1.53427 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869585788 ecr 2869585788], length 0
1759497763.442126 IP 127.0.0.1.53427 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869585788 ecr 2869585788], length 0
1759497763.442141 IP 127.0.0.1.40307 > 127.0.0.1.8080: Flags [S], seq 3993083719, win 65495, options [mss 65495,sackOK,TS val 2869585788 ecr 0,nop,wscale 7], length 0
1759497763.442144 IP 127.0.0.1.8080 > 127.0.0.1.40307: Flags [S.], seq 112439983, ack 3993083720, win 65483, options [mss 65495,sackOK,TS val 2869585788 ecr 2869585788,nop,wscale 7], length 0
1759497763.442146 IP 127.0.0.1.40307 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869585788 ecr 2869585788], length 0
1759497763.442150 IP 127.0.0.1.40307 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869585788 ecr 2869585788], length 0
1759497763.442154 IP 127.0.0.1.40307 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869585788 ecr 2869585788], length 0
1759497763.442165 IP 127.0.0.1.49759 > 127.0.0.1.8080: Flags [S], seq 1749046737, win 65495, options [mss 65495,sackOK,TS val 2869585788 ecr 0,nop,wscale 7], length 0
1759497763.442168 IP 127.0.0.1.8080 > 127.0.0.1.49759: Flags [S.], seq 3586118163, ack 1749046738, win 65483, options [mss 65495,sackOK,TS val 2869585788 ecr 2869585788,nop,wscale 7], length 0
1759497763.442170 IP 127.0.0.1.49759 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869585788 ecr 2869585788], length 0
1759497763.442174 IP 127.0.0.1.49759 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869585788 ecr 2869585788], length 0
1759497763.442177 IP 127.0.0.1.49759 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869585788 ecr 2869585788], length 0
1759497763.442187 IP 127.0.0.1.41785 > 127.0.0.1.8080: Flags [S], seq 825908605, win 65495, options [mss 65495,sackOK,TS val 2869585788 ecr 0,nop,wscale 7], length 0
1759497763.442190 IP 127.0.0.1.8080 > 127.0.0.1.41785: Flags [S.], seq 2453735373, ack 825908606, win 65483, options [mss 65495,sackOK,TS val 2869585788 ecr 2869585788,nop,wscale 7], length 0
1759497763.442193 IP 127.0.0.1.41785 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869585788 ecr 2869585788], length 0
1759497763.442198 IP 127.0.0.1.41785 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869585788 ecr 2869585788], length 0
1759497763.442201 IP 127.0.0.1.41785 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869585788 ecr 2869585788], length 0
1759497763.442211 IP 127.0.0.1.56459 > 127.0.0.1.8080: Flags [S], seq 245948863, win 65495, options [mss 65495,sackOK,TS val 2869585788 ecr 0,nop,wscale 7], length 0
1759497764.465677 IP 127.0.0.1.56459 > 127.0.0.1.8080: Flags [S], seq 245948863, win 65495, options [mss 65495,sackOK,TS val 2869586812 ecr 0,nop,wscale 7], length 0
1759497764.465694 IP 127.0.0.1.8080 > 127.0.0.1.56459: Flags [S.], seq 2659532645, ack 245948864, win 65483, options [mss 65495,sackOK,TS val 2869586812 ecr 2869586812,nop,wscale 7], length 0
1759497764.465708 IP 127.0.0.1.56459 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869586812 ecr 2869586812], length 0
1759497764.465924 IP 127.0.0.1.56459 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869586812 ecr 2869586812], length 0
1759497764.465942 IP 127.0.0.1.56459 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869586812 ecr 2869586812], length 0
1759497764.466012 IP 127.0.0.1.47395 > 127.0.0.1.8080: Flags [S], seq 4131557124, win 65495, options [mss 65495,sackOK,TS val 2869586812 ecr 0,nop,wscale 7], length 0
1759497764.466017 IP 127.0.0.1.8080 > 127.0.0.1.47395: Flags [S.], seq 2459377674, ack 4131557125, win 65483, options [mss 65495,sackOK,TS val 2869586812 ecr 2869586812,nop,wscale 7], length 0
1759497764.466024 IP 127.0.0.1.47395 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869586812 ecr 2869586812], length 0
1759497764.466030 IP 127.0.0.1.47395 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869586812 ecr 2869586812], length 0
1759497764.466033 IP 127.0.0.1.47395 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869586812 ecr 2869586812], length 0
1759497764.466045 IP 127.0.0.1.43089 > 127.0.0.1.8080: Flags [S], seq 3473646663, win 65495, options [mss 65495,sackOK,TS val 2869586812 ecr 0,nop,wscale 7], length 0
1759497764.466047 IP 127.0.0.1.8080 > 127.0.0.1.43089: Flags [S.], seq 3597356325, ack 3473646664, win 65483, options [mss 65495,sackOK,TS val 2869586812 ecr 2869586812,nop,wscale 7], length 0
1759497764.466050 IP 127.0.0.1.43089 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869586812 ecr 2869586812], length 0
1759497764.466053 IP 127.0.0.1.43089 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869586812 ecr 2869586812], length 0
1759497764.466056 IP 127.0.0.1.43089 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869586812 ecr 2869586812], length 0
1759497764.466065 IP 127.0.0.1.56911 > 127.0.0.1.8080: Flags [S], seq 2620405539, win 65495, options [mss 65495,sackOK,TS val 2869586812 ecr 0,nop,wscale 7], length 0
1759497764.466067 IP 127.0.0.1.8080 > 127.0.0.1.56911: Flags [S.], seq 3432675542, ack 2620405540, win 65483, options [mss 65495,sackOK,TS val 2869586812 ecr 2869586812,nop,wscale 7], length 0
1759497764.466069 IP 127.0.0.1.56911 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869586812 ecr 2869586812], length 0
1759497764.466072 IP 127.0.0.1.56911 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869586812 ecr 2869586812], length 0
1759497764.466076 IP 127.0.0.1.56911 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869586812 ecr 2869586812], length 0
1759497764.466085 IP 127.0.0.1.56479 > 127.0.0.1.8080: Flags [S], seq 2385623348, win 65495, options [mss 65495,sackOK,TS val 2869586812 ecr 0,nop,wscale 7], length 0
1759497764.466087 IP 127.0.0.1.8080 > 127.0.0.1.56479: Flags [S.], seq 960778007, ack 2385623349, win 65483, options [mss 65495,sackOK,TS val 2869586812 ecr 2869586812,nop,wscale 7], length 0
1759497764.466089 IP 127.0.0.1.56479 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869586812 ecr 2869586812], length 0
1759497764.466092 IP 127.0.0.1.56479 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869586812 ecr 2869586812], length 0
1759497764.466094 IP 127.0.0.1.56479 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869586812 ecr 2869586812], length 0
1759497764.466102 IP 127.0.0.1.42857 > 127.0.0.1.8080: Flags [S], seq 3426803016, win 65495, options [mss 65495,sackOK,TS val 2869586812 ecr 0,nop,wscale 7], length 0
1759497764.466104 IP 127.0.0.1.8080 > 127.0.0.1.42857: Flags [S.], seq 36354607, ack 3426803017, win 65483, options [mss 65495,sackOK,TS val 2869586812 ecr 2869586812,nop,wscale 7], length 0
1759497764.466106 IP 127.0.0.1.42857 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869586812 ecr 2869586812], length 0
1759497764.466109 IP 127.0.0.1.42857 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869586812 ecr 2869586812], length 0
1759497764.466111 IP 127.0.0.1.42857 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869586812 ecr 2869586812], length 0
1759497764.466119 IP 127.0.0.1.36117 > 127.0.0.1.8080: Flags [S], seq 305340803, win 65495, options [mss 65495,sackOK,TS val 2869586812 ecr 0,nop,wscale 7], length 0
1759497764.466121 IP 127.0.0.1.8080 > 127.0.0.1.36117: Flags [S.], seq 3942322426, ack 305340804, win 65483, options [mss 65495,sackOK,TS val 2869586812 ecr 2869586812,nop,wscale 7], length 0
1759497764.466123 IP 127.0.0.1.36117 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869586812 ecr 2869586812], length 0
1759497764.466126 IP 127.0.0.1.36117 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869586812 ecr 2869586812], length 0
1759497764.466128 IP 127.0.0.1.36117 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869586812 ecr 2869586812], length 0
1759497764.466138 IP 127.0.0.1.38875 > 127.0.0.1.8080: Flags [S], seq 4095913450, win 65495, options [mss 65495,sackOK,TS val 2869586812 ecr 0,nop,wscale 7], length 0
1759497765.489754 IP 127.0.0.1.38875 > 127.0.0.1.8080: Flags [S], seq 4095913450, win 65495, options [mss 65495,sackOK,TS val 2869587836 ecr 0,nop,wscale 7], length 0
1759497765.489779 IP 127.0.0.1.8080 > 127.0.0.1.38875: Flags [S.], seq 4253391591, ack 4095913451, win 65483, options [mss 65495,sackOK,TS val 2869587836 ecr 2869587836,nop,wscale 7], length 0
1759497765.489793 IP 127.0.0.1.38875 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869587836 ecr 2869587836], length 0
1759497765.489963 IP 127.0.0.1.38875 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869587836 ecr 2869587836], length 0
1759497765.489975 IP 127.0.0.1.38875 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869587836 ecr 2869587836], length 0
1759497765.490043 IP 127.0.0.1.49843 > 127.0.0.1.8080: Flags [S], seq 1562833814, win 65495, options [mss 65495,sackOK,TS val 2869587836 ecr 0,nop,wscale 7], length 0
1759497765.490047 IP 127.0.0.1.8080 > 127.0.0.1.49843: Flags [S.], seq 959884834, ack 1562833815, win 65483, options [mss 65495,sackOK,TS val 2869587836 ecr 2869587836,nop,wscale 7], length 0
1759497765.490051 IP 127.0.0.1.49843 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869587836 ecr 2869587836], length 0
1759497765.490056 IP 127.0.0.1.49843 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869587836 ecr 2869587836], length 0
1759497765.490058 IP 127.0.0.1.49843 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869587836 ecr 2869587836], length 0
1759497765.490069 IP 127.0.0.1.49149 > 127.0.0.1.8080: Flags [S], seq 4253773234, win 65495, options [mss 65495,sackOK,TS val 2869587836 ecr 0,nop,wscale 7], length 0
1759497765.490071 IP 127.0.0.1.8080 > 127.0.0.1.49149: Flags [S.], seq 306973039, ack 4253773235, win 65483, options [mss 65495,sackOK,TS val 2869587836 ecr 2869587836,nop,wscale 7], length 0
1759497765.490073 IP 127.0.0.1.49149 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869587836 ecr 2869587836], length 0
1759497765.490087 IP 127.0.0.1.49149 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869587836 ecr 2869587836], length 0
1759497765.490089 IP 127.0.0.1.49149 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869587836 ecr 2869587836], length 0
1759497765.490098 IP 127.0.0.1.60123 > 127.0.0.1.8080: Flags [S], seq 2068050044, win 65495, options [mss 65495,sackOK,TS val 2869587836 ecr 0,nop,wscale 7], length 0
1759497765.490101 IP 127.0.0.1.8080 > 127.0.0.1.60123: Flags [S.], seq 2885638773, ack 2068050045, win 65483, options [mss 65495,sackOK,TS val 2869587836 ecr 2869587836,nop,wscale 7], length 0
1759497765.490103 IP 127.0.0.1.60123 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869587836 ecr 2869587836], length 0
1759497765.490106 IP 127.0.0.1.60123 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869587836 ecr 2869587836], length 0
1759497765.490108 IP 127.0.0.1.60123 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869587836 ecr 2869587836], length 0
1759497765.490117 IP 127.0.0.1.40273 > 127.0.0.1.8080: Flags [S], seq 542646541, win 65495, options [mss 65495,sackOK,TS val 2869587836 ecr 0,nop,wscale 7], length 0
1759497765.490119 IP 127.0.0.1.8080 > 127.0.0.1.40273: Flags [S.], seq 848804675, ack 542646542, win 65483, options [mss 65495,sackOK,TS val 2869587836 ecr 2869587836,nop,wscale 7], length 0
1759497765.490121 IP 127.0.0.1.40273 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869587836 ecr 2869587836], length 0
1759497765.490127 IP 127.0.0.1.40273 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869587836 ecr 2869587836], length 0
1759497765.490129 IP 127.0.0.1.40273 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869587836 ecr 2869587836], length 0
1759497765.490137 IP 127.0.0.1.60217 > 127.0.0.1.8080: Flags [S], seq 3492103357, win 65495, options [mss 65495,sackOK,TS val 2869587836 ecr 0,nop,wscale 7], length 0
1759497765.490139 IP 127.0.0.1.8080 > 127.0.0.1.60217: Flags [S.], seq 1454729601, ack 3492103358, win 65483, options [mss 65495,sackOK,TS val 2869587836 ecr 2869587836,nop,wscale 7], length 0
1759497765.490141 IP 127.0.0.1.60217 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869587836 ecr 2869587836], length 0
1759497765.490144 IP 127.0.0.1.60217 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869587836 ecr 2869587836], length 0
1759497765.490146 IP 127.0.0.1.60217 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869587836 ecr 2869587836], length 0
1759497765.490154 IP 127.0.0.1.53989 > 127.0.0.1.8080: Flags [S], seq 3225161788, win 65495, options [mss 65495,sackOK,TS val 2869587836 ecr 0,nop,wscale 7], length 0
1759497765.490156 IP 127.0.0.1.8080 > 127.0.0.1.53989: Flags [S.], seq 371780175, ack 3225161789, win 65483, options [mss 65495,sackOK,TS val 2869587836 ecr 2869587836,nop,wscale 7], length 0
1759497765.490158 IP 127.0.0.1.53989 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869587836 ecr 2869587836], length 0
1759497765.490160 IP 127.0.0.1.53989 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869587836 ecr 2869587836], length 0
1759497765.490162 IP 127.0.0.1.53989 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869587836 ecr 2869587836], length 0
1759497765.490172 IP 127.0.0.1.55529 > 127.0.0.1.8080: Flags [S], seq 2595216986, win 65495, options [mss 65495,sackOK,TS val 2869587836 ecr 0,nop,wscale 7], length 0
1759497766.513586 IP 127.0.0.1.55529 > 127.0.0.1.8080: Flags [S], seq 2595216986, win 65495, options [mss 65495,sackOK,TS val 2869588860 ecr 0,nop,wscale 7], length 0
1759497766.513601 IP 127.0.0.1.8080 > 127.0.0.1.55529: Flags [S.], seq 2040564411, ack 2595216987, win 65483, options [mss 65495,sackOK,TS val 2869588860 ecr 2869588860,nop,wscale 7], length 0
1759497766.513614 IP 127.0.0.1.55529 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869588860 ecr 2869588860], length 0
1759497766.513798 IP 127.0.0.1.55529 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869588860 ecr 2869588860], length 0
1759497766.513813 IP 127.0.0.1.55529 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869588860 ecr 2869588860], length 0
1759497766.513886 IP 127.0.0.1.52615 > 127.0.0.1.8080: Flags [S], seq 1432907320, win 65495, options [mss 65495,sackOK,TS val 2869588860 ecr 0,nop,wscale 7], length 0
1759497766.513892 IP 127.0.0.1.8080 > 127.0.0.1.52615: Flags [S.], seq 2516099460, ack 1432907321, win 65483, options [mss 65495,sackOK,TS val 2869588860 ecr 2869588860,nop,wscale 7], length 0
1759497766.513899 IP 127.0.0.1.52615 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869588860 ecr 2869588860], length 0
1759497766.513908 IP 127.0.0.1.52615 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869588860 ecr 2869588860], length 0
1759497766.513910 IP 127.0.0.1.52615 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869588860 ecr 2869588860], length 0
1759497766.513926 IP 127.0.0.1.60609 > 127.0.0.1.8080: Flags [S], seq 2751351979, win 65495, options [mss 65495,sackOK,TS val 2869588860 ecr 0,nop,wscale 7], length 0
1759497766.513929 IP 127.0.0.1.8080 > 127.0.0.1.60609: Flags [S.], seq 1660744852, ack 2751351980, win 65483, options [mss 65495,sackOK,TS val 2869588860 ecr 2869588860,nop,wscale 7], length 0
1759497766.513931 IP 127.0.0.1.60609 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869588860 ecr 2869588860], length 0
1759497766.513934 IP 127.0.0.1.60609 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869588860 ecr 2869588860], length 0
1759497766.513936 IP 127.0.0.1.60609 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869588860 ecr 2869588860], length 0
1759497766.513945 IP 127.0.0.1.40807 > 127.0.0.1.8080: Flags [S], seq 1537581256, win 65495, options [mss 65495,sackOK,TS val 2869588860 ecr 0,nop,wscale 7], length 0
1759497766.513947 IP 127.0.0.1.8080 > 127.0.0.1.40807: Flags [S.], seq 1347694160, ack 1537581257, win 65483, options [mss 65495,sackOK,TS val 2869588860 ecr 2869588860,nop,wscale 7], length 0
1759497766.513949 IP 127.0.0.1.40807 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869588860 ecr 2869588860], length 0
1759497766.513954 IP 127.0.0.1.40807 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869588860 ecr 2869588860], length 0
1759497766.513957 IP 127.0.0.1.40807 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869588860 ecr 2869588860], length 0
1759497766.513967 IP 127.0.0.1.59833 > 127.0.0.1.8080: Flags [S], seq 4116169446, win 65495, options [mss 65495,sackOK,TS val 2869588860 ecr 0,nop,wscale 7], length 0
1759497766.513970 IP 127.0.0.1.8080 > 127.0.0.1.59833: Flags [S.], seq 653827507, ack 4116169447, win 65483, options [mss 65495,sackOK,TS val 2869588860 ecr 2869588860,nop,wscale 7], length 0
1759497766.513973 IP 127.0.0.1.59833 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869588860 ecr 2869588860], length 0
1759497766.513976 IP 127.0.0.1.59833 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869588860 ecr 2869588860], length 0
1759497766.513979 IP 127.0.0.1.59833 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869588860 ecr 2869588860], length 0
1759497766.513989 IP 127.0.0.1.58447 > 127.0.0.1.8080: Flags [S], seq 2989914112, win 65495, options [mss 65495,sackOK,TS val 2869588860 ecr 0,nop,wscale 7], length 0
1759497766.513992 IP 127.0.0.1.8080 > 127.0.0.1.58447: Flags [S.], seq 4287468479, ack 2989914113, win 65483, options [mss 65495,sackOK,TS val 2869588860 ecr 2869588860,nop,wscale 7], length 0
1759497766.513994 IP 127.0.0.1.58447 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869588860 ecr 2869588860], length 0
1759497766.513997 IP 127.0.0.1.58447 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869588860 ecr 2869588860], length 0
1759497766.514000 IP 127.0.0.1.58447 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869588860 ecr 2869588860], length 0
1759497766.514015 IP 127.0.0.1.39545 > 127.0.0.1.8080: Flags [S], seq 2106457024, win 65495, options [mss 65495,sackOK,TS val 2869588860 ecr 0,nop,wscale 7], length 0
1759497766.514017 IP 127.0.0.1.8080 > 127.0.0.1.39545: Flags [S.], seq 562101254, ack 2106457025, win 65483, options [mss 65495,sackOK,TS val 2869588860 ecr 2869588860,nop,wscale 7], length 0
1759497766.514019 IP 127.0.0.1.39545 > 127.0.0.1.8080: Flags [.], ack 1, win 512, options [nop,nop,TS val 2869588860 ecr 2869588860], length 0
1759497766.514022 IP 127.0.0.1.39545 > 127.0.0.1.8080: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 2869588860 ecr 2869588860], length 0
1759497766.514024 IP 127.0.0.1.39545 > 127.0.0.1.8080: Flags [R.], seq 2, ack 1, win 512, options [nop,nop,TS val 2869588860 ecr 2869588860], length 0
1759497766.514036 IP 127.0.0.1.37067 > 127.0.0.1.8080: Flags [S], seq 469239738, win 65495, options [mss 65495,sackOK,TS val 2869588860 ecr 0,nop,wscale 7], length 0

without forking

with forking