'Hello, world!' HTTP/1.1 servers in C

README.md

Blog 2023/8/13

<- previous | index | next ->

'Hello, world!' HTTP/1.1 servers in C

To learn more about implementing high-performance HTTP API's, I've implemented a few trivial 'Hello, world!' servers and run some basic benchmarks.

These servers simply respond to any HTTP request with a 200 OK text/plain response of Hello World!.

In this post I'll cover three server implementations:

• Single-threaded
• Thread-per-connection
• Fixed-size pool of threads

Spoiler

Implementations

Single-threaded

We start with a bare-bones HTTP socket server implementation: only one connection is handled at a time.

See single-threaded.c

Thread-per-connection

This implementation spawns a new thread for each connection, and each thread is discarded after handling the connection.

See thread-per-connection.c

Fixed-size pool of threads

This implementation spawns a fixed quantity of threads upon startup and uses them to handle all connections.

A stack is used to hand off the connection sockets from the main thread to the worker threads, protected by a pthread condition and mutex.

The main thread accepts connections and pushes the resulting file descriptors onto a stack, then signals the worker threads to wake up, consume the connections from the stack and reply to the incoming HTTP requests.

(Note: in retrospect, a stack is perhaps not a good choice, as the connections towards the bottom of the stack could suffer from starvation / excess latency).

See fixed-thread-pool.c

Benchmarks

The players:

• 127.0.0.1 (indium)
  • Mac Mini (M1, gigabit Ethernet)
  • macOS Monterey
• plasma
  • Macbook Pro (M1, gigabit Ethernet)
  • macOS Ventura
• flouride
  • 2012 Macbook Pro (i7-3820QM @2.7GHz, gigabit Ethernet)
  • macOS Ventura (thanks to OpenCore Legacy Patcher)
• opti7050
  • Dell Optiplex 7050 (i5-7500 @3.8GHz, gigabit Ethernet)
  • Ubuntu 22.10
• thinkpad
  • Thinkpad T500 (Core 2 Duo P8600 @2.4GHz, gigabit Ethernet)
  • Ubuntu 23.04
• pi2b-1
  • Raspberry Pi 2 Model B (ARMv7 @900MHz, 100Mbit Ethernet)
  • Raspbian Bullseye
• nslu2
  • Linksys NSLU2 (ARMv5 @266MHz, 100Mbit Ethernet)
  • Debian Jessie
• pmacg5
  • PowerMac G5 (PowerPC G5 @2.3GHz x2, gigabit Ethernet)
  • OS X Leopard
• emac3
  • eMac (PowerPC G4 @1.25GHz, 100Mbit Ethernet)
  • OS X Leopard
• graphite
  • PowerMac G4 (PowerPC G4 @500MHz x2, gigabit Ethernet)
  • OS X Tiger
• pmacg3
  • PowerMac G3 (PowerPC G3 @400MHz, 100Mbit Ethernet)
  • OS X Tiger

Max bandwidth (iperf)

Before we test the HTTP server implementations, first let's guage the network bandwidth capabilities of these hosts using iperf.

Typical output from iperf:

cell@indium(master)$ iperf -c plasma
------------------------------------------------------------
Client connecting to plasma, TCP port 5001
TCP window size:  129 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.98 port 63271 connected with 192.168.1.176 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.00-10.04 sec  1.09 GBytes   935 Mbits/sec

See the full output: iperf.txt

Summary of results for all hosts:

• 127.0.0.1: 68.6 Gbits/sec 🤯
• plasma: 935 Mbits/sec
• flouride: 937 Mbits/sec
• opti7050: 939 Mbits/sec
• thinkpad: 939 Mbits/sec
• pi2b-1: 94.0 Mbits/sec
• nslu2: 72.9 Mbits/sec ⚠️
• pmacg5: 939 Mbits/sec
• emac3: 94.0 Mbits/sec
• graphite: 496 Mbits/sec ⚠️
• pmacg3: 93.7 Mbits/sec

Of note are:

• the crazy-high loopback bandwidth to localhost!
• despite having gigabit, the dual G4 500MHz (graphite) can't quite saturate it
• the NSLU2 can't quite saturate 100Mbit with its 266MHz ARM processor

Requests per second (wrk)

I used wrk to load-test these HTTP servers.

Typical wrk output:

cell@indium(master)$ wrk -t8 -c32 -d5s http://flouride:8080
Running 5s test @ http://flouride:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   542.74us   18.81us   1.41ms   89.78%
    Req/Sec     1.84k    27.67     1.99k    90.20%
  9316 requests in 5.10s, 700.52KB read
Requests/sec:   1826.70
Transfer/sec:    137.36KB

I tested each host using -t8 -c32 (8 threads pumping 32 connections) as well as -t8 -c64 (8 threads pumping 64 connections).

• results-single-threaded.txt
• results-thread-per-conn.txt
• results-thread-pool.txt

First, let's look at the "fast" machines:

Across the board, we see that the threaded implementations far surpass the performance of the single-threaded implementation. No surprise there.

We also see that testing over loopback to the same host (127.0.0.1) is by far the winner. No surprise there either.

What was a bit surprising was that we see virtually no performance difference at all between the thread-per-connection and thread-pool implementations. I would guess this is due to my naivete with pthread conditions / mutexes.

The results for plasma (my work laptop) appear to be anomalous. I have no idea why its network performance is so poor. iperf proved it could saturate gigabit Ethernet, so I'm not sure what the issue is.

For the three x86_64 machines (flouride, opti7050, thinkpad), we see that bumping wrk from 32 to 64 connections increases performance by anywhere from 28% to 82%.

Now let's take a look at the "slower" machines:

Interestingly, here we see a slight to significant decrease in performance when jumping from 32 to 64 connections, especially with the G5 (pmacg5).

Coincidentally, the performance of an (older) Raspberry Pi (pi2b-1) very closely matches that of the two G4 machines (emac3, graphite).

Also interesting to note that a dual-processor 500MHz G4 (graphite) matches the performance of a single-processor 1.25GHz G4 (emac3), and that the dual G4 500MHz (graphite) is more than twice as fast as the G3 400MHz (pmacg3).

Surprisingly, threading does not seem to help at all on the 266MHz NSLU2.

fixed-thread-pool.c

// A thread-pool HTTP/1.1 'Hello, world!' server.

// Configuration defaults.  Override via ENV vars.
int PORT = 8080;
int BACKLOG = 64;
int NUM_THREADS = 64;

// Thanks to:
// https://www2.cs.uh.edu/~gnawali/courses/cosc4377-s12/hw2/http.html
// https://medium.com/from-the-scratch/http-server-what-do-you-need-to-know-to-build-a-simple-http-server-from-scratch-d1ef8945e4fa
// https://www.youtube.com/watch?v=Pg_4Jz8ZIH4
// https://www.youtube.com/watch?v=FMNnusHqjpw

// Code formatting note: the prefix 'g_' indicates a global variable.

// unistd.h needs _DEFAULT_SOURCE for usleep().
#define _DEFAULT_SOURCE

#include <stdlib.h>
#include <stdio.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <string.h> // memset
#include <stdbool.h>
#include <unistd.h>
#include <pthread.h>
#include <assert.h>
#include <fcntl.h>
#include <sys/errno.h>

// Thread synchronization.
pthread_cond_t g_condition;
pthread_mutex_t g_mutex;


// An array-based stack to store accepted socket connection file descriptors.
int* g_conn_fds_stack;
// Top is the first free slot on the stack.
int g_conn_fds_stack_top = 0;

bool can_pop_conn_fd() {
    return g_conn_fds_stack_top > 0;
}

int pop_conn_fd() {
    assert(g_conn_fds_stack_top > 0);
    g_conn_fds_stack_top -= 1;
    int conn_fd = g_conn_fds_stack[g_conn_fds_stack_top];
    g_conn_fds_stack[g_conn_fds_stack_top] = 0;
    return conn_fd;
}

bool can_push_conn_fd() {
    return g_conn_fds_stack_top < NUM_THREADS;
}

void push_conn_fd(int conn_fd) {
    assert(g_conn_fds_stack_top < NUM_THREADS);
    assert(g_conn_fds_stack[g_conn_fds_stack_top] == 0);
    g_conn_fds_stack[g_conn_fds_stack_top] = conn_fd;
    g_conn_fds_stack_top += 1;
}


// Thanks to https://stackoverflow.com/a/71064517
char* write_buff = "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: 12\r\n\r\nHello World!";

void handle_connection(int conn_fd) {
    char read_buff[65536] = {0};

    // HTTP 1.1 clients can send multiple requests over the same connection.
    // Loop over the input handling all requests.
    // Note: the 'conn_done' flag is used to break from the inner-while loop.
    bool conn_done = false;
    while (!conn_done) {
        ssize_t nbytes_read;
        nbytes_read = read(conn_fd, read_buff, sizeof(read_buff));
        if (nbytes_read == -1) {
            perror("read");
            break;
        } else if (nbytes_read == 0) {
            // The client has closed the connection.
            break;
        }

        int i = 0;
        while (i < nbytes_read) {
            // Find individual requests by the sequence '\r\n\r\n' (which
            // separates the headers from the body).
            if (read_buff[i] != '\r') {
                i++;
                continue;
            }
            i++;
            if (!(i < nbytes_read && read_buff[i] == '\n')) {
                continue;
            }
            i++;
            if (!(i < nbytes_read && read_buff[i] == '\r')) {
                continue;
            }
            i++;
            if (!(i < nbytes_read && read_buff[i] == '\n')) {
                continue;
            }
            i++;

            // Found a request, send a response.
            int flags = 0;
            // Use MSG_NOSIGNAL (if available) to avoid getting killed by SIGPIPE.
            #ifdef MSG_NOSIGNAL
            flags |= MSG_NOSIGNAL;
            #endif
            int nbytes_sent = send(conn_fd, write_buff, strlen(write_buff), flags);
            if (nbytes_sent == -1) {
                perror("write");
                conn_done = true;
                break;
            }
        }
    }

    int ret = close(conn_fd);
    if (ret != 0) {
        perror("close");
    }
}


void* worker_thread(void* arg) {
    int ret;
    while (true) {
        ret = pthread_mutex_lock(&g_mutex);
        if (ret != 0) {
            perror("pthread_mutex_lock");
            continue;
        }
        ret = pthread_cond_wait(&g_condition, &g_mutex);
        if (ret != 0) {
            perror("pthread_cond_wait");
            continue;
        }

        int conn_fd = 0;
        if (can_pop_conn_fd()) {
            conn_fd = pop_conn_fd();
        }

        ret = pthread_mutex_unlock(&g_mutex);
        if (ret != 0) {
            perror("pthread_mutex_unlock");
            continue;
        }

        if (conn_fd != 0) {
            handle_connection(conn_fd);
        }
    }
    return NULL;
}


void init_conn_fd_stack() {
    // Create the conn_fd stack.
    size_t size = NUM_THREADS * sizeof(int);
    g_conn_fds_stack = (int*)malloc(size);
    if (g_conn_fds_stack == NULL) {
        perror("malloc");
        exit(1);
    }
    memset(g_conn_fds_stack, 0, size);
}


void init_condition() {
    int ret;
    // Setup the condition.
    pthread_mutexattr_t* mattrs = NULL;
    ret = pthread_mutex_init(&g_mutex, mattrs);
    if (ret != 0) {
        perror("listen");
        exit(1);
    }
    pthread_condattr_t* cattrs = NULL;
    ret = pthread_cond_init(&g_condition, cattrs);
    if (ret != 0) {
        perror("listen");
        exit(1);
    }
}


void init_threads() {
    int ret;
    // Populate the thread pool.
    for (int i = 0; i < NUM_THREADS; i++) {
        pthread_t thread;
        pthread_attr_t* attributes = NULL;
        void* arg = NULL;
        ret = pthread_create(&thread, attributes, worker_thread, arg);
        if (ret != 0) {
            perror("pthread_create");
            exit(1);
        }
    }
}


int init_socket(struct sockaddr_in* addr, socklen_t addr_len) {
    int ret;

    // Create a TCP socket.
    int domain = AF_INET;
    int type = SOCK_STREAM;
    int protocol = 0;
    int sock_fd = socket(domain, type, protocol);
    if (sock_fd == -1) {
        perror("socket");
        exit(1);
    }

    // Set the socket to be non-blocking.
    ret = fcntl(sock_fd, F_GETFL, 0);
    if (ret == -1) {
        perror("fcntl F_GETFL");
        exit(1);
    }
    int flags = ret | O_NONBLOCK;
    ret = fcntl(sock_fd, F_SETFL, flags);
    if (ret == -1) {
        perror("fcntl F_SETFL");
        exit(1);
    }

    // Avoid "bind: address already in use" when restarting the server.
    int level = SOL_SOCKET;
    int option_name = SO_REUSEADDR;
    int option = 1;
    ret = setsockopt(sock_fd, level, option_name, &option, sizeof(option));
    if (ret != 0) {
        perror("setsockopt");
        exit(1);
    }

    // Use SO_NOSIGPIPE (if available) to avoid getting killed by SIGPIPE.
    // See https://web.archive.org/web/20070502105136/http://lists.apple.com/archives/macnetworkprog/2002/Dec/msg00091.html
    #ifdef SO_NOSIGPIPE
    level = SOL_SOCKET;
    option_name = SO_NOSIGPIPE;
    option = 1;
    ret = setsockopt(sock_fd, level, option_name, &option, sizeof(option));
    if (ret != 0) {
        perror("setsockopt");
        exit(1);
    }
    #endif

    // Bind the socket.
    ret = bind(sock_fd, (struct sockaddr*)addr, addr_len);
    if (ret != 0) {
        perror("bind");
        exit(1);
    }

    // Listen on the socket.
    ret = listen(sock_fd, BACKLOG);
    if (ret != 0) {
        perror("listen");
        exit(1);
    }
    printf("Listening on port %d.\n", PORT);
    printf("Test with e.g. 'wrk -t8 -c32 -d5s http://127.0.0.1:%d'.\n", PORT);

    return sock_fd;
}


int get_int_from_env(char* name) {
    char* str_value = getenv(name);
    if (str_value == NULL) {
        return -1;
    }
    char** endptr = NULL;
    int base = 10;
    long long_value = strtol(str_value, endptr, base);
    if (long_value == 0 && errno != 0) {
        perror("strtol");
        return -1;
    }
    return (int)long_value;
}


void init_from_env() {
    int value;
    // Read configuration values from the environment.
    value = get_int_from_env("PORT");
    if (value != -1) {
        PORT = value;
    }
    value = get_int_from_env("BACKLOG");
    if (value != -1) {
        BACKLOG = value;
    }
    value = get_int_from_env("NUM_THREADS");
    if (value != -1) {
        NUM_THREADS = value;
    }
    printf("Configuration: PORT=%d, BACKLOG=%d, NUM_THREADS=%d.\n", PORT, BACKLOG, NUM_THREADS);
    printf("Override these values using ENV vars.\n");
}


int main(int argc, char** argv) {
    int ret;

    init_from_env();

    // Socket address configuration.
    struct sockaddr_in addr;
    memset(&addr, 0, sizeof(addr));
    addr.sin_family = AF_INET;
    addr.sin_addr.s_addr = htonl(INADDR_ANY);
    addr.sin_port = htons(PORT);
    socklen_t addr_len = sizeof(addr);

    init_conn_fd_stack();
    int sock_fd = init_socket(&addr, addr_len);
    init_condition();
    init_threads();

    while (true) {
        ret = pthread_mutex_lock(&g_mutex);
        if (ret != 0) {
            perror("pthread_mutex_lock");
            exit(1);
        }

        // Wake up a worker if there are accepted connections to handle.
        if (can_pop_conn_fd()) {
            ret = pthread_cond_signal(&g_condition);
            if (ret != 0) {
                perror("pthread_cond_signal");
                exit(1);
            }
            ret = pthread_mutex_unlock(&g_mutex);
            if (ret != 0) {
                perror("pthread_mutex_unlock");
                exit(1);
            }
            continue;
        }

        // Accept as many waiting connections as we can.
        bool continue_outer = false;
        while (can_push_conn_fd()) {
            // Accept a new connection.
            int conn_fd;
            conn_fd = accept(sock_fd, (struct sockaddr*)&addr, &addr_len);
            if (conn_fd == -1) {
                if (errno == EWOULDBLOCK) {
                    continue_outer = false;
                    break;
                } else {
                    perror("accept");
                    continue_outer = true;
                    break;
                }
            }

            // Set the accepted socket to be blocking.
            ret = fcntl(conn_fd, F_GETFL, 0);
            if (ret == -1) {
                perror("fcntl F_GETFL");
                exit(1);
            }
            int flags = ret & ~O_NONBLOCK;
            ret = fcntl(conn_fd, F_SETFL, flags);
            if (ret == -1) {
                perror("fcntl F_SETFL");
                exit(1);
            }

            push_conn_fd(conn_fd);
            continue_outer = true;
        }
        ret = pthread_mutex_unlock(&g_mutex);
        if (ret != 0) {
            perror("pthread_mutex_unlock");
            exit(1);
        }
        if (continue_outer) {
            continue;
        }

        // No work to do, no connections to accept, so sleep for a bit.
        useconds_t one_ms = 1000;
        usleep(one_ms);
    }

    return 0;
}

iperf.txt

cell@indium(master)$ [1] iperf -c 127.0.0.1
------------------------------------------------------------
Client connecting to 127.0.0.1, TCP port 5001
TCP window size:  144 KByte (default)
------------------------------------------------------------
[  1] local 127.0.0.1 port 63178 connected with 127.0.0.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.00-10.00 sec  79.8 GBytes  68.6 Gbits/sec
cell@indium(master)$ iperf -c plasma
------------------------------------------------------------
Client connecting to plasma, TCP port 5001
TCP window size:  129 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.98 port 63271 connected with 192.168.1.176 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.00-10.04 sec  1.09 GBytes   935 Mbits/sec
cell@indium(master)$ iperf -c flouride
------------------------------------------------------------
Client connecting to flouride, TCP port 5001
TCP window size:  129 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.98 port 63342 connected with 192.168.1.100 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.00-10.06 sec  1.10 GBytes   937 Mbits/sec
cell@indium(master)$ iperf -c opti7050
------------------------------------------------------------
Client connecting to opti7050, TCP port 5001
TCP window size:  129 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.98 port 63461 connected with 192.168.1.58 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.00-10.04 sec  1.10 GBytes   939 Mbits/sec
cell@indium(master)$ iperf -c thinkpad
------------------------------------------------------------
Client connecting to thinkpad, TCP port 5001
TCP window size:  129 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.98 port 63530 connected with 192.168.1.172 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.00-10.04 sec  1.10 GBytes   939 Mbits/sec
cell@indium(master)$ iperf -c pi2b-1
------------------------------------------------------------
Client connecting to pi2b-1, TCP port 5001
TCP window size:  129 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.98 port 63639 connected with 192.168.1.152 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.00-10.18 sec   114 MBytes  94.0 Mbits/sec
cell@indium(master)$ iperf -c nslu2
------------------------------------------------------------
Client connecting to nslu2, TCP port 5001
TCP window size:  129 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.98 port 63934 connected with 192.168.1.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.00-10.02 sec  87.1 MBytes  72.9 Mbits/sec
cell@indium(master)$ iperf -c ibookg3
------------------------------------------------------------
Client connecting to ibookg3, TCP port 5001
TCP window size:  129 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.98 port 64025 connected with 192.168.1.81 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.00-10.03 sec   112 MBytes  93.7 Mbits/sec
cell@indium(master)$ iperf -c pmacg5
------------------------------------------------------------
Client connecting to pmacg5, TCP port 5001
TCP window size:  129 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.98 port 64095 connected with 192.168.1.90 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.00-10.01 sec  1.09 GBytes   939 Mbits/sec
cell@indium(master)$ iperf -c emac3
------------------------------------------------------------
Client connecting to emac3, TCP port 5001
TCP window size:  129 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.98 port 64153 connected with 192.168.1.83 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.00-10.07 sec   113 MBytes  94.0 Mbits/sec
cell@indium(master)$ iperf -c graphite
------------------------------------------------------------
Client connecting to graphite, TCP port 5001
TCP window size:  129 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.98 port 64224 connected with 192.168.1.86 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.00-10.01 sec   592 MBytes   496 Mbits/sec
cell@indium(master)$ iperf -c pmacg3
------------------------------------------------------------
Client connecting to pmacg3, TCP port 5001
TCP window size:  129 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.98 port 64285 connected with 192.168.1.57 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.00-10.03 sec   112 MBytes  93.7 Mbits/sec
cell@indium(master)$ 

Makefile

serve: serve.c
	gcc -std=c99 -Wall -Werror -O3 -o serve serve.c

clean:
	rm -f serve
.PHONY: clean

results-single-threaded.txt

cell@indium(master)$ for host in 127.0.0.1 plasma flouride opti7050 thinkpad pi2b-1 nslu2 pmacg5 emac3 graphite pmacg3 ; do sleep 1 ; echo ; echo $host ; wrk -t8 -c32 -d5s http://$host:8080 ; done

127.0.0.1
Running 5s test @ http://127.0.0.1:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    18.38us    4.13us   1.03ms   94.18%
    Req/Sec    52.27k     2.30k   53.24k    96.08%
  265199 requests in 5.10s, 19.47MB read
Requests/sec:  52009.93
Transfer/sec:      3.82MB

plasma
Running 5s test @ http://plasma:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   537.16us  327.78us  13.98ms   93.67%
    Req/Sec     1.85k   125.63     2.17k    70.59%
  9407 requests in 5.10s, 707.36KB read
Requests/sec:   1844.86
Transfer/sec:    138.73KB

flouride
Running 5s test @ http://flouride:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   542.74us   18.81us   1.41ms   89.78%
    Req/Sec     1.84k    27.67     1.99k    90.20%
  9316 requests in 5.10s, 700.52KB read
Requests/sec:   1826.70
Transfer/sec:    137.36KB

opti7050
Running 5s test @ http://opti7050:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   400.78us   62.54us   5.26ms   94.06%
    Req/Sec     2.48k   177.55     2.83k    74.51%
  12575 requests in 5.10s, 0.92MB read
Requests/sec:   2466.09
Transfer/sec:    185.44KB

thinkpad
Running 5s test @ http://thinkpad:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   327.13us  115.72us  10.05ms   98.27%
    Req/Sec     3.03k   109.70     3.26k    72.55%
  15389 requests in 5.10s, 1.13MB read
Requests/sec:   3018.63
Transfer/sec:    226.99KB

pi2b-1
Running 5s test @ http://pi2b-1:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   670.61us  142.33us  10.41ms   97.79%
    Req/Sec     1.49k    48.63     1.55k    76.47%
  7544 requests in 5.10s, 567.27KB read
Requests/sec:   1479.30
Transfer/sec:    111.24KB

nslu2
Running 5s test @ http://nslu2:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   589.13us  158.36us  14.98ms   99.63%
    Req/Sec     1.69k    38.32     1.73k    96.08%
  8592 requests in 5.10s, 646.08KB read
Requests/sec:   1685.12
Transfer/sec:    126.71KB

pmacg5
Running 5s test @ http://pmacg5:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   389.92us   34.30us   1.67ms   91.26%
    Req/Sec     2.55k    62.36     2.64k    60.78%
  12918 requests in 5.10s, 0.95MB read
Requests/sec:   2533.52
Transfer/sec:    190.51KB

emac3
Running 5s test @ http://emac3:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   353.97us   34.60us   2.39ms   90.65%
    Req/Sec     2.80k    84.91     2.95k    66.67%
  14210 requests in 5.10s, 1.04MB read
Requests/sec:   2786.26
Transfer/sec:    209.51KB

graphite
Running 5s test @ http://graphite:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   460.61us   70.22us   7.66ms   99.80%
    Req/Sec     2.16k    27.08     2.23k    88.24%
  10959 requests in 5.10s, 824.07KB read
Requests/sec:   2149.10
Transfer/sec:    161.60KB

pmacg3
Running 5s test @ http://pmacg3:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   509.74us  340.72us  30.79ms   99.80%
    Req/Sec     1.95k   122.67     2.03k    96.08%
  9902 requests in 5.10s, 744.58KB read
Requests/sec:   1941.92
Transfer/sec:    146.02KB



cell@indium(master)$ for host in 127.0.0.1 plasma flouride opti7050 thinkpad pi2b-1 nslu2 pmacg5 emac3 graphite pmacg3 ; do sleep 1 ; echo ; echo $host ; wrk -t8 -c64 -d5s http://$host:8080 ; done

127.0.0.1
Running 5s test @ http://127.0.0.1:8080
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    18.47us    7.20us   2.20ms   98.36%
    Req/Sec    52.02k     2.31k   58.26k    96.08%
  263916 requests in 5.10s, 19.38MB read
Requests/sec:  51754.69
Transfer/sec:      3.80MB

plasma
Running 5s test @ http://plasma:8080
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   566.59us  347.03us   9.09ms   93.08%
    Req/Sec     1.76k   110.06     2.08k    76.47%
  8925 requests in 5.10s, 671.12KB read
Requests/sec:   1749.85
Transfer/sec:    131.58KB

flouride
Running 5s test @ http://flouride:8080
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   540.91us   35.04us   3.06ms   89.18%
    Req/Sec     1.84k    27.55     1.92k    86.27%
  9347 requests in 5.10s, 702.85KB read
Requests/sec:   1833.02
Transfer/sec:    137.83KB

opti7050
Running 5s test @ http://opti7050:8080
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   376.13us   34.80us 828.00us   64.33%
    Req/Sec     2.64k   197.91     2.85k    58.82%
  13388 requests in 5.10s, 0.98MB read
Requests/sec:   2625.69
Transfer/sec:    197.44KB

thinkpad
Running 5s test @ http://thinkpad:8080
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   319.52us   31.66us   1.33ms   83.08%
    Req/Sec     3.10k    66.86     3.38k    76.47%
  15731 requests in 5.10s, 1.16MB read
Requests/sec:   3085.19
Transfer/sec:    231.99KB

pi2b-1
Running 5s test @ http://pi2b-1:8080
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   654.84us  132.92us  10.04ms   99.09%
    Req/Sec     1.52k    48.64     1.56k    96.08%
  7724 requests in 5.10s, 580.81KB read
Requests/sec:   1514.90
Transfer/sec:    113.91KB

nslu2
Running 5s test @ http://nslu2:8080
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   597.93us  233.85us  21.70ms   99.83%
    Req/Sec     1.67k    57.94     1.74k    86.27%
  8461 requests in 5.10s, 636.23KB read
Requests/sec:   1659.34
Transfer/sec:    124.77KB

pmacg5
Running 5s test @ http://pmacg5:8080
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   393.32us   38.34us   3.66ms   94.74%
    Req/Sec     2.53k    48.55     2.62k    60.78%
  12815 requests in 5.10s, 0.94MB read
Requests/sec:   2513.27
Transfer/sec:    188.99KB

emac3
Running 5s test @ http://emac3:8080
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   362.12us   88.05us   9.48ms   99.50%
    Req/Sec     2.74k    48.77     2.88k    90.20%
  13898 requests in 5.10s, 1.02MB read
Requests/sec:   2725.35
Transfer/sec:    204.93KB

graphite
Running 5s test @ http://graphite:8080
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   459.68us   67.26us   7.36ms   99.93%
    Req/Sec     2.16k    34.95     2.25k    84.31%
  10981 requests in 5.10s, 825.72KB read
Requests/sec:   2153.42
Transfer/sec:    161.93KB

pmacg3
Running 5s test @ http://pmacg3:8080
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   532.17us    1.35ms 102.90ms   99.80%
    Req/Sec     1.91k   267.33     2.08k    94.12%
  9682 requests in 5.10s, 728.04KB read
Requests/sec:   1898.13
Transfer/sec:    142.73KB

results-thread-per-conn.txt

cell@indium(master)$ for host in 127.0.0.1 plasma flouride opti7050 thinkpad pi2b-1 nslu2 pmacg5 emac3 graphite pmacg3 ; do sleep 1 ; echo ; echo $host ; wrk -t8 -c32 -d5s http://$host:8080 ; done

127.0.0.1
Running 5s test @ http://127.0.0.1:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     9.30ms   24.34ms 250.03ms   89.82%
    Req/Sec    24.67k    17.70k   85.90k    71.39%
  941127 requests in 5.09s, 69.11MB read
Requests/sec: 184725.63
Transfer/sec:     13.56MB

plasma
Running 5s test @ http://plasma:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.32ms  637.23us  15.14ms   97.21%
    Req/Sec     3.15k   210.95     3.33k    94.12%
  127886 requests in 5.10s, 9.39MB read
Requests/sec:  25071.99
Transfer/sec:      1.84MB

flouride
Running 5s test @ http://flouride:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   670.35us  108.14us   4.00ms   73.32%
    Req/Sec     5.96k    85.18     6.23k    75.98%
  241952 requests in 5.10s, 17.77MB read
Requests/sec:  47435.18
Transfer/sec:      3.48MB

opti7050
Running 5s test @ http://opti7050:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   426.61us   71.50us   1.29ms   63.73%
    Req/Sec     9.30k   216.78     9.90k    67.65%
  377555 requests in 5.10s, 27.72MB read
Requests/sec:  74032.85
Transfer/sec:      5.44MB

thinkpad
Running 5s test @ http://thinkpad:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   450.18us  259.56us  11.40ms   96.52%
    Req/Sec     9.16k   427.19    10.51k    78.13%
  370880 requests in 5.10s, 27.23MB read
Requests/sec:  72730.18
Transfer/sec:      5.34MB

pi2b-1
Running 5s test @ http://pi2b-1:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.32ms  704.02us  31.37ms   96.82%
    Req/Sec     1.74k   136.29     2.37k    94.58%
  70382 requests in 5.10s, 5.17MB read
Requests/sec:  13794.90
Transfer/sec:      1.01MB

nslu2
Running 5s test @ http://nslu2:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    17.90ms   19.82ms 323.63ms   97.57%
    Req/Sec   258.06     33.96   404.00     94.94%
  10186 requests in 5.02s, 765.94KB read
Requests/sec:   2028.68
Transfer/sec:    152.55KB

pmacg5
Running 5s test @ http://pmacg5:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.57ms    7.25ms 125.54ms   98.66%
    Req/Sec     4.86k   494.09     6.49k    73.28%
  197194 requests in 5.10s, 14.48MB read
Requests/sec:  38666.82
Transfer/sec:      2.84MB

emac3
Running 5s test @ http://emac3:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    22.69ms   40.54ms 375.38ms   83.45%
    Req/Sec     1.81k   705.81     4.45k    71.53%
  73044 requests in 5.10s, 5.36MB read
Requests/sec:  14319.90
Transfer/sec:      1.05MB

graphite
Running 5s test @ http://graphite:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    23.37ms   50.49ms 375.96ms   87.34%
    Req/Sec     1.88k   811.71     5.57k    75.37%
  75527 requests in 5.10s, 5.55MB read
Requests/sec:  14800.76
Transfer/sec:      1.09MB

pmacg3
Running 5s test @ http://pmacg3:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    57.13ms  101.40ms 627.76ms   86.28%
    Req/Sec   608.49    411.17     1.91k    70.60%
  23794 requests in 5.02s, 1.75MB read
Requests/sec:   4742.97
Transfer/sec:    356.65KB
cell@indium(master)$ 



cell@indium(master)$ for host in 127.0.0.1 plasma flouride opti7050 thinkpad pi2b-1 nslu2 pmacg5 emac3 graphite pmacg3 ; do sleep 1 ; echo ; echo $host ; wrk -t8 -c64 -d5s http://$host:8080 ; done

127.0.0.1
Running 5s test @ http://127.0.0.1:8080
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     8.11ms   20.73ms 226.42ms   89.48%
    Req/Sec    25.68k    19.11k  117.84k    75.50%
  978240 requests in 5.08s, 71.84MB read
Requests/sec: 192732.88
Transfer/sec:     14.15MB

plasma
Running 5s test @ http://plasma:8080
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.55ms  797.19us  28.83ms   97.71%
    Req/Sec     3.18k   224.68     4.27k    94.58%
  128656 requests in 5.10s, 9.45MB read
Requests/sec:  25226.59
Transfer/sec:      1.85MB

flouride
Running 5s test @ http://flouride:8080
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   737.55us  168.19us   5.18ms   79.23%
    Req/Sec    10.85k   367.75    11.64k    75.49%
  440547 requests in 5.10s, 32.35MB read
Requests/sec:  86381.71
Transfer/sec:      6.34MB

opti7050
Running 5s test @ http://opti7050:8080
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   433.19us   75.05us   1.95ms   76.77%
    Req/Sec    18.32k   578.83    20.66k    95.83%
  743642 requests in 5.10s, 54.61MB read
Requests/sec: 145803.69
Transfer/sec:     10.71MB

thinkpad
Running 5s test @ http://thinkpad:8080
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.29ms    2.57ms 117.58ms   91.42%
    Req/Sec    11.73k     1.69k   16.41k    68.63%
  476069 requests in 5.10s, 34.96MB read
Requests/sec:  93345.22
Transfer/sec:      6.85MB

pi2b-1
Running 5s test @ http://pi2b-1:8080
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     5.18ms    2.42ms  64.24ms   98.46%
    Req/Sec     1.59k   158.24     1.68k    94.00%
  63298 requests in 5.00s, 4.65MB read
Requests/sec:  12649.96
Transfer/sec:      0.93MB

nslu2
Running 5s test @ http://nslu2:8080
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    63.95ms  149.34ms   1.40s    94.44%
    Req/Sec   248.32     50.99   560.00     86.90%
  9851 requests in 5.02s, 740.75KB read
Requests/sec:   1961.95
Transfer/sec:    147.53KB

pmacg5
Running 5s test @ http://pmacg5:8080
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    30.16ms   59.27ms 376.30ms   86.21%
    Req/Sec     3.68k     1.11k    7.20k    72.79%
  149470 requests in 5.10s, 10.98MB read
Requests/sec:  29305.00
Transfer/sec:      2.15MB

emac3
Running 5s test @ http://emac3:8080
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    36.91ms   79.10ms 750.55ms   89.58%
    Req/Sec     1.73k     1.01k   14.00k    83.54%
  68432 requests in 5.10s, 5.03MB read
Requests/sec:  13417.93
Transfer/sec:      0.99MB

graphite
Running 5s test @ http://graphite:8080
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    43.96ms   87.14ms 626.31ms   88.64%
    Req/Sec     1.69k     0.93k    5.45k    73.75%
  67723 requests in 5.10s, 4.97MB read
Requests/sec:  13267.62
Transfer/sec:      0.97MB

pmacg3
Running 5s test @ http://pmacg3:8080
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    78.34ms  130.16ms   1.00s    85.84%
    Req/Sec   619.68    412.10     2.34k    72.44%
  23872 requests in 5.02s, 1.75MB read
Requests/sec:   4759.56
Transfer/sec:    357.90KB

results-thread-pool.txt

cell@indium(master)$ for host in 127.0.0.1 plasma flouride opti7050 thinkpad pi2b-1 nslu2 pmacg5 emac3 graphite pmacg3 ; do sleep 1 ; echo ; echo $host ; wrk -t8 -c32 -d5s http://$host:8080 ; done

127.0.0.1
Running 5s test @ http://127.0.0.1:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     9.11ms   26.11ms 250.01ms   90.60%
    Req/Sec    26.72k    20.01k  106.15k    75.07%
  1013415 requests in 5.10s, 74.42MB read
Requests/sec: 198563.95
Transfer/sec:     14.58MB

plasma
Running 5s test @ http://plasma:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.29ms  671.72us  25.12ms   97.05%
    Req/Sec     3.21k   434.24     6.54k    97.03%
  128930 requests in 5.10s, 9.47MB read
Requests/sec:  25284.21
Transfer/sec:      1.86MB

flouride
Running 5s test @ http://flouride:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   672.19us  106.26us   5.01ms   73.94%
    Req/Sec     5.95k    67.52     6.13k    65.93%
  241588 requests in 5.10s, 17.74MB read
Requests/sec:  47363.61
Transfer/sec:      3.48MB

opti7050
Running 5s test @ http://opti7050:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   424.77us   74.68us   3.38ms   69.44%
    Req/Sec     9.35k   225.83    10.03k    73.77%
  379574 requests in 5.10s, 27.87MB read
Requests/sec:  74424.06
Transfer/sec:      5.47MB

thinkpad
Running 5s test @ http://thinkpad:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   435.92us  114.84us   3.86ms   73.00%
    Req/Sec     9.15k   169.51     9.54k    75.25%
  364103 requests in 5.00s, 26.74MB read
Requests/sec:  72817.02
Transfer/sec:      5.35MB

pi2b-1
Running 5s test @ http://pi2b-1:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.48ms    1.88ms  73.07ms   98.97%
    Req/Sec     1.69k    80.52     1.83k    90.75%
  67265 requests in 5.00s, 4.94MB read
Requests/sec:  13446.63
Transfer/sec:      0.99MB

nslu2
Running 5s test @ http://nslu2:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    16.73ms    7.99ms 171.22ms   97.37%
    Req/Sec   247.98     23.17   292.00     91.23%
  9887 requests in 5.02s, 743.46KB read
Requests/sec:   1970.73
Transfer/sec:    148.19KB

pmacg5
Running 5s test @ http://pmacg5:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.87ms    6.90ms 125.62ms   97.00%
    Req/Sec     4.95k     1.02k    7.62k    71.01%
  200465 requests in 5.10s, 14.72MB read
Requests/sec:  39304.30
Transfer/sec:      2.89MB

emac3
Running 5s test @ http://emac3:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    20.83ms   42.86ms 259.65ms   86.43%
    Req/Sec     1.71k   604.19     4.58k    76.66%
  69350 requests in 5.10s, 5.09MB read
Requests/sec:  13595.52
Transfer/sec:      1.00MB

graphite
Running 5s test @ http://graphite:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    26.15ms   51.57ms 375.73ms   85.69%
    Req/Sec     1.88k     0.87k    5.13k    72.26%
  75241 requests in 5.10s, 5.53MB read
Requests/sec:  14744.37
Transfer/sec:      1.08MB

pmacg3
Running 5s test @ http://pmacg3:8080
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    74.65ms  126.98ms 750.96ms   85.11%
    Req/Sec   621.63    512.25     2.78k    73.83%
  23608 requests in 5.02s, 1.73MB read
Requests/sec:   4701.14
Transfer/sec:    353.50KB



cell@indium(master)$ for host in 127.0.0.1 plasma flouride opti7050 thinkpad pi2b-1 nslu2 pmacg5 emac3 graphite pmacg3 ; do sleep 1 ; echo ; echo $host ; wrk -t8 -c64 -d5s http://$host:8080 ; done

127.0.0.1
Running 5s test @ http://127.0.0.1:8080
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     7.81ms   20.79ms 166.88ms   90.00%
    Req/Sec    25.45k    18.05k   97.29k    69.60%
  973958 requests in 5.10s, 71.52MB read
Requests/sec: 191146.25
Transfer/sec:     14.04MB

plasma
Running 5s test @ http://plasma:8080
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.54ms  477.46us  13.55ms   96.78%
    Req/Sec     3.18k   211.38     3.32k    94.12%
  128937 requests in 5.10s, 9.47MB read
Requests/sec:  25275.93
Transfer/sec:      1.86MB

flouride
Running 5s test @ http://flouride:8080
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   740.81us  173.98us   6.75ms   79.79%
    Req/Sec    10.81k   379.62    11.44k    72.79%
  438741 requests in 5.10s, 32.22MB read
Requests/sec:  86019.96
Transfer/sec:      6.32MB

opti7050
Running 5s test @ http://opti7050:8080
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   432.69us   78.09us   3.94ms   78.87%
    Req/Sec    18.35k   524.22    20.47k    95.59%
  744845 requests in 5.10s, 54.70MB read
Requests/sec: 146040.94
Transfer/sec:     10.72MB

thinkpad
Running 5s test @ http://thinkpad:8080
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.39ms    2.33ms  31.43ms   89.24%
    Req/Sec    11.98k     2.40k   17.81k    65.93%
  486590 requests in 5.10s, 35.73MB read
Requests/sec:  95403.78
Transfer/sec:      7.01MB

pi2b-1
Running 5s test @ http://pi2b-1:8080
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     5.25ms    1.51ms  83.19ms   98.09%
    Req/Sec     1.55k   109.16     1.64k    95.83%
  62846 requests in 5.10s, 4.61MB read
Requests/sec:  12312.43
Transfer/sec:      0.90MB

nslu2
Running 5s test @ http://nslu2:8080
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    35.68ms   31.32ms 458.64ms   94.83%
    Req/Sec   250.73     32.57   474.00     93.67%
  9946 requests in 5.02s, 747.89KB read
Requests/sec:   1980.85
Transfer/sec:    148.95KB

pmacg5
Running 5s test @ http://pmacg5:8080
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    38.24ms   68.89ms 376.75ms   85.08%
    Req/Sec     3.69k     1.88k   12.19k    70.82%
  147540 requests in 5.10s, 10.83MB read
Requests/sec:  28921.90
Transfer/sec:      2.12MB

emac3
Running 5s test @ http://emac3:8080
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    44.79ms   93.99ms 625.41ms   88.72%
    Req/Sec     1.81k     1.38k   13.15k    80.80%
  68648 requests in 5.10s, 5.04MB read
  Socket errors: connect 0, read 69, write 0, timeout 0
Requests/sec:  13459.17
Transfer/sec:      0.99MB

graphite
Running 5s test @ http://graphite:8080
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    57.46ms  100.63ms 626.31ms   85.25%
    Req/Sec     1.67k     0.88k    5.23k    73.86%
  66073 requests in 5.01s, 4.85MB read
Requests/sec:  13196.17
Transfer/sec:      0.97MB

pmacg3
Running 5s test @ http://pmacg3:8080
  8 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   101.00ms  174.56ms   1.38s    86.94%
    Req/Sec   632.42    463.64     3.72k    74.66%
  23810 requests in 5.02s, 1.75MB read
Requests/sec:   4742.04
Transfer/sec:    356.58KB

single-threaded.c

// A single-threaded HTTP/1.1 'Hello, world!' server.

// Configuration defaults.  Override via ENV vars.
int PORT = 8080;
int BACKLOG = 64;

// Thanks to:
// https://www2.cs.uh.edu/~gnawali/courses/cosc4377-s12/hw2/http.html
// https://medium.com/from-the-scratch/http-server-what-do-you-need-to-know-to-build-a-simple-http-server-from-scratch-d1ef8945e4fa
// https://www.youtube.com/watch?v=Pg_4Jz8ZIH4

// Code formatting note: the prefix 'g_' indicates a global variable.

#include <stdlib.h>
#include <stdio.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <string.h> // memset
#include <stdbool.h>
#include <unistd.h>
#include <sys/errno.h>


// Thanks to https://stackoverflow.com/a/71064517
char* write_buff = "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: 12\r\n\r\nHello World!";

void handle_connection(int conn_fd) {
    char read_buff[65536] = {0};

    // HTTP 1.1 clients can send multiple requests over the same connection.
    // Loop over the input handling all requests.
    // Note: the 'conn_done' flag is used to break from the inner-while loop.
    bool conn_done = false;
    while (!conn_done) {
        ssize_t nbytes_read;
        nbytes_read = read(conn_fd, read_buff, sizeof(read_buff));
        if (nbytes_read == -1) {
            perror("read");
            break;
        } else if (nbytes_read == 0) {
            // The client has closed the connection.
            break;
        }

        int i = 0;
        while (i < nbytes_read) {
            // Find individual requests by the sequence '\r\n\r\n' (which
            // separates the headers from the body).
            if (read_buff[i] != '\r') {
                i++;
                continue;
            }
            i++;
            if (!(i < nbytes_read && read_buff[i] == '\n')) {
                continue;
            }
            i++;
            if (!(i < nbytes_read && read_buff[i] == '\r')) {
                continue;
            }
            i++;
            if (!(i < nbytes_read && read_buff[i] == '\n')) {
                continue;
            }
            i++;

            // Found a request, send a response.
            int flags = 0;
            // Use MSG_NOSIGNAL (if available) to avoid getting killed by SIGPIPE.
            #ifdef MSG_NOSIGNAL
            flags |= MSG_NOSIGNAL;
            #endif
            int nbytes_sent = send(conn_fd, write_buff, strlen(write_buff), flags);
            if (nbytes_sent == -1) {
                perror("write");
                conn_done = true;
                break;
            }
        }
    }

    int ret = close(conn_fd);
    if (ret != 0) {
        perror("close");
    }
}


int init_socket(struct sockaddr_in* addr, socklen_t addr_len) {
    int ret;

    // Create a TCP socket.
    int domain = AF_INET;
    int type = SOCK_STREAM;
    int protocol = 0;
    int sock_fd = socket(domain, type, protocol);
    if (sock_fd == -1) {
        perror("socket");
        exit(1);
    }

    // Avoid "bind: address already in use" when restarting the server.
    int level = SOL_SOCKET;
    int option_name = SO_REUSEADDR;
    int option = 1;
    ret = setsockopt(sock_fd, level, option_name, &option, sizeof(option));
    if (ret != 0) {
        perror("setsockopt");
        exit(1);
    }

    // Use SO_NOSIGPIPE (if available) to avoid getting killed by SIGPIPE.
    // See https://web.archive.org/web/20070502105136/http://lists.apple.com/archives/macnetworkprog/2002/Dec/msg00091.html
    #ifdef SO_NOSIGPIPE
    level = SOL_SOCKET;
    option_name = SO_NOSIGPIPE;
    option = 1;
    ret = setsockopt(sock_fd, level, option_name, &option, sizeof(option));
    if (ret != 0) {
        perror("setsockopt");
        exit(1);
    }
    #endif

    // Bind the socket.
    ret = bind(sock_fd, (struct sockaddr*)addr, addr_len);
    if (ret != 0) {
        perror("bind");
        exit(1);
    }

    // Listen on the socket.
    ret = listen(sock_fd, BACKLOG);
    if (ret != 0) {
        perror("listen");
        exit(1);
    }
    printf("Listening on port %d.\n", PORT);
    printf("Test with e.g. 'wrk -t8 -c32 -d5s http://127.0.0.1:%d'.\n", PORT);

    return sock_fd;
}


int get_int_from_env(char* name) {
    char* str_value = getenv(name);
    if (str_value == NULL) {
        return -1;
    }
    char** endptr = NULL;
    int base = 10;
    long long_value = strtol(str_value, endptr, base);
    if (long_value == 0 && errno != 0) {
        perror("strtol");
        return -1;
    }
    return (int)long_value;
}


void init_from_env() {
    int value;
    // Read configuration values from the environment.
    value = get_int_from_env("PORT");
    if (value != -1) {
        PORT = value;
    }
    value = get_int_from_env("BACKLOG");
    if (value != -1) {
        BACKLOG = value;
    }
    printf("Configuration: PORT=%d, BACKLOG=%d.\n", PORT, BACKLOG);
    printf("Override these values using ENV vars.\n");
}


int main(int argc, char** argv) {

    init_from_env();

    // Socket address configuration.
    struct sockaddr_in addr;
    memset(&addr, 0, sizeof(addr));
    addr.sin_family = AF_INET;
    addr.sin_addr.s_addr = htonl(INADDR_ANY);
    addr.sin_port = htons(PORT);
    socklen_t addr_len = sizeof(addr);

    int sock_fd = init_socket(&addr, addr_len);

    while (true) {
        // Accept a new connection.
        int conn_fd;
        conn_fd = accept(sock_fd, (struct sockaddr*)&addr, &addr_len);
        if (conn_fd == -1) {
            perror("accept");
            continue;
        }

        handle_connection(conn_fd);
    }

    return 0;
}

thread-per-connection.c

// A threaded HTTP/1.1 'Hello, world!' server.

// Configuration defaults.  Override via ENV vars.
int PORT = 8080;
int BACKLOG = 64;

// Thanks to:
// https://www2.cs.uh.edu/~gnawali/courses/cosc4377-s12/hw2/http.html
// https://medium.com/from-the-scratch/http-server-what-do-you-need-to-know-to-build-a-simple-http-server-from-scratch-d1ef8945e4fa
// https://www.youtube.com/watch?v=Pg_4Jz8ZIH4
// https://www.youtube.com/watch?v=FMNnusHqjpw

// Code formatting note: the prefix 'g_' indicates a global variable.

#include <stdlib.h>
#include <stdio.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <string.h> // memset
#include <stdbool.h>
#include <unistd.h>
#include <pthread.h>
#include <sys/errno.h>


// Thanks to https://stackoverflow.com/a/71064517
char* write_buff = "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nContent-Length: 12\r\n\r\nHello World!";

void* handle_connection(void* arg) {
    char read_buff[65536] = {0};

    int conn_fd = (int)(long)arg;

    // HTTP 1.1 clients can send multiple requests over the same connection.
    // Loop over the input handling all requests.
    // Note: the 'conn_done' flag is used to break from the inner-while loop.
    bool conn_done = false;
    while (!conn_done) {
        ssize_t nbytes_read;
        nbytes_read = read(conn_fd, read_buff, sizeof(read_buff));
        if (nbytes_read == -1) {
            perror("read");
            break;
        } else if (nbytes_read == 0) {
            // The client has closed the connection.
            break;
        }

        int i = 0;
        while (i < nbytes_read) {
            // Find individual requests by the sequence '\r\n\r\n' (which
            // separates the headers from the body).
            if (read_buff[i] != '\r') {
                i++;
                continue;
            }
            i++;
            if (!(i < nbytes_read && read_buff[i] == '\n')) {
                continue;
            }
            i++;
            if (!(i < nbytes_read && read_buff[i] == '\r')) {
                continue;
            }
            i++;
            if (!(i < nbytes_read && read_buff[i] == '\n')) {
                continue;
            }
            i++;

            // Found a request, send a response.
            int flags = 0;
            // Use MSG_NOSIGNAL (if available) to avoid getting killed by SIGPIPE.
            #ifdef MSG_NOSIGNAL
            flags |= MSG_NOSIGNAL;
            #endif
            int nbytes_sent = send(conn_fd, write_buff, strlen(write_buff), flags);
            if (nbytes_sent == -1) {
                perror("write");
                conn_done = true;
                break;
            }
        }
    }

    int ret = close(conn_fd);
    if (ret != 0) {
        perror("close");
    }

    return NULL;
}


int init_socket(struct sockaddr_in* addr, socklen_t addr_len) {
    int ret;

    // Create a TCP socket.
    int domain = AF_INET;
    int type = SOCK_STREAM;
    int protocol = 0;
    int sock_fd = socket(domain, type, protocol);
    if (sock_fd == -1) {
        perror("socket");
        exit(1);
    }

    // Avoid "bind: address already in use" when restarting the server.
    int level = SOL_SOCKET;
    int option_name = SO_REUSEADDR;
    int option = 1;
    ret = setsockopt(sock_fd, level, option_name, &option, sizeof(option));
    if (ret != 0) {
        perror("setsockopt");
        exit(1);
    }

    // Use SO_NOSIGPIPE (if available) to avoid getting killed by SIGPIPE.
    // See https://web.archive.org/web/20070502105136/http://lists.apple.com/archives/macnetworkprog/2002/Dec/msg00091.html
    #ifdef SO_NOSIGPIPE
    level = SOL_SOCKET;
    option_name = SO_NOSIGPIPE;
    option = 1;
    ret = setsockopt(sock_fd, level, option_name, &option, sizeof(option));
    if (ret != 0) {
        perror("setsockopt");
        exit(1);
    }
    #endif

    // Bind the socket.
    ret = bind(sock_fd, (struct sockaddr*)addr, addr_len);
    if (ret != 0) {
        perror("bind");
        exit(1);
    }

    // Listen on the socket.
    ret = listen(sock_fd, BACKLOG);
    if (ret != 0) {
        perror("listen");
        exit(1);
    }
    printf("Listening on port %d.\n", PORT);
    printf("Test with e.g. 'wrk -t8 -c32 -d5s http://127.0.0.1:%d'.\n", PORT);

    return sock_fd;
}


int get_int_from_env(char* name) {
    char* str_value = getenv(name);
    if (str_value == NULL) {
        return -1;
    }
    char** endptr = NULL;
    int base = 10;
    long long_value = strtol(str_value, endptr, base);
    if (long_value == 0 && errno != 0) {
        perror("strtol");
        return -1;
    }
    return (int)long_value;
}


void init_from_env() {
    int value;
    // Read configuration values from the environment.
    value = get_int_from_env("PORT");
    if (value != -1) {
        PORT = value;
    }
    value = get_int_from_env("BACKLOG");
    if (value != -1) {
        BACKLOG = value;
    }
    printf("Configuration: PORT=%d, BACKLOG=%d\n", PORT, BACKLOG);
    printf("Override these values using ENV vars.\n");
}


int main(int argc, char** argv) {
    int ret;

    init_from_env();

    // Socket address configuration.
    struct sockaddr_in addr;
    memset(&addr, 0, sizeof(addr));
    addr.sin_family = AF_INET;
    addr.sin_addr.s_addr = htonl(INADDR_ANY);
    addr.sin_port = htons(PORT);
    socklen_t addr_len = sizeof(addr);

    int sock_fd = init_socket(&addr, addr_len);

    while (true) {
        // Accept a new connection.
        int conn_fd;
        conn_fd = accept(sock_fd, (struct sockaddr*)&addr, &addr_len);
        if (conn_fd == -1) {
            perror("accept");
            continue;
        }

        pthread_t thread;
        pthread_attr_t* attributes = NULL;
        ret = pthread_create(&thread, attributes, handle_connection, (void*)(long)conn_fd);
        if (ret != 0) {
            perror("pthread_create");
            exit(1);
        }
    }

    return 0;
}