Skip to content

Instantly share code, notes, and snippets.

@luizbafilho

luizbafilho/perf_overview.md

Last active September 18, 2019 12:55
Show Gist options
  • Save luizbafilho/bcdf3b9c3aa000f5e33131657a344d2a to your computer and use it in GitHub Desktop.
Save luizbafilho/bcdf3b9c3aa000f5e33131657a344d2a to your computer and use it in GitHub Desktop.
Download ZIP
Performance overview from K6
Raw
perf_overview.md

k6 performance overview

In order to get a general idea of how k6 is performing, and to see if there are any low-hanging fruit in terms of optimizations we could do, I did a series of tests running k6 against a local server, testing different changes to the k6 code base.

Local environment

macOS High Sierra 10.13.13
MacBook Pro(Retina, 15-inch, Mid 2014)
Processor: 2,2Ghz Intel Core i7
Memory: 16GB 1600MHz DDR3

Here is the code of the webserver I used.

package main

import (
    "net/http"
)

func sayHello(w http.ResponseWriter, r *http.Request) {
    w.Write([]byte("Hello"))
}

func main() {
    http.HandleFunc("/", sayHello)
    if err := http.ListenAndServe(":8000", nil); err != nil {
        panic(err)
    }
}

In order to get how fast this code can go, I used wrk, the fastest load testing tool I could find.

➜  ~ wrk -t 4 -c 4 -d 10 http://localhost:8000
Running 10s test @ http://localhost:8000
  4 threads and 4 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    86.31us   25.00us   1.93ms   88.16%
    Req/Sec    10.95k   310.97    12.43k    85.15%
  440269 requests in 10.10s, 50.80MB read
Requests/sec:  43591.55
Transfer/sec:      5.03MB

After running wrk, to get a benchmark of what's possible on the machine, I started the performance investigation of k6. I did 3 different tests with the following script:

import http from "k6/http";

export default function() {
  http.get("http://127.0.0.1:8000");
}

The tests were:

  1. No changes to k6 code
  2. Changed the http.request method, given that it's the method where k6 spends the most time.
  3. Created a new Runner that uses pure Go code, without anything related to Goja, the JavaScript implementation we use. To understand how much overhead Goja adds to k6.

Tests

1. No changes

          /\      |‾‾|  /‾‾/  /‾/
     /\  /  \     |  |_/  /  / /
    /  \/    \    |      |  /  ‾‾\
   /          \   |  |‾\  \ | (_) |
  / __________ \  |__|  \__\ \___/ .io

execution: local
     output: -
     script: script.js

    duration: 10s, iterations: -
         vus: 4,   max: 4
         
    data_received..............: 20 MB  2.0 MB/s
    data_sent..................: 14 MB  1.3 MB/s
    http_req_blocked...........: avg=2.18µs   min=741ns   med=1.68µs   max=1.58ms   p(90)=2.52µs   p(95)=3.51µs
    http_req_connecting........: avg=7ns      min=0s      med=0s       max=319.79µs p(90)=0s       p(95)=0s
    http_req_duration..........: avg=157.36µs min=63.55µs med=137.32µs max=9.71ms   p(90)=214.66µs p(95)=262.45µs
    http_req_receiving.........: avg=30.85µs  min=9.77µs  med=24.73µs  max=9.4ms    p(90)=45.69µs  p(95)=57.42µs
    http_req_sending...........: avg=13.11µs  min=4.66µs  med=9.92µs   max=3.63ms   p(90)=20.78µs  p(95)=27.38µs
    http_req_tls_handshaking...: avg=0s       min=0s      med=0s       max=0s       p(90)=0s       p(95)=0s
    http_req_waiting...........: avg=113.39µs min=41.27µs med=98.64µs  max=7.65ms   p(90)=154.87µs p(95)=187.85µs
    http_reqs..................: 166497 16649.62556/s
    iteration_duration.........: avg=223.02µs min=93.76µs med=193.26µs max=12.34ms  p(90)=310.16µs p(95)=378.89µs
    iterations.................: 166496 16649.52556/s
    vus........................: 4      min=4 max=4
    vus_max....................: 4      min=4 max=4

2. With Goja, but simpler http.request

Here is the much simpler http.request implementation, compared to the original one:

func (h *HTTP) request(ctx context.Context, rt *goja.Runtime, state *common.State, method string, url URL, args ...goja.Value) (*HTTPResponse, []stats.Sample, error) {
    req := &http.Request{
        Method: "GET",
        URL:    url.URL,
    }

    client := http.Client{
        Transport: state.HTTPTransport,
    }

    tracer := netext.Tracer{}
    resp, err := client.Do(req.WithContext(netext.WithTracer(ctx, &tracer)))
    if err != nil {
        return nil, nil, err
    }
    io.Copy(ioutil.Discard, resp.Body)
    resp.Body.Close()
    trail := tracer.Done()

    return &HTTPResponse{ctx: ctx}, trail.Samples(map[string]string{}), nil
}
          /\      |‾‾|  /‾‾/  /‾/
     /\  /  \     |  |_/  /  / /
    /  \/    \    |      |  /  ‾‾\
   /          \   |  |‾\  \ | (_) |
  / __________ \  |__|  \__\ \___/ .io

execution: local
     output: -
     script: script.js

    duration: 10s, iterations: -
         vus: 4,   max: 4
    
    data_received..............: 23 MB  2.3 MB/s
    data_sent..................: 14 MB  1.4 MB/s
    http_req_blocked...........: avg=2.16µs   min=803ns   med=1.64µs   max=11.38ms  p(90)=2.47µs   p(95)=3.38µs
    http_req_connecting........: avg=7ns      min=0s      med=0s       max=371.46µs p(90)=0s       p(95)=0s
    http_req_duration..........: avg=152.56µs min=51.51µs med=134.11µs max=10.59ms  p(90)=206.94µs p(95)=253.12µs
    http_req_receiving.........: avg=28.04µs  min=8.08µs  med=22.78µs  max=9.91ms   p(90)=41.74µs  p(95)=52.15µs
    http_req_sending...........: avg=12.71µs  min=4.14µs  med=9.61µs   max=10.06ms  p(90)=20.46µs  p(95)=26.77µs
    http_req_tls_handshaking...: avg=0s       min=0s      med=0s       max=0s       p(90)=0s       p(95)=0s
    http_req_waiting...........: avg=111.8µs  min=37.89µs med=97.52µs  max=9.3ms    p(90)=153.8µs  p(95)=187.86µs
    http_reqs..................: 187513 18750.704568/s
    iteration_duration.........: avg=194.24µs min=72.1µs  med=170.76µs max=11.57ms  p(90)=264.4µs  p(95)=323.58µs
    iterations.................: 187509 18750.30458/s
    vus........................: 4      min=4 max=4
    vus_max....................: 4      min=4 max=4

3. No Goja runner

Here is the code of the Runner that executes without using Goja:

func (vu PerfRunnerVU) RunOnce(ctx context.Context) ([]stats.Sample, error) {
    uri, _ := url.Parse("http://127.0.0.1:8000")
    req := &http.Request{
        Method: "GET",
        URL:    uri,
    }

    client := http.Client{
        Transport: vu.HTTPTransport,
    }

    tracer := netext.Tracer{}
    resp, err := client.Do(req.WithContext(netext.WithTracer(ctx, &tracer)))
    if err != nil {
        return nil, err
    }
    io.Copy(ioutil.Discard, resp.Body)
    resp.Body.Close()
    trail := tracer.Done()

    return trail.Samples(map[string]string{}), nil
} 
          /\      |‾‾|  /‾‾/  /‾/
     /\  /  \     |  |_/  /  / /
    /  \/    \    |      |  /  ‾‾\
   /          \   |  |‾\  \ | (_) |
  / __________ \  |__|  \__\ \___/ .io

execution: local
     output: -
     script: script.js

    duration: 10s, iterations: -
         vus: 4,   max: 4

    http_req_blocked...........: avg=2.06µs   min=640ns   med=1.65µs   max=4.58ms   p(90)=2.4µs    p(95)=3.3µs
    http_req_connecting........: avg=12ns     min=0s      med=0s       max=784.64µs p(90)=0s       p(95)=0s
    http_req_duration..........: avg=141.6µs  min=52.39µs med=127.53µs max=7.82ms   p(90)=186.78µs p(95)=225.36µs
    http_req_receiving.........: avg=26.04µs  min=7.03µs  med=22.02µs  max=7.7ms    p(90)=38.29µs  p(95)=46.5µs
    http_req_sending...........: avg=11.07µs  min=3.51µs  med=8.61µs   max=2.79ms   p(90)=17.84µs  p(95)=22.93µs
    http_req_tls_handshaking...: avg=0s       min=0s      med=0s       max=0s       p(90)=0s       p(95)=0s
    http_req_waiting...........: avg=104.48µs min=38.23µs med=93.41µs  max=5.9ms    p(90)=140.32µs p(95)=167.8µs
    http_reqs..................: 237082 23708.072984/s
    iterations.................: 237080 23707.872985/s
    vus........................: 4      min=4 max=4
    vus_max....................: 4      min=4 max=4

As we can see, Goja adds some significant overhead, but that is a trade-off we need to accept in order for k6 to be able to run JavaScript.

Bombardier

To compare k6 with a HTTP benchmarking tool written in Go, I ran bombardier against the server using the standard HTTP client and the fasthttp lib, an HTTP library tuned for high performance.

bombardier --http1 -c 4 -d 10s -l http://127.0.0.1:8000
Bombarding http://127.0.0.1:8000 for 10s using 4 connection(s)
[========================================================] 10s
Done!
Statistics        Avg      Stdev        Max
  Reqs/sec     26780.95    1420.26   29373.67
  Latency      146.91us    32.82us     6.26ms
  Latency Distribution
     50%   140.00us
     75%   156.00us
     90%   177.00us
     95%   197.00us
     99%   320.00us
  HTTP codes:
    1xx - 0, 2xx - 267770, 3xx - 0, 4xx - 0, 5xx - 0
    others - 0
  Throughput:     5.51MB/s
bombardier --fasthttp -c 4 -d 10s -l http://127.0.0.1:8000
Bombarding http://127.0.0.1:8000 for 10s using 4 connection(s)
[========================================================] 10s
Done!
Statistics        Avg      Stdev        Max
  Reqs/sec     42227.74    1556.34   48234.90
  Latency       92.62us    16.00us     4.72ms
  Latency Distribution
     50%    89.00us
     75%   104.00us
     90%   120.00us
     95%   131.00us
     99%   160.00us
  HTTP codes:
    1xx - 0, 2xx - 422172, 3xx - 0, 4xx - 0, 5xx - 0
    others - 0
  Throughput:     7.37MB/s

As you can see, the fasthttp lib is way faster, but unfortunately we can't use it because it doesn't support tracing or HTTP2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment