Benchmarking Enum vs MapSet usage for ajvondrak/remote_ip

bench.exs

defmodule Algorithm do
  @enum ~w[forwarded x-forwarded-for x-client-ip x-real-ip]
  @mapset MapSet.new(@enum)

  def using_enum(input) do
    Enum.filter(input, &Enum.member?(@enum, &1))
  end

  def using_mapset(input) do
    Enum.filter(input, &MapSet.member?(@mapset, &1))
  end

  # This is effectively what RemoteIp.from/2 does.
  def using_mapset_at_runtime(input) do
    Enum.filter(input, &MapSet.member?(MapSet.new(@enum), &1))
  end
end

algorithms = %{
  "algorithm using Enum" => &Algorithm.using_enum/1,
  "algorithm using MapSet" => &Algorithm.using_mapset/1,
  "algorithm using MapSet at runtime" => &Algorithm.using_mapset_at_runtime/1,
}

inputs = %{
  "hit 1" => ~w[forwarded],
  "hit 2" => ~w[x-forwarded-for],
  "hit 3" => ~w[x-client-ip],
  "hit 4" => ~w[x-real-ip],
  "miss 1" => ~w[via],
  "miss 2" => ~w[x-forwarded-fo],
  "representative browser headers" => Enum.shuffle(~w[
    accept
    accept-encoding
    accept-language
    host
    referer
    sec-gpc
    upgrade-insecure-requests
    cache-control
    connection
    user-agent
    x-http-proto
    x-forwarded-for
  ]),
}

Benchee.run(algorithms, inputs: inputs)

mix.exs

defmodule Bench.MixProject do
  use Mix.Project

  def project do
    [app: :bench, version: "0.0.1", deps: deps()]
  end

  def application do
    [extra_applications: [:logger]]
  end

  defp deps do
    [{:benchee, "~> 1.0"}]
  end
end

output

$ mix do deps.get, deps.compile, run bench.exs | tee -a output
Resolving Hex dependencies...
Dependency resolution completed:
New:
  benchee 1.0.1
  deep_merge 1.0.0
* Getting benchee (Hex package)
* Getting deep_merge (Hex package)
==> deep_merge
Compiling 2 files (.ex)
Generated deep_merge app
==> benchee
Compiling 39 files (.ex)
Generated benchee app
==> bench
Generated bench app
Operating System: macOS
CPU Information: Intel(R) Core(TM) i5-1038NG7 CPU @ 2.00GHz
Number of Available Cores: 8
Available memory: 16 GB
Elixir 1.11.3
Erlang 23.2.2

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 5 s
memory time: 0 ns
parallel: 1
inputs: hit 1, hit 2, hit 3, hit 4, miss 1, miss 2, representative browser headers
Estimated total run time: 2.45 min

Benchmarking algorithm using Enum with input hit 1...
Benchmarking algorithm using Enum with input hit 2...
Benchmarking algorithm using Enum with input hit 3...
Benchmarking algorithm using Enum with input hit 4...
Benchmarking algorithm using Enum with input miss 1...
Benchmarking algorithm using Enum with input miss 2...
Benchmarking algorithm using Enum with input representative browser headers...
Benchmarking algorithm using MapSet with input hit 1...
Benchmarking algorithm using MapSet with input hit 2...
Benchmarking algorithm using MapSet with input hit 3...
Benchmarking algorithm using MapSet with input hit 4...
Benchmarking algorithm using MapSet with input miss 1...
Benchmarking algorithm using MapSet with input miss 2...
Benchmarking algorithm using MapSet with input representative browser headers...
Benchmarking algorithm using MapSet at runtime with input hit 1...
Benchmarking algorithm using MapSet at runtime with input hit 2...
Benchmarking algorithm using MapSet at runtime with input hit 3...
Benchmarking algorithm using MapSet at runtime with input hit 4...
Benchmarking algorithm using MapSet at runtime with input miss 1...
Benchmarking algorithm using MapSet at runtime with input miss 2...
Benchmarking algorithm using MapSet at runtime with input representative browser headers...

##### With input hit 1 #####
Name                                        ips        average  deviation         median         99th %
algorithm using MapSet                   3.60 M      277.49 ns  ±2480.55%           0 ns        1000 ns
algorithm using Enum                     3.59 M      278.63 ns  ±2393.10%           0 ns        1000 ns
algorithm using MapSet at runtime        1.47 M      682.14 ns  ±4208.20%        1000 ns        1000 ns

Comparison: 
algorithm using MapSet                   3.60 M
algorithm using Enum                     3.59 M - 1.00x slower +1.14 ns
algorithm using MapSet at runtime        1.47 M - 2.46x slower +404.65 ns

##### With input hit 2 #####
Name                                        ips        average  deviation         median         99th %
algorithm using Enum                     3.56 M      280.58 ns  ±2230.71%           0 ns        1000 ns
algorithm using MapSet                   2.92 M      342.15 ns  ±2374.57%           0 ns        1000 ns
algorithm using MapSet at runtime        1.34 M      747.36 ns  ±4383.74%        1000 ns        1000 ns

Comparison: 
algorithm using Enum                     3.56 M
algorithm using MapSet                   2.92 M - 1.22x slower +61.57 ns
algorithm using MapSet at runtime        1.34 M - 2.66x slower +466.77 ns

##### With input hit 3 #####
Name                                        ips        average  deviation         median         99th %
algorithm using Enum                     3.48 M      287.70 ns  ±2254.13%           0 ns        1000 ns
algorithm using MapSet                   3.34 M      299.79 ns  ±2258.34%           0 ns        1000 ns
algorithm using MapSet at runtime        1.36 M      733.00 ns  ±4158.46%        1000 ns        1000 ns

Comparison: 
algorithm using Enum                     3.48 M
algorithm using MapSet                   3.34 M - 1.04x slower +12.09 ns
algorithm using MapSet at runtime        1.36 M - 2.55x slower +445.30 ns

##### With input hit 4 #####
Name                                        ips        average  deviation         median         99th %
algorithm using Enum                     3.35 M      298.54 ns  ±2080.90%           0 ns        1000 ns
algorithm using MapSet                   3.12 M      320.89 ns  ±2317.90%           0 ns        1000 ns
algorithm using MapSet at runtime        1.46 M      683.78 ns  ±3906.30%        1000 ns        1000 ns

Comparison: 
algorithm using Enum                     3.35 M
algorithm using MapSet                   3.12 M - 1.07x slower +22.35 ns
algorithm using MapSet at runtime        1.46 M - 2.29x slower +385.23 ns

##### With input miss 1 #####
Name                                        ips        average  deviation         median         99th %
algorithm using Enum                     3.67 M      272.20 ns  ±3208.98%           0 ns        1000 ns
algorithm using MapSet                   3.32 M      300.76 ns  ±3106.08%           0 ns        1000 ns
algorithm using MapSet at runtime        1.57 M      638.71 ns  ±4331.06%        1000 ns        1000 ns

Comparison: 
algorithm using Enum                     3.67 M
algorithm using MapSet                   3.32 M - 1.10x slower +28.56 ns
algorithm using MapSet at runtime        1.57 M - 2.35x slower +366.51 ns

##### With input miss 2 #####
Name                                        ips        average  deviation         median         99th %
algorithm using Enum                     3.65 M      274.18 ns  ±3484.37%           0 ns        1000 ns
algorithm using MapSet                   3.20 M      312.64 ns  ±3018.26%           0 ns        1000 ns
algorithm using MapSet at runtime        1.58 M      633.82 ns  ±4338.29%        1000 ns        1000 ns

Comparison: 
algorithm using Enum                     3.65 M
algorithm using MapSet                   3.20 M - 1.14x slower +38.45 ns
algorithm using MapSet at runtime        1.58 M - 2.31x slower +359.64 ns

##### With input representative browser headers #####
Name                                        ips        average  deviation         median         99th %
algorithm using Enum                  1047.28 K        0.95 μs   ±534.49%           1 μs           2 μs
algorithm using MapSet                 816.26 K        1.23 μs   ±586.98%           1 μs           2 μs
algorithm using MapSet at runtime      220.81 K        4.53 μs   ±294.16%           4 μs           8 μs

Comparison: 
algorithm using Enum                  1047.28 K
algorithm using MapSet                 816.26 K - 1.28x slower +0.27 μs
algorithm using MapSet at runtime      220.81 K - 4.74x slower +3.57 μs