fennec type checking benchmark

fennec-out.txt

fennec check
error: argument 2 of function 'Qux\foo' expected `int`, but `non-empty-list { 12 }` was found
  ┌─ main.php:9:14
  │
9 │ foo("hello", [12], 1.0);
  │              ^^^^ argument 2 has type `non-empty-list { 12 }`
  │
  ┌─ foo.php:7:27
  │
7 │ function foo(string $bar, int $baz, bool $f): void
  │                           --- parameter `$baz` has type `int`
  │
  = help: ensure the argument type is compatible with the parameter type

error: argument 3 of function 'Qux\foo' expected `bool`, but `float` was found
  ┌─ main.php:9:20
  │
9 │ foo("hello", [12], 1.0);
  │                    ^^^ argument 3 has type `float`
  │
  ┌─ foo.php:7:37
  │
7 │ function foo(string $bar, int $baz, bool $f): void
  │                                     ---- parameter `$f` has type `bool`
  │
  = help: ensure the argument type is compatible with the parameter type

error: found 2 issues: 2 error(s)

foo.php

<?php

declare(strict_types=1);

namespace Qux;

function foo(string $bar, int $baz, bool $f): void
{
    $str = (string) $baz;

    echo $bar, " => ", $str, "\n";

    if ($f) {
      echo "f\n";
    }
}

main.php

<?php

declare(strict_types=1);

use function Qux\foo;

require __DIR__ . '/foo.php';

foo("hello", [12], 1.0);

phpstan-out.txt

./../phpstan-src/bin/phpstan analyze
Note: Using configuration file /Users/azjezz/Project/type-checking-example/phpstan.neon.
 2/2 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100%

 ------ ---------------------------------------------------------------------------
  Line   main.php
 ------ ---------------------------------------------------------------------------
  9      Parameter #2 $baz of function Qux\foo expects int, array<int, int> given.
         🪪  argument.type
  9      Parameter #3 $f of function Qux\foo expects bool, float given.
         🪪  argument.type
 ------ ---------------------------------------------------------------------------



 [ERROR] Found 2 errors

psalm-out.txt

./../psalm/psalm
Target PHP version: 8.3 (inferred from current PHP version).
Scanning files...
Analyzing files...



ERROR: InvalidArgument - main.php:9:14 - Argument 2 of Qux\foo expects int, but list{12} provided (see https://psalm.dev/004)
foo("hello", [12], 1.0);


ERROR: InvalidArgument - main.php:9:20 - Argument 3 of Qux\foo expects bool, but float(1) provided (see https://psalm.dev/004)
foo("hello", [12], 1.0);


------------------------------
2 errors found
------------------------------

Checks took 0.01 seconds and used 2.724MB of memory
No files analyzed
Psalm was able to infer types for 100% of the codebase

result.txt

hyperfine \
   -n fennec "fennec check" \
   -n hello-world "php -r \"echo 'hello, world';\"" \
   -n phpstan-tracing-no-config "php -dopcache.jit=tracing ./../phpstan-src/bin/phpstan analyze main.php foo.php" \
   -n phpstan-tracing "php -dopcache.jit=tracing ./../phpstan-src/bin/phpstan analyze" \
   -n psalm-tracing "php -dopcache.jit=tracing ./../psalm/psalm" \
   -n phpstan-function-no-config "php -dopcache.jit=1205 ./../phpstan-src/bin/phpstan analyze main.php foo.php" \
   -n phpstan-function "php -dopcache.jit=1205 ./../phpstan-src/bin/phpstan analyze" \
   -n psalm-function "php -dopcache.jit=1025 ./../psalm/psalm" \
   -n phpstan-no-opcache-no-config "php -dopcache.enabled_cli=0 ./../phpstan-src/bin/phpstan analyze main.php foo.php" \
   -n phpstan-no-opcache "php -dopcache.enabled_cli=0 ./../phpstan-src/bin/phpstan analyze" \
   -n psalm-no-opcache "php -dopcache.enabled_cli=0 ./../psalm/psalm" --min-runs 50 -w 10 -i 2> /dev/null
Benchmark 1: fennec
  Time (mean ± σ):       2.2 ms ±   0.1 ms    [User: 1.0 ms, System: 1.3 ms]
  Range (min … max):     1.9 ms …   3.2 ms    681 runs

Benchmark 2: hello-world
  Time (mean ± σ):      26.7 ms ±   0.7 ms    [User: 16.9 ms, System: 7.2 ms]
  Range (min … max):    25.5 ms …  28.9 ms    106 runs

Benchmark 3: phpstan-tracing-no-config
  Time (mean ± σ):     635.1 ms ±   4.7 ms    [User: 446.0 ms, System: 111.5 ms]
  Range (min … max):   625.0 ms … 644.7 ms    50 runs

Benchmark 4: phpstan-tracing
  Time (mean ± σ):     214.5 ms ±   4.8 ms    [User: 167.8 ms, System: 43.8 ms]
  Range (min … max):   204.9 ms … 230.6 ms    50 runs

Benchmark 5: psalm-tracing
  Time (mean ± σ):     197.3 ms ±   2.0 ms    [User: 162.3 ms, System: 29.8 ms]
  Range (min … max):   191.5 ms … 200.5 ms    50 runs

Benchmark 6: phpstan-function-no-config
  Time (mean ± σ):     873.2 ms ±  10.2 ms    [User: 661.5 ms, System: 138.4 ms]
  Range (min … max):   845.6 ms … 889.1 ms    50 runs

Benchmark 7: phpstan-function
  Time (mean ± σ):     440.8 ms ±   8.6 ms    [User: 374.4 ms, System: 63.8 ms]
  Range (min … max):   416.2 ms … 447.6 ms    50 runs

Benchmark 8: psalm-function
  Time (mean ± σ):     197.0 ms ±   1.3 ms    [User: 163.1 ms, System: 28.7 ms]
  Range (min … max):   191.5 ms … 198.6 ms    50 runs

Benchmark 9: phpstan-no-opcache-no-config
  Time (mean ± σ):     637.0 ms ±   4.8 ms    [User: 448.6 ms, System: 111.4 ms]
  Range (min … max):   626.6 ms … 646.4 ms    50 runs

Benchmark 10: phpstan-no-opcache
  Time (mean ± σ):     209.8 ms ±   1.9 ms    [User: 166.4 ms, System: 40.8 ms]
  Range (min … max):   203.9 ms … 214.7 ms    50 runs

Benchmark 11: psalm-no-opcache
  Time (mean ± σ):     196.8 ms ±   1.6 ms    [User: 162.7 ms, System: 29.1 ms]
  Range (min … max):   191.5 ms … 199.7 ms    50 runs

Summary
  fennec ran
   12.14 ± 0.88 times faster than hello-world
   89.46 ± 6.14 times faster than psalm-no-opcache
   89.51 ± 6.13 times faster than psalm-function
   89.65 ± 6.18 times faster than psalm-tracing
   95.33 ± 6.55 times faster than phpstan-no-opcache
   97.48 ± 6.99 times faster than phpstan-tracing
  200.33 ± 14.19 times faster than phpstan-function
  288.61 ± 19.78 times faster than phpstan-tracing-no-config
  289.49 ± 19.85 times faster than phpstan-no-opcache-no-config
  396.86 ± 27.44 times faster than phpstan-function-no-config