Mutation Testing for PHPUnit's own test suite

errors.md

Question	Answer
Infection version	0.28.1
Test Framework version	PHPUnit 11.1-dev
PHP version	8.3.4
Platform	Fedora
Github Repo	https://github.com/sebastianbergmann/phpunit

The Finished class has a public method named test.

Infection's PublicVisibility mutator replaces public and protected and Infection then claims that this mutation survived, as can be seen in this screenshot:

However, when I perform this mutation manually ...

diff --git a/src/Event/Events/Test/Lifecycle/Finished.php b/src/Event/Events/Test/Lifecycle/Finished.php
index 72032cf71..ae75d35c1 100644
--- a/src/Event/Events/Test/Lifecycle/Finished.php
+++ b/src/Event/Events/Test/Lifecycle/Finished.php
@@ -37,7 +37,7 @@ public function telemetryInfo(): Telemetry\Info
         return $this->telemetryInfo;
     }
 
-    public function test(): Code\Test
+    protected function test(): Code\Test
     {
         return $this->test;
     }

... and try to run PHPUnit's test suite then I get this error:

$ ./phpunit --testsuite unit
PHPUnit 11.1-gb18d5f2fe3 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.3.4
Configuration: /usr/local/src/phpunit/phpunit.xml

.

An error occurred inside PHPUnit.

Message:  Call to protected method PHPUnit\Event\Test\Finished::test() from scope PHPUnit\Runner\ResultCache\ResultCacheHandler
Location: /usr/local/src/phpunit/src/Runner/ResultCache/ResultCacheHandler.php:119

#0 /usr/local/src/phpunit/src/Runner/ResultCache/Subscriber/TestFinishedSubscriber.php(27): PHPUnit\Runner\ResultCache\ResultCacheHandler->testFinished()
#1 /usr/local/src/phpunit/src/Event/Dispatcher/DirectDispatcher.php(102): PHPUnit\Runner\ResultCache\TestFinishedSubscriber->notify()
#2 /usr/local/src/phpunit/src/Event/Dispatcher/DeferringDispatcher.php(45): PHPUnit\Event\DirectDispatcher->dispatch()
#3 /usr/local/src/phpunit/src/Event/Emitter/DispatchingEmitter.php(962): PHPUnit\Event\DeferringDispatcher->dispatch()
#4 /usr/local/src/phpunit/src/Framework/TestRunner.php(235): PHPUnit\Event\DispatchingEmitter->testFinished()
#5 /usr/local/src/phpunit/src/Framework/TestCase.php(337): PHPUnit\Framework\TestRunner->run()
#6 /usr/local/src/phpunit/src/Framework/TestSuite.php(369): PHPUnit\Framework\TestCase->run()
#7 /usr/local/src/phpunit/src/Framework/TestSuite.php(369): PHPUnit\Framework\TestSuite->run()
#8 /usr/local/src/phpunit/src/Framework/TestSuite.php(369): PHPUnit\Framework\TestSuite->run()
#9 /usr/local/src/phpunit/src/TextUI/TestRunner.php(62): PHPUnit\Framework\TestSuite->run()
#10 /usr/local/src/phpunit/src/TextUI/Application.php(200): PHPUnit\TextUI\TestRunner->run()
#11 /usr/local/src/phpunit/phpunit(104): PHPUnit\TextUI\Application->run()
#12 {main}

The mutation causes an error in PHPUnit's test runner that is not specific to a single test. Maybe mutating the test framework's / test runner's code so that it no longer works is a very special case? :)

How to reproduce

$ git clone [email protected]:sebastianbergmann/phpunit.git

$ cd phpunit

$ git show --oneline -s
b18d5f2fe3 (HEAD -> main, origin/main, origin/HEAD) Add tests

$ ./tools/composer install

$ ./tools/infection --version                                                                               
Infection - PHP Mutation Testing Framework version 0.28.1

$ ./tools/infection --threads=max --ignore-msi-with-no-mutations --only-covered --logger-html=infection.html

mutation-testing-phpunit-b18d5f2fe3a1a0258f7b0de4fab2e32611dd87bc.md

5cd77a049c142595a4a1f64bb9a8f2ccbf61ee64 (before b18d5f2fe3a1a0258f7b0de4fab2e32611dd87bc)

6429 mutations were generated:
    2790 mutants were killed
       0 mutants were configured to be ignored
       0 mutants were not covered by tests
    1483 covered mutants were not detected
    2098 errors were encountered
      48 syntax errors were encountered
      10 time outs were encountered
       0 mutants required more time than configured

Metrics:
         Mutation Score Indicator (MSI): 76%
         Mutation Code Coverage: 100%
         Covered Code MSI: 76%

b18d5f2fe3a1a0258f7b0de4fab2e32611dd87bc

6432 mutations were generated:
     694 mutants were killed
       0 mutants were configured to be ignored
       0 mutants were not covered by tests
    1105 covered mutants were not detected
    4630 errors were encountered
       0 syntax errors were encountered
       3 time outs were encountered
       0 mutants required more time than configured

Metrics:
         Mutation Score Indicator (MSI): 82%
         Mutation Code Coverage: 100%
         Covered Code MSI: 82%