Testing in Magento 2

Testing in Magento 2

Use existing Magento 2 tests as examples, especially for testing API calls.

Note, however, that different Magento 2 sections have been written by different developers, who all have different approaches to bootstrapping the environment, creating fixtures, etc, so don't take any one approach as gospel. They are all equally valid. If one approach doesn't work for your use case, try and find a different one.

Quick start quide to testing existing classes

First, decide what to test. This will then inform whether the first test to create will be a unit or integration test
Save the class with the suffix Test in the Test subdirectory (e.g. Test/Integration/ or Test/Unit/ as appropriate)
Extend the class with \PHPUnit\Framework\TestCase
Update the namespace
Add the Test suffix to the class name
Add a demo test:

    public function testTestEnvironmentIsSetupCorrectly()
    {
        $condition = true;
        $this->assertTrue($condition);
    }

Open the class under test in a split window whilst writing the test

Environment setup

Configuring PHPUnit

Use this sample phpunit.xml file for integration tests:

// File: dev/tests/integration/phpunit.xml
<?xml version="1.0" encoding="UTF-8"?>
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:noNamespaceSchemaLocation="http://schema.phpunit.de/6.2/phpunit.xsd"
         colors="true"
         columns="max"
         beStrictAboutTestsThatDoNotTestAnything="false"
         bootstrap="./framework/bootstrap.php"
         stderr="true"
>
    <!-- Test suites definition -->
    <testsuites>
        <!-- Memory tests run first to prevent influence of other tests on accuracy of memory measurements -->
        <testsuite name="Memory Usage Tests">
            <file>testsuite/Magento/MemoryUsageTest.php</file>
        </testsuite>
        <testsuite name="Magento Integration Tests">
            <directory suffix="Test.php">testsuite</directory>
            <exclude>testsuite/Magento/MemoryUsageTest.php</exclude>
        </testsuite>
        <!-- Only run tests in custom modules -->
        <testsuite name="Mage2Kata Tests">
            <directory>../../../app/code/*/*/Test/*</directory>
            <exclude>../../../app/code/Magento</exclude>
        </testsuite>
    </testsuites>
    <!-- Code coverage filters -->
    <filter>
        <whitelist addUncoveredFilesFromWhiteList="true">
            <directory suffix=".php">../../../app/code/Magento</directory>
            <directory suffix=".php">../../../lib/internal/Magento</directory>
            <exclude>
                <directory>../../../app/code/*/*/Test</directory>
                <directory>../../../lib/internal/*/*/Test</directory>
                <directory>../../../lib/internal/*/*/*/Test</directory>
                <directory>../../../setup/src/*/*/Test</directory>
            </exclude>
        </whitelist>
    </filter>
    <!-- PHP INI settings and constants definition -->
    <php>
        <includePath>.</includePath>
        <includePath>testsuite</includePath>
        <ini name="date.timezone" value="Europe/London"/>
        <ini name="xdebug.max_nesting_level" value="200"/>
        <!-- Local XML configuration file ('.dist' extension will be added, if the specified file doesn't exist) -->
        <const name="TESTS_INSTALL_CONFIG_FILE" value="etc/install-config-mysql.php"/>
        <!-- Local XML configuration file ('.dist' extension will be added, if the specified file doesn't exist) -->
        <const name="TESTS_GLOBAL_CONFIG_FILE" value="etc/config-global.php"/>
        <!-- Semicolon-separated 'glob' patterns, that match global XML configuration files -->
        <const name="TESTS_GLOBAL_CONFIG_DIR" value="../../../app/etc"/>
        <!-- Whether to cleanup the application before running tests or not -->
        <const name="TESTS_CLEANUP" value="disabled"/>
        <!-- Memory usage and estimated leaks thresholds -->
        <!--<const name="TESTS_MEM_USAGE_LIMIT" value="1024M"/>-->
        <const name="TESTS_MEM_LEAK_LIMIT" value=""/>
        <!-- Whether to output all CLI commands executed by the bootstrap and tests -->
        <!--<const name="TESTS_EXTRA_VERBOSE_LOG" value="1"/>-->
        <!-- Path to Percona Toolkit bin directory -->
        <!--<const name="PERCONA_TOOLKIT_BIN_DIR" value=""/>-->
        <!-- CSV Profiler Output file -->
        <!--<const name="TESTS_PROFILER_FILE" value="profiler.csv"/>-->
        <!-- Bamboo compatible CSV Profiler Output file name -->
        <!--<const name="TESTS_BAMBOO_PROFILER_FILE" value="profiler.csv"/>-->
        <!-- Metrics for Bamboo Profiler Output in PHP file that returns array -->
        <!--<const name="TESTS_BAMBOO_PROFILER_METRICS_FILE" value="../../build/profiler_metrics.php"/>-->
        <!-- Whether to output all CLI commands executed by the bootstrap and tests -->
        <const name="TESTS_EXTRA_VERBOSE_LOG" value="1"/>
        <!-- Magento mode for tests execution. Possible values are "default", "developer" and "production". -->
        <const name="TESTS_MAGENTO_MODE" value="developer"/>
        <!-- Minimum error log level to listen for. Possible values: -1 ignore all errors, and level constants form http://tools.ietf.org/html/rfc5424 standard -->
        <const name="TESTS_ERROR_LOG_LISTENER_LEVEL" value="DEBUG"/>
        <!-- Connection parameters for MongoDB library tests -->
        <!--<const name="MONGODB_CONNECTION_STRING" value="mongodb://localhost:27017"/>-->
        <!--<const name="MONGODB_DATABASE_NAME" value="magento_integration_tests"/>-->
    </php>
    <!-- Test listeners -->
    <listeners>
        <listener class="Magento\TestFramework\Event\PhpUnit"/>
        <listener class="Magento\TestFramework\ErrorLog\Listener"/>
    </listeners>
</phpunit>

Use this sample phpunit.xml file for unit tests:

// File: dev/tests/unit/phpunit.xml
<?xml version="1.0" encoding="UTF-8"?>
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:noNamespaceSchemaLocation="http://schema.phpunit.de/4.1/phpunit.xsd"
         colors="true"
         bootstrap="./framework/bootstrap.php"
>
    <testsuite name="Mage2Katas Unit Tests">
        <directory suffix="Test.php">../../../app/code/*/*/Test/Unit</directory>
    </testsuite>
    <php>
        <ini name="date.timezone" value="Europe/London"/>
        <ini name="xdebug.max_nesting_level" value="200"/>
    </php>
    <filter>
        <whitelist addUncoveredFilesFromWhiteList="true">
            <directory suffix=".php">../../../app/code/*</directory>
            <directory suffix=".php">../../../lib/internal/Magento</directory>
            <directory suffix=".php">../../../setup/src/*</directory>
            <exclude>
                <directory>../../../app/code/*/*/Test</directory>
                <directory>../../../lib/internal/*/*/Test</directory>
                <directory>../../../lib/internal/*/*/*/Test</directory>
                <directory>../../../setup/src/*/*/Test</directory>
            </exclude>
        </whitelist>
    </filter>
</phpunit>

Configuring PhpStorm

Tell PhpStorm which version of PHPUnit it should use

Go to Settings, Languages & Frameworks, PHP, Test Frameworks
Add a new instance of PHPUnit.
Under PHPUnit library:
1. Select Use Composer autoloader.
2. In Path to script: , enter the path to the autoloader, e.g. /var/www/html/project-name/htdocs/vendor/autoload.php
The remaining settings can be left at their defaults.

Tell PhpStorm how to run the tests

These instructions are for running unit tests, but the configuration for unit tests is identical - just substitute unit for integration.

Go to Run, Edit Configurations.
Create a new PHPUnit configuration with the following values:
- Name: Mage2Katas Integration Test Rig
- Test Runner:
  - Test Scope: Defined in the configuration file
  - Use alternative configuration file: /path/to/magento/root/dev/tests/integration/phpunit.xml
  - Test Runner options: --testsuite "Mage2Kata Tests"

Configure the database (for integration tests)

Copy the install-config-mysql-php.dist file and update the database connection details accordingly:

projecteight@devbox:/var/www/vhosts/magento2.localhost.com$ cp -f dev/tests/integration/etc/install-config-mysql.php.dist dev/tests/integration/etc/install-config-mysql.php

There are more detailed notes on configuring the environment for integration tests in the Magento 2 DevDocs [3]3

Unit tests

Writing tests

The tests for a class Class go into a class ClassTest.

ClassTest inherits (most of the time) from PHPUnit\Framework\TestCase.

The tests are public methods that are named test*.

Alternatively, you can use the @test annotation in a method's docblock to mark it as a test method.

Inside the test methods, assertion methods such as assertEquals() are used to assert that an actual value matches an expected value.

Tests must be stored in [vendor name]/[module name]/Test/Unit/. The directory structure under here mirrors the directory structure of the class being tested by convention.

[vendor name]
└──[module name]
   └───Model
   │   └───Feature.php
   └───Test
       └───Unit
           └───Model
               └───FeatureTest.php

The name of the file containing the test class is, by convention, the name of the class plus the suffix Test, e.g. The tests for Feature.php would be in the file FeatureTest.php.

Tests should generally be named after the class they are testing:

<?php
// The methods in this class...
class Feature {
	public function getConfigValue() { ... }
}
// ...are tested by the tests in this class
class FeatureTest {
	public function testGetConfigValue() { ... }
}

Test names should be descriptive, e.g. testsThatValueIsNotNull() rather than testValue().

Data Providers: Passing data into tests

Specify a Data Provider method in the docblock of the test method:

/**
 * Tests that the table name returned from Magento matches the one we expect (from the data provider)
 * @dataProvider tableNameProvider
 */
public function testTableName($tableName)
{
    $this->assertEquals($this->_defaultIndexerResource->getIdxTable($tableName), $tableName);
}

/**
 * @return array
 */
public function tableNameProvider()
{
    return 'projecteight_bestsellersindex_product_index_bestseller';
}

Testing Exceptions

/**
 * @expectedException LocalizedException
 */
public function testLocalizedExceptionIsThrown()
{

}

// Alternatively you can the setExpectedException() method:
public function testExceptionHasRightMessage()
{
    $this->setExpectedException(
      'InvalidArgumentException', 'Right Message'
    );
    throw new InvalidArgumentException('Right Message');
}

Fixtures

If you want to re-use a data provider across multiple test cases, then you can create a fixture class and call it statically:

<?php

declare(strict_types=1);

namespace PavingDirect\TradeRegistration\Test\Integration\Fixtures;

use \Magento\UrlRewrite\Model\OptionProvider;
use \Magento\UrlRewrite\Model\UrlRewrite;

/**
 * Class CreateUrlRewrite
 *
 * Fixture class. Adds a custom rewrite to the db.
 *
 * @package PavingDirect\TradeRegistration\Test\Integration\Fixtures
 */
class CreateUrlRewrite
{
    /**
     * @throws \Magento\Framework\Exception\AlreadyExistsException
     */
    public static function createUrlRewrite()
    {
        /** @var \Magento\Framework\ObjectManagerInterface $objectManager */
        $objectManager = \Magento\TestFramework\Helper\Bootstrap::getObjectManager();

        /** @var \Magento\UrlRewrite\Model\ResourceModel\UrlRewrite $rewriteResource */
        $rewriteResource = $objectManager->create(
            \Magento\UrlRewrite\Model\ResourceModel\UrlRewrite::class
        );

        $storeId = 1;

        /** @var UrlRewrite $rewrite */
        $rewrite = $objectManager->create(UrlRewrite::class);
        $rewrite->setEntityType('custom')
                ->setEntityId(0)
                ->setRequestPath('trade/account/login')
                ->setTargetPath('customer/account/login')
                ->setRedirectType(OptionProvider::TEMPORARY)
                ->setStoreId($storeId)
                ->setDescription(null)
                ->setMetadata(null);
        $rewriteResource->save($rewrite);
    }
}

Then call it wherever it is needed:

<?php

declare(strict_types=1);

namespace PavingDirect\TradeRegistration\Test\Integration\Setup;

use Magento\UrlRewrite\Service\V1\Data\UrlRewrite;

class UpgradeDataTest extends \PHPUnit\Framework\TestCase
{
    public function testTheRewriteIsDeleted()
    {
        // Call our fixture class
        \PavingDirect\TradeRegistration\Test\Integration\Fixtures\CreateUrlRewrite::createUrlRewrite();

        // Now test as normal
        /** @var $objectManager \Magento\TestFramework\ObjectManager */
        $objectManager = \Magento\TestFramework\Helper\Bootstrap::getObjectManager();

        /** @var \Magento\Store\Api\StoreResolverInterface $storeResolver */
        $storeResolver = $objectManager->get(\Magento\Store\Api\StoreResolverInterface::class);

        /** @var \Magento\UrlRewrite\Model\UrlFinderInterface $urlFinder */
        $urlFinder = $objectManager->get(\Magento\UrlRewrite\Model\UrlFinderInterface::class);
        $rewrite   = $urlFinder->findOneByData([
            UrlRewrite::REQUEST_PATH => \PavingDirect\TradeRegistration\Setup\UpgradeData::TRADE_ACCOUNT_LOGIN_REQUEST_PATH,
            UrlRewrite::STORE_ID     => $storeResolver->getCurrentStoreId(),
        ]);

        $this->assertNotNull($rewrite);
    }
}

Best practices

Vinai Kopp has written an article summarising best practices for writing code which is easy to test [2]2. In general, they can be summarised thus:

What should be tested? Behaviour. Any custom behaviour which has been added.
Minimise dependencies
Prefer interfaces over classes (in particular, avoid using methods of concrete classes which aren't in the interface)
Keep classes and methods as small, simple and SOLID as possible
If you need to test private or protected methods, this is a sign the class is doing too much. Consider extracting the private functionality into a separate class and using that class as a collaborator. The extracted class then provides the functionality using a public method and can easily be tested.
Obey the "Tell, don't ask" principle. Avoid using getters on objects. Push that functionality into the class itself.
Obey the Law of Demeter. It states that methods can only be called on objects that:
- Are received as constructor arguments
- Are received as arguments to the current method
- Are instantiated in the current method
Avoid method chaining, i.e. $this->getObject()->setFoo($foo)->doBar().
It is as important to write tests which confirm that the logic under test does not affect anything else, as it is to write tests which confirm it does what it is supposed to.
- E.g. For a plugin that sets a value on specific pages, tests should be written that confirm that the value is set on the specific pages, but also that the value is not set on any other page
The DevDocs say:

2.2.2. Factories SHOULD be used for object instantiation instead of new keyword. An object SHOULD be replaceable for testing or extensibility purposes. Exception: DTOs. There is no behavior in DTOs, so there is no reason for its replaceability. Tests can create real DTOs for stubs. Data interfaces, Exceptions and Zend_Db_Expr are examples of DTOs.

Integration or unit test?

If the answer to any of these is 'yes', then choose integration over unit test:

Does the class/method under test have numerous dependencies or a lot of 'behavioural' logic?
Does the class/method under test interact with any resources (databases, file system, 3rd party APIs), directly or indirectly?
Does the class/method under test include any 'glue' or 'wiring' of collaborators?
Is the class/method under test a controller, a model triad class (model/resource model/collection), or a factory?

If the answer to any of these is 'yes', then a unit test is more appropriate:

Does the class/method under test have a minimal amount of 'behaviour' (i.e. A class with one method that does one thing succinctly)?
Are there few, if any dependencies?
Is it possible to write a 'black box' test (i.e. A test which tests inputs and outputs, rather than the internal logic of a method)?
Only involves testing a single method (or 'unit') of a class?

Object Manager

It is considered best practice to avoid using the Object Manager in unit tests [1]1 (though this doesn't stop the core team from doing it in their tests).

Using Stubs

@todo

Using Mocks

Only interfaces should be mocked.

<?php

class FullTest extends \PHPUnit\Framework\TestCase {
	
    /** @var Bestseller | \PHPUnit\Framework\MockObject\MockObject */
    protected $_defaultIndexerResource;
	
    /**
     * Is called before running a test
     */
    protected function setUp()
    {
        $this->_defaultIndexerResource = $this->getMockBuilder(Bestseller::class)
                                      ->disableOriginalConstructor()
                                      ->getMock();
    }
}

Running unit tests from the CLI

Executing unit tests from outside Docker

This command must be executed in the folder with the docker-compose.yml file.

# docker-compose exec <container id> <program> <program arguments>
docker-compose exec --user=magento2 web /var/www/magento2/bin/magento dev:tests:run unit

Executing unit tests from inside Docker

Start a shell to the Docker container as the Magento 2 user:

# docker-compose exec <user> <container name without docker-composer prefix> <command>
docker-compose exec --user=magento2 web /bin/bash

Then execute the start unit test command:

magento dev:tests:run unit

Running unit tests from PhpStorm

With Docker

Setup a remote PHP interpreter which connects to PHP running inside the Docker container
Configure PHPUnit in settings.
1. Select Use custom autoloader and enter the path to vendor/autoload.php
2. (Optional) Specify dev/tests/unit/phpunit.xml.dist as the default configuration
Right-click any folder or class containing tests and select Run.
1. For extra options, you can create a custom configuration by going to Run > Edit Configurations...

Without Docker

Specify a PHP interpreter which matches the version of PHPUnit you're using (Magento 2.3 uses PHPUnit 6.5.14)
Configure PHPUnit in settings.
1. Select Use custom autoloader and enter the path to vendor/autoload.php
2. (Optional) Specify dev/tests/unit/phpunit.xml.dist as the default configuration
Right-click any folder or class containing tests and select Run.
1. For extra options, you can create a custom configuration by going to Run > Edit Configurations...

Troubleshooting

The Magento_Developer module must be enabled:

php -f bin/magento module:enable Magento_Developer

The configuration for each test type is located in dev/tests/<type>

Exception: Could not connect to the Amqp Server

Assuming you don't want Magento to do this when running integration tests, just remove the AMQP details from the dev/tests/integration/etc/install-config-mysql.local.php file and make sure that the TESTS_INSTALL_CONFIG_FILE value in dev/tests/integration/phpunit.local.xml has been set to the same value (i.e. To use your config file, not the default 'dist' one).

An expectation has not been met, even if you've stepped through the code under test and verified it has been called

Try removing the 'test everything is working' test method, especially if you're using a setup method, because that gets called before every test method.

Error : Class 'Magento\TestFramework\Helper\Bootstrap' not found

This may occur when running Integration tests. Verify that PhpStorm is using the PHPUnit configuration you created and not one it auto-generated.

This happens because PHPUnit can't find the autoloader and so can't autoload classes. Assuming you setup PHPUnit as described at the beginning of this document, this should never happen.

'No method matcher is set'

Verify that expectations set on mocks have a method() call on them:

// From this
        $this->messageResourceFactoryMock->expects($this->once())
                                         ->willReturn($this->messageResourceMock);
// To this
        $this->messageResourceFactoryMock->expects($this->once())
                                         ->method('create')
                                         ->willReturn($this->messageResourceMock);

More than one node matching the query

Full error message:

Magento\Framework\Exception\LocalizedException : More than one node matching the query: /config/extension_attributes[@for='Magento\Catalog\Api\Data\ProductInterface']/attribute[@code='test_stock_item']/join/field, Xml is: <?xml version="1.0"?> // ... truncated

This error happened when running integration tests and was causing them to fail.

The error message in this case had something to do with the dev/tests/integration/_files/Magento/TestModuleExtensionAttributes module, which is installed automatically during the setup for integration tests. Commenting out the contents of registration.php prevented Magento from installing the module and prevented the error from occurring. Why this error occurs is beyond me, though.

Magento tries to install the same module twice when running integration tests

When running integration tests, Magento is tries to install a module twice, which produces an Integrity Constraint error:

[Progress: 144 / 713]
Module 'PavingDirect_AddToBasketModalCrossSellsStatusFilter':
...
[Progress: 497 / 713]
Module 'PavingDirect_AddToBasketModalCrossSellsStatusFilter':
  [Magento\Framework\DB\Adapter\DuplicateException]
  SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'PavingDirect_AddToBasketModalCrossSellsStatusFilte' for key 'PRIMARY', query was: INSERT INTO `setup_module` (`module`, `data_version`) VALUES (?, ?)

This can happen if the module name is too long. Magento 2 will happily enable a 51-character long module name, but the setup_module db table truncates module names at 50 characters (note the truncated module name PavingDirect_AddToBasketModalCrossSellsStatusFilte). So there ends up being a discrepancy between the module name in the code and the database.

The solution is to try a module name less than 50 characters.

'No tests executed!', a.k.a. PHPUnit runs successfully, but no tests were executed

Given a test suite definition in phpunit.xml:

<!-- Test suites definition -->
<testsuites>
    <testsuite name="MooreLarge_FeedImportBase Integration Tests">
        <directory suffix="Test.php">../../../app/code/MooreLarge/FeedImportBase/Test/Integration</directory>
    </testsuite>
</testsuites>

Verify that:

Do no include slashes or asterisks at the end of the directory path
Verify the path is correct. It is relative to the location of the phpunit.xml file location
Copy the test suite name from the testsuite node into the PhpStorm configuration, to make absolutely sure there is no chance of typos or spaces being accidentally introduced
The test classes in the directory share the same suffix as define in the directory node (usually Test.php)

Setting 'TESTS_GLOBAL_CONFIG_FILE' specifies the non-existing file ''.

You have most likely saved a copy of phpunit.xml.dist and started editing it.

Unfortunately, Magento (or PHPUnit) does not merge your copy with the dist copy, so you most likely removed (or commented) the line which defines the TESTS_GLOBAL_CONFIG_FILE variable.

The solution, then, is to simply re-instate it.

When running tests in PhpStorm: 'PHP Fatal error: Class 'PHPUnit_TextUI_ResultPrinter' not found in /tmp/ide-phpunit.php on line 231'

Verify that PHPUnit is a dependency of your project.

When running integration tests on 2.2.8, the following error occurs when trying to run the setup:install command: `SQLSTATE[42S02]: Base table or view not found: 1146 Table 'wam_pavingdirect_intgrtn_tests.store_website' doesn't exist, query was: SELECT` store_website`.* FROM` store_website``

There is an extension which has an install script which is trying to do something before the database is ready, e.g. Adding an attribute to an attribute set which does not exist yet. Disabling the module should prevent the error from re-occurring.

To find the offending module, try enabling the query log with stack traces:

./n98-magerun.phar dev:query-log:enable

Alternatively, you can use the following tip to progressively disable modules until you find the culprit:

Copy the original dev/tests/integration/framework/bootstrap.php to dev/tests/integration/etc/bootstrap.php and edit as follows:

<?php

use Magento\Framework\Autoload\AutoloaderRegistry;

require_once __DIR__ . '/../../../../app/bootstrap.php';
require_once __DIR__ . '/../framework/autoload.php';

$testsBaseDir   = dirname(__DIR__);
$fixtureBaseDir = $testsBaseDir . '/testsuite';

if (!defined('TESTS_TEMP_DIR')) {
    define('TESTS_TEMP_DIR', $testsBaseDir . '/tmp');
}

if (!defined('INTEGRATION_TESTS_DIR')) {
    define('INTEGRATION_TESTS_DIR', $testsBaseDir);
}

$testFrameworkDir = __DIR__;
require_once __DIR__ . '/../framework/deployTestModules.php';

try {
    setCustomErrorHandler();

    /* Bootstrap the application */
    $settings = new \Magento\TestFramework\Bootstrap\Settings($testsBaseDir, get_defined_constants());

    if ($settings->get('TESTS_EXTRA_VERBOSE_LOG')) {
        $filesystem       = new \Magento\Framework\Filesystem\Driver\File();
        $exceptionHandler = new \Magento\Framework\Logger\Handler\Exception($filesystem);
        $loggerHandlers   = [
            'system' => new \Magento\Framework\Logger\Handler\System($filesystem, $exceptionHandler),
            'debug'  => new \Magento\Framework\Logger\Handler\Debug($filesystem),
        ];
        $shell            = new \Magento\Framework\Shell(
            new \Magento\Framework\Shell\CommandRenderer(),
            new \Monolog\Logger('main', $loggerHandlers)
        );
    } else {
        $shell = new \Magento\Framework\Shell(new \Magento\Framework\Shell\CommandRenderer());
    }

    $installConfigFile = $settings->getAsConfigFile('TESTS_INSTALL_CONFIG_FILE');
    if (!file_exists($installConfigFile)) {
        $installConfigFile .= '.dist';
    }
    $globalConfigFile = $settings->getAsConfigFile('TESTS_GLOBAL_CONFIG_FILE');
    if (!file_exists($globalConfigFile)) {
        $globalConfigFile .= '.dist';
    }
    $sandboxUniqueId = md5(sha1_file($installConfigFile));
    $installDir      = TESTS_TEMP_DIR . "/sandbox-{$settings->get('TESTS_PARALLEL_THREAD', 0)}-{$sandboxUniqueId}";
    
    // Edits start here: Manipulate existing application class to inject the projects' config.php:
    $application = new class(
        $shell,
        $installDir,
        $installConfigFile,
        $globalConfigFile,
        $settings->get('TESTS_GLOBAL_CONFIG_DIR'),
        $settings->get('TESTS_MAGENTO_MODE'),
        AutoloaderRegistry::getAutoloader(),
        true
    ) extends \Magento\TestFramework\Application {
        /**
         * @inheritDoc
         */
        public function install($cleanup)
        {
            $this->_ensureDirExists($this->installDir);
            $this->_ensureDirExists($this->_configDir);

            $file       = $this->_globalConfigDir . '/config.php';
            $targetFile = $this->_configDir . str_replace($this->_globalConfigDir, '', $file);

            $this->_ensureDirExists(dirname($targetFile));
            if ($file !== $targetFile) {
                copy($file, $targetFile);
            }

            parent::install($cleanup);
        }

        /**
         * @inheritDoc
         */
        public function isInstalled()
        {
            // Always return false, otherwise DB credentials will be empty
            return false;
        }
    };
    // Edits end here

    $bootstrap = new \Magento\TestFramework\Bootstrap(
        $settings,
        new \Magento\TestFramework\Bootstrap\Environment(),
        new \Magento\TestFramework\Bootstrap\DocBlock("{$testsBaseDir}/testsuite"),
        new \Magento\TestFramework\Bootstrap\Profiler(new \Magento\Framework\Profiler\Driver\Standard()),
        $shell,
        $application,
        new \Magento\TestFramework\Bootstrap\MemoryFactory($shell)
    );
    $bootstrap->runBootstrap();
    if ($settings->getAsBoolean('TESTS_CLEANUP')) {
        $application->cleanup();
    }
    if (!$application->isInstalled()) {
        $application->install($settings->getAsBoolean('TESTS_CLEANUP'));
    }
    $application->initialize([]);

    \Magento\TestFramework\Helper\Bootstrap::setInstance(new \Magento\TestFramework\Helper\Bootstrap($bootstrap));

    $dirSearch        = \Magento\TestFramework\Helper\Bootstrap::getObjectManager()
                                                               ->create(\Magento\Framework\Component\DirSearch::class);
    $themePackageList = \Magento\TestFramework\Helper\Bootstrap::getObjectManager()
                                                               ->create(\Magento\Framework\View\Design\Theme\ThemePackageList::class);
    \Magento\Framework\App\Utility\Files::setInstance(
        new Magento\Framework\App\Utility\Files(
            new \Magento\Framework\Component\ComponentRegistrar(),
            $dirSearch,
            $themePackageList
        )
    );

    /* Unset declared global variables to release the PHPUnit from maintaining their values between tests */
    unset($testsBaseDir, $logWriter, $settings, $shell, $application, $bootstrap);
} catch (\Exception $e) {
    echo $e . PHP_EOL;
    exit(1);
}

/**
 * Set custom error handler
 */
function setCustomErrorHandler()
{
    set_error_handler(
        function ($errNo, $errStr, $errFile, $errLine) {
            if (error_reporting()) {
                $errorNames = [
                    E_ERROR             => 'Error',
                    E_WARNING           => 'Warning',
                    E_PARSE             => 'Parse',
                    E_NOTICE            => 'Notice',
                    E_CORE_ERROR        => 'Core Error',
                    E_CORE_WARNING      => 'Core Warning',
                    E_COMPILE_ERROR     => 'Compile Error',
                    E_COMPILE_WARNING   => 'Compile Warning',
                    E_USER_ERROR        => 'User Error',
                    E_USER_WARNING      => 'User Warning',
                    E_USER_NOTICE       => 'User Notice',
                    E_STRICT            => 'Strict',
                    E_RECOVERABLE_ERROR => 'Recoverable Error',
                    E_DEPRECATED        => 'Deprecated',
                    E_USER_DEPRECATED   => 'User Deprecated',
                ];

                $errName = isset($errorNames[$errNo]) ? $errorNames[$errNo] : "";

                throw new \PHPUnit\Framework\Exception(
                    sprintf("%s: %s in %s:%s.", $errName, $errStr, $errFile, $errLine),
                    $errNo
                );
            }
        }
    );
}

Finally, update your dev/tests/integration/phpunit.xml to use the new bootstrap.php file:

<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:noNamespaceSchemaLocation="http://schema.phpunit.de/6.2/phpunit.xsd"
         colors="true"
         columns="max"
         beStrictAboutTestsThatDoNotTestAnything="false"
         bootstrap="./etc/bootstrap.php"
         stderr="true"
>

You should now be able to disable modules using bin/magento module:disable and the Magento 2 Integration Test Framework should respect that.

Source: https://magento.stackexchange.com/a/225931

Note: It is best to change back to using ./framework/bootstrap.php once you've finished debugging, otherwise it has been known to cause problems when used in conjunction with the TESTS_CLEANUP=disabled option.

These modules have been known to cause problems with integration tests in Magento 2.2.8:

Firebear_ImportExport
Magebees_QuotationManagerPro
Amasty_Rolepermissions
PavingDirect_PatternFile
Magento_Braintree

Which can be disabled by:

pdmr mod:dis Firebear_ImportExport PavingDirect_PatternFile Magebees_QuotationManagerPro Amasty_Rolepermissions Magento_Braintree && pdreset

If disabling the module is not an option, then you may be able to change the order the modules are loaded by Magento by declaring the module as a dependency of another module (even a dummy module, which does nothing except act as a receptacle for the dependency definition), so that the module is loaded later on. This may solve the problem, depending on what the offending module is trying to do.

`There are no commands defined in the "setup" namespace.`

Try setting the TESTS_CLEANUP variable to disabled in phpunit.xml:

// File: shared/webroot/dev/tests/integration/phpunit.xml
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:noNamespaceSchemaLocation="http://schema.phpunit.de/4.1/phpunit.xsd"
         colors="true"
         bootstrap="./framework/bootstrap.php"
>
    <php>
        <const name="TESTS_CLEANUP" value="disabled"/>
        ...

If that doesn't work, it's an indication that there is an error somewhere/exception being thrown and not caught which is preventing the CLI tool from executing properly.

Try running the same command as sudo or as the Magento file system owner:

sudo -u magento2 <command>

`PHP Fatal error: Class 'PHPUnit_Framework_TestCase' not found in app/code/Vendor/Namespace/Test/Integration/ModuleTest.php on line X`

This is usually caused by running a verison of PHPUnit which does not support the old, underscore namespacing style. Change the class name to use the PHP 5.3+ style with backslashes, i.e. Change PHPUnit_Framework_TestCase to PHPUnit\Framework\TestCase. If you can't change the class name, you'll just have to downgrade PHPUnit to PHPUnit 5.7.0 instead.

`PHP Fatal error: Interface 'PHPUnit\Framework\TestListener' not found in /var/www/html/dfl/htdocs/dev/tests/integration/framework/Magento/TestFramework/Event/PhpUnit.php`

Your PHPUnit version is out of date. A test is referencing a class which doesn't exist in the version of PHPUnit you're version. Try upgrading PHPUnit to at least 6.2.0.

`PHP Fatal error: Class 'Magento\TestFramework\ObjectManager' not found`

Check that you don't have a default phpunit.xml or bootstrap.php file configured in PhpStorm settings. If you have, disable them and specify the phpunit.xml in the Test Case Configuration screen.

`Magento\Framework\Exception\LocalizedException: Setting 'TESTS_INSTALL_CONFIG_FILE' specifies the non-existing file ''. in /var/www/html/dfl/htdocs/dev/tests/integration/framework/Magento/TestFramework/Bootstrap/Settings.php`

Verify that PHPUnit is using the phpunit.xml file you think it is. You can force PHPUnit to use a specific file with the -c flag.
Verify that the constant actually has a value in the phpunit.xml file.
Verify that you are using a version of PHPUnit which is compatible with the tests you are trying to run.

`Trying to configure method "getAbsoluteFilename" which cannot be configured because it does not exist, has not been specified, is final, or is static`

The method getAbsoluteFilename is not defined in the class.
Is the method name getAbsoluteFilename definitely correct?
Double and triple check that the method getAbsoluteFilename absolutely does exist before moving onto other debugging steps

SQLSTATE[42S22]: Column not found: 1054 Unknown column 'e.customer_type' in 'field list'

Module 'Magento_Customer':
Running data recurring...
In Mysql.php line 110:

[Zend_Db_Statement_Exception (42)]
SQLSTATE[42S22]: Column not found: 1054 Unknown column 'e.customer_type' in 'field list', query was: SELECT `e`.`entity_id`, TRIM(CONCAT_WS(' ', IF(`e`.`prefix` <> '', `e`.`prefix`, NULL), IF(`e`.`firstname` <> '', `e`.`firstname`, NULL), IF(`e`.`middlename` <> '', `e`.`middlename`, NULL), IF(`e`.`lastname` <> '', `e`.`lastname`, NULL), IF(`e`.`suffix` <> '', `e`.`suffix`, NULL))) AS `name`, `e`.`email`, `e`.`group_id`, `e`.`created_at`, `e`.`website_id`, `e`.`confirmation`, `e`.`created_in`, `e`.`dob`, `e`.`gender`, `e`.`taxvat`, `e`.`lock_expires`, `e`.`customer_type`, TRIM(CONCAT_WS(' ', IF(`shipping`.`street` <> '', `shipping`.`street`, NULL), IF(`shipping`.`city` <> '', `shipping`.`city`, NULL), IF(`shipping`.`region` <> '', `shipping`.`region`, NULL), IF(`shipping`.`postcode` <> '', `shipping`.`postcode`, NULL))) AS `shipping_full`, TRIM(CONCAT_WS(' ', IF(`billing`.`street` <> '', `billing`.`street`, NULL), IF(`billing`.`city` <> '', `billing`.`city`, NULL), IF(`billing`.`region` <> '', `billing`.`region`, NULL), IF(`billing`.`postcode` <> '', `billing`.`postcode`, NULL))) AS `billing_full`, `billing`.`firstname` AS `billing_firstname`, `billing`.`lastname` AS `billing_lastname`, `billing`.`telephone` AS `billing_telephone`, `billing`.`postcode` AS `billing_postcode`, `billing`.`country_id` AS `billing_country_id`, `billing`.`region` AS `billing_region`, `billing`.`street` AS `billing_street`, `billing`.`city` AS `billing_city`, `billing`.`fax` AS `billing_fax`, `billing`.`vat_id` AS `billing_vat_id`, `billing`.`company` AS `billing_company` FROM `customer_entity` AS `e`
LEFT JOIN `customer_address_entity` AS `shipping` ON shipping.entity_id=e.default_shipping
LEFT JOIN `customer_address_entity` AS `billing` ON billing.entity_id=e.default_billing

There is something in the code trying to use the customer_type field from Magento_Company. Presumably the Magento_Company module is not enabled when the integration test runs.

Integration tests not behaving as expected

Remember to clear the integration test cache if you've disabled the TESTS_CLEANUP environment variable:

project8@project8-aurora-r5:/var/www/vhosts/magento2.localhost.com$ rm -rf dev/tests/integration/tmp/sandbox-*

Writing a module from scratch using TDD

This assumes that the Magento 2 test framework (including integration tests) and your IDE are already setup and configured to run tests.
- Refer to the DevDocs for a quick guide on setting up integration tests [3]3 and on setting up PhpStorm with PHPUnit [4]4
Start with Integration tests first.
Manually create the following folder structure module in the app/code directory:

app
    code
        [Vendor Name]
            [Module Name]
                Test
                    Integration

Create your first test class and a 'test nothing' method. We'll use this empty test to check our framework and IDE are setup correctly:

<?php

namespace Mage2Kata\ModuleSkeleton\Test\Integration;

class SkeletonModuleConfigTest extends \PHPUnit\Framework\TestCase
{
	public function testNothing()
	{
		$this->markTestSkipped('Testing that PhpStorm and test framework is setup correctly');
	}
}

The next step is to write the next most basic test: To check that the module exists according to Magento.

In other words, we test for the existence of the registration.php file:

private $moduleName = 'Mage2Kata_SkeletonModule';

public function testTheModuleIsRegistered()
{
    $registrar = new ComponentRegistrar();
    $this->assertArrayHasKey(
        $this->moduleName,
        $registrar->getPaths( ComponentRegistrar::MODULE)
    );
}

We've extracted the module name into a member variable so we can re-use it in other tests.

At this point you may get an error if you try to run the test. This is because Magento expects every module to have, at the bare minimum, a registration.php file and a module.xml file with a setup_version attribute.

Let's now move onto the next step in creating a module - the module.xml.

Here's the test:

public function testTheModuleIsConfiguredAndEnabled()
{
    /** @var ObjectManager $objectManager */
    $objectManager = ObjectManager::getInstance();

    /** @var ModuleList $moduleList */
    $moduleList = $objectManager->create( ModuleList::class);

    $this->assertTrue( $moduleList->has( $this->moduleName), 'The module is not enabled');
}

If we run this test now, it will fail. If we create the module.xml file, then it should pass.

// File: Mage2Kata/SkeletonModule/etc/module.xml
        <?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework:Module/etc/module.xsd">
    <module name="Mage2Kata_SkeletonModule" setup_version="0.1.0">
    </module>
</config>

So that's our extremely basic module created using TDD.

Examples

Module setup

Tests for the existence of a registration.php file

Type: Integration.

    /**
     * Tests for the existence of a registration.php file
     */
    public function testTheModuleIsRegistered()
    {
        $registrar = new \Magento\Framework\Component\ComponentRegistrar();
        $this->assertArrayHasKey(
            $this->moduleName,
            $registrar->getPaths(\Magento\Framework\Component\ComponentRegistrar::MODULE),
            'Module is not registered with Magento 2. Does registration.php exist?'
        );
    }

Test that the module is enabled

Type: Integration.

    public function testTheModuleIsConfiguredAndEnabledInTheTestEnvironment()
    {
        /** @var \Magento\TestFramework\ObjectManager $objectManager */
        $objectManager = \Magento\TestFramework\ObjectManager::getInstance();

        /** @var \Magento\Framework\Module\ModuleList $moduleList */
        $moduleList = $objectManager->create(\Magento\Framework\Module\ModuleList::class);

        $this->assertTrue(
            $moduleList->has($this->moduleName),
            'The module is not enabled in the test environment'
        );
    }

Events

What you should test

Test that the config XML has been defined correctly (integration)
Test the logic in the observer class (unit/integration)

Resources:

See Mage2Katas: 13. The Event Observer Kata

Testing the configuration is correct

@todo

Framework

Set the area code in a unit test

Example taken from https://github.com/ProcessEight/Mage2Katas/blob/mage2katas-plugin-config/app/code/Mage2Kata/Interceptor/Test/Integration/Plugin/CustomerRepositoryPluginTest.php

	/**
	 * @param string $areaCode
	 */
	protected function setArea( $areaCode )
	{
        /** @var \Magento\TestFramework\ObjectManager $objectManager */
        $objectManager = \Magento\TestFramework\ObjectManager::getInstance();

		/** @var \Magento\TestFramework\App\State $appArea */
		$appArea = $objectManager->get( \Magento\TestFramework\App\State::class );
		$appArea->setAreaCode( $areaCode );
	}

	public function testTheModuleDoesNotInterceptCallsToTheCustomerRepositoryInGlobalScope( )
	{
		$this->setArea( Area::AREA_GLOBAL);
		// ... test continues
    }
    
	protected function tearDown()
	{
		$this->setArea( null);
	}

Plugins

What you should test

Test that the config XML has been defined correctly (integration)
Test the logic in the plugin class (unit/integration)

Test that a module is defined correctly in the global scope

Type: Integration.

    /**
     * Test assumes that di.xml is in Vendor/Module/etc/di.xml
     */
    public function testThatThePluginConfigXmlIsDefinedCorrectlyInGlobalScope()
    {
        /** @var \Magento\TestFramework\ObjectManager $objectManager */
        $objectManager = \Magento\TestFramework\ObjectManager::getInstance();

        $pluginList = $objectManager->create(\Magento\TestFramework\Interception\PluginList::class);
        $pluginInfo = $pluginList->get(\Magento\Customer\Api\CustomerRepositoryInterface::class, []);

        $this->assertArrayHasKey(
            'vendor_module',
            $pluginInfo,
            'Plugin name attribute was not found in global config XML scope. Does it exist in the etc/di.xml file?'
        );
        $this->assertSame(
            \Vendor\Module\Plugin\CustomerRepositoryPlugin::class,
            $pluginInfo['vendor_module']['instance'],
            'Plugin class attribute value did not match expected type. Verify it is the correct value in the etc/di.xml file.'
        );
    }

Routes

Tests that Magento has read the routes.xml and added the custom router to its list of routes

Type: Integration.

    /**
     * Assert that the route has been configured correctly in routes.xml
     * 
     * IMPORTANT: Area code was set to 'adminhtml' in setup method and reset to 'null' in teardown method 
     * 'dpdcsv' is the frontName in this example 
     *
     * @magentoAppArea adminhtml
     */
    public function testDpdcsvRouteIsConfigured()
    {
        /** @var \Magento\Framework\App\Route\ConfigInterface $routeConfig */
        $routeConfig = $this->objectManager->create(\Magento\Framework\App\Route\ConfigInterface::class);
        $this->assertContains(
            'PavingDirect_DpdSalesOrderCsvExport',
            $routeConfig->getModulesByFrontName('dpdcsv', 'adminhtml'),
            'Could not find module which defines the frontName dpdcsv - is the module enabled?'
        );
    }

Tests that the route is working correctly by asking Magento to match the custom route to the correct custom controller

Type: Integration.

    /**
     * Assert that a request for the route can be matched to the correct controller
     *
     * IMPORTANT: Area code was set to 'adminhtml' in setup method and reset to 'null' in teardown method 
     * 'dpdcsv' is the frontName in this example 
     *
     * @throws \Magento\Framework\Exception\LocalizedException
     */
    public function testGridToDpdCsvActionControllerIsFound()
    {
        // Mock the request object
        /** @var Request $request */
        $request = $this->objectManager->create(Request::class);
        $request->setModuleName('dpdcsv')
                ->setControllerName('export')
                ->setActionName('gridToDpdCSV');

        // Ask the \Magento\Backend\App\Router class to match our mock request to our controller action class
        /** @var \Magento\Backend\App\Router $backendRouter */
        $backendRouter  = $this->objectManager->create(\Magento\Backend\App\Router::class);
        $expectedAction = \PavingDirect\DpdSalesOrderCsvExport\Controller\Adminhtml\Export\GridToDpdCsv::class;
        $this->assertInstanceOf(
            $expectedAction,
            $backendRouter->match($request),
            'Magento could not match the request to the controller. Is the controller action class in the right location?'
        );
    }

Controllers

Test that a GET controller can accept GET requests (i.e. Test our custom controller is correctly defined)

    /**
     * Test that we can actually load the controller action
     */
    public function testCanHandleGetRequests()
    {
        $this->getRequest()->setMethod(\Magento\TestFramework\Request::METHOD_GET);
        // Note that the backend-frontname is 'backend' here (rather than, say, 'admin') because that's what's defined in html/dev/tests/integration/etc/install-config-mysql.php.dist
        $this->dispatch('backend/feedreporter/message/delete');
        // After executing, this particular controller redirects back to an admin grid
        $this->assertSame(302, $this->getResponse()->getHttpResponseCode());
    }

Test that a GET controller is unable to accept POST requests

    /**
     * Test that we can only make GET requests to GET controller action
     */
    public function testCannotHandlePostRequests()
    {
        $this->getRequest()->setMethod(\Magento\TestFramework\Request::METHOD_POST);
        $this->dispatch('backend/feedreporter/message/delete');
        $this->assertSame(404, $this->getResponse()->getHttpResponseCode());
    }

khoimm92/Testing in Magento 2.md

Testing in Magento 2

Table of Contents

Quick start quide to testing existing classes

Environment setup

Configuring PHPUnit

Configuring PhpStorm

Tell PhpStorm which version of PHPUnit it should use

Tell PhpStorm how to run the tests

Configure the database (for integration tests)

Unit tests

Writing tests

Data Providers: Passing data into tests

Testing Exceptions

Fixtures

Best practices

Integration or unit test?

Object Manager

Using Stubs

Using Mocks

Running unit tests from the CLI

Executing unit tests from outside Docker

Executing unit tests from inside Docker

Running unit tests from PhpStorm

With Docker

Without Docker

Troubleshooting

Exception: Could not connect to the Amqp Server

An expectation has not been met, even if you've stepped through the code under test and verified it has been called

Error : Class 'Magento\TestFramework\Helper\Bootstrap' not found

'No method matcher is set'

More than one node matching the query

Magento tries to install the same module twice when running integration tests

'No tests executed!', a.k.a. PHPUnit runs successfully, but no tests were executed

Setting 'TESTS_GLOBAL_CONFIG_FILE' specifies the non-existing file ''.

When running tests in PhpStorm: 'PHP Fatal error: Class 'PHPUnit_TextUI_ResultPrinter' not found in /tmp/ide-phpunit.php on line 231'

When running integration tests on 2.2.8, the following error occurs when trying to run the setup:install command: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'wam_pavingdirect_intgrtn_tests.store_website' doesn't exist, query was: SELECT store_website.* FROM store_website``

There are no commands defined in the "setup" namespace.

PHP Fatal error: Class 'PHPUnit_Framework_TestCase' not found in app/code/Vendor/Namespace/Test/Integration/ModuleTest.php on line X

PHP Fatal error: Interface 'PHPUnit\Framework\TestListener' not found in /var/www/html/dfl/htdocs/dev/tests/integration/framework/Magento/TestFramework/Event/PhpUnit.php

PHP Fatal error: Class 'Magento\TestFramework\ObjectManager' not found

Magento\Framework\Exception\LocalizedException: Setting 'TESTS_INSTALL_CONFIG_FILE' specifies the non-existing file ''. in /var/www/html/dfl/htdocs/dev/tests/integration/framework/Magento/TestFramework/Bootstrap/Settings.php

Trying to configure method "getAbsoluteFilename" which cannot be configured because it does not exist, has not been specified, is final, or is static

SQLSTATE[42S22]: Column not found: 1054 Unknown column 'e.customer_type' in 'field list'

Integration tests not behaving as expected

Writing a module from scratch using TDD

Examples

Module setup

Tests for the existence of a registration.php file

Test that the module is enabled

Events

What you should test

Resources:

Testing the configuration is correct

Framework

Set the area code in a unit test

Plugins

What you should test

Test that a module is defined correctly in the global scope

Routes

Tests that Magento has read the routes.xml and added the custom router to its list of routes

Tests that the route is working correctly by asking Magento to match the custom route to the correct custom controller

Controllers

Test that a GET controller can accept GET requests (i.e. Test our custom controller is correctly defined)

Test that a GET controller is unable to accept POST requests

Sources

When running integration tests on 2.2.8, the following error occurs when trying to run the setup:install command: `SQLSTATE[42S02]: Base table or view not found: 1146 Table 'wam_pavingdirect_intgrtn_tests.store_website' doesn't exist, query was: SELECT` store_website`.* FROM` store_website``

`There are no commands defined in the "setup" namespace.`

`PHP Fatal error: Class 'PHPUnit_Framework_TestCase' not found in app/code/Vendor/Namespace/Test/Integration/ModuleTest.php on line X`

`PHP Fatal error: Interface 'PHPUnit\Framework\TestListener' not found in /var/www/html/dfl/htdocs/dev/tests/integration/framework/Magento/TestFramework/Event/PhpUnit.php`

`PHP Fatal error: Class 'Magento\TestFramework\ObjectManager' not found`

`Magento\Framework\Exception\LocalizedException: Setting 'TESTS_INSTALL_CONFIG_FILE' specifies the non-existing file ''. in /var/www/html/dfl/htdocs/dev/tests/integration/framework/Magento/TestFramework/Bootstrap/Settings.php`

`Trying to configure method "getAbsoluteFilename" which cannot be configured because it does not exist, has not been specified, is final, or is static`