Unit testing: PHPUnit + Cypress

I've been testing my gallery plugin using local and remote URLs.

This seemed reasonable until I realised that it breaks the release process.

This is because the release needs to test an updated environment, but the environment is only updated by the release. So my tests would pass on local dev, but fail when run through the CI, which could only see the remote URL.

Note: I just remembered that there's actually a way to expose a local URL to other internet users, I didn't try this.

How it all fits together

A. Set up

1. Use composer to install WP-CLI and PHPUnit dependencies

wpdtrt-plugin-boilerplate/composer.json:

    "require-dev": {
        "phpunit/phpunit": "^7.5.14",
        "psy/psysh" : "~0.6",
        "wp-coding-standards/wpcs": "^0.14.1",
        "dealerdirect/phpcodesniffer-composer-installer": "^0.4.4",
        "wp-cli/wp-cli": "^2.3"
    },

2. Install a fresh WordPress site

A fresh site prevents wiping the existing database from being wiped out by test data during setup.

Download latest WP from wordpress.org
Unzip to new_folder in websites directory
Edit wp-config.php to add DB user/pass and define( 'DB_HOST', '127.0.0.1' );
Create empty wp_dbname database using SequelPro
Create a virtual host in MAMP Pro
Visit http://virtualhost to run the WordPress 5 minute set up

3. Create a plugin folder

cd new_folder
mkdir wp-content/plugins/my-plugin-name

4. Scaffold the plugin test files

wp scaffold plugin-tests my-plugin-name (using wp-cli/scaffold-command, which can also be used for theme tests). See also https://developer.wordpress.org/cli/commands/scaffold/plugin-tests/.

Success: Created test files.

The following files are created in my-plugin-name/:

./phpunit.xml.dist is the configuration file for PHPUnit.

./travis.yml is the configuration file for Travis CI. Use --ci=<provider> to select a different service.

./bin/install-wp-tests.sh configures the WordPress test suite and a test database.

./tests/bootstrap.php is the file that makes the current plugin active when running the test suite.

./tests/test-sample.php is a sample file containing the actual tests.

./phpcs.xml.dist is a collection of PHP_CodeSniffer rules.

From: wp scaffold plugin-tests

5. Integrate these files

./phpunit.xml.dist is merged into my file of the same name
./travis.yml is merged into my CI script
./bin/install-wp-tests.sh is called by gulp-modules/dependencies.js, customised to pass environmental variables in from my bash_profile
./tests/bootstrap.php is loaded via phpunit.dist.xml, customised for my use case.
./tests/test-sample.php is merged into my test script.
./phpcs.xml.dist is merged into my file of the same name

6. Commit `bin/install-wp-tests.sh` to the plugin repo

e.g. wpdtrt-plugin-boilerplate/bin

B. Running tests

1. Gulp runs install-wp-tests.sh

yarn run dependencies

This calls:

gulp-modules/dependencies.js:

async function wpUnit() {
    // ...
    // bootstrap.php checks for bin/install-wp-tests.sh
    // and throws an error if it is not found.
    // ./bin/install-wp-tests.sh is stored in wpdtrt-plugin-boilerplate
    const dbName = `${pluginNameSafe}_wpunit_${Date.now()}`;
    const wpVersion = 'latest';
    let installerPath = 'bin/';

    if ( boilerplatePath().length ) {
        installerPath = `${boilerplatePath()}bin/`;
    }

    const { stdout, stderr } = await exec( `bash ${installerPath}install-wp-tests.sh ${dbName} ${wpVersion}` );
    // ...
}

2. install-wp-tests.sh creates a WordPress site and installs the testing framework

downloads the latest version of WordPress from https://wordpress.org/nightly-builds/wordpress-latest.zip
downloads wp-mysqli from the Github repo
clones the testing suite (wordpress-tests-lib) from develop.svn.wordpress.org to the user's temporary folder (see below)
sets WP_TESTS_DIR and WP_CORE_DIR
customises wp-tests-config-sample.php and outputs it to wordpress-tests-lib/wp-tests-config.php
creates a separate database for running unit tests

This dynamic instance of WordPress allow plugins to be tested in isolation of a standard WordPress folder.

`install-wp-tests.sh` vs Cypress

install-wp-tests.sh uses sed (a special editor for modifying files automatically) to replace the following placeholders in wp-tests-config-sample.php:

define( 'DB_NAME', 'youremptytestdbnamehere' );
define( 'DB_USER', 'yourusernamehere' );
define( 'DB_PASSWORD', 'yourpasswordhere' );

However it appears to hardcode the values which control which domain is used to host the temporary WordPress site which exists while tests are running.

This is problematic, as the generated code assigns the domain of example.org to the dummy site.

define( 'WP_TESTS_DOMAIN', 'example.org' );
define( 'WP_TESTS_EMAIL', '[email protected]' );

example.org is not a real domain, so Cypress cannot visit it to start the end-to-end tests.

Possible solutions

override the constant with some kind of hack
redirect example.org to ? using hosts file
use my own wp-tests-config.php somehow
buy example.org (joke)

Note that I've already modified install-wp-tests.sh, to use the following environmental variables:

WPUNIT_DB_USER
WPUNIT_DB_PASS
WPUNIT_DB_HOST

This change was made, to allow local dev and Travis to use different values here (via bash_profile and Travis settings), without exposing these in .yo-rc.json or on the command line (Travis build logs are public for open source projects).

It's therefore feasible that I could edit this file again. I could potentially follow the developer's lead and use sed calls to change out example.org for something more practical.

Make WordPress Core ticket #34000 suggests that WP_TEST_DOMAIN should be configurable.

And there's this:

wp-cli/wp-cli/features/bootstrap.feature:

    # Use try to cater for wp-db errors in old WPs.
    When I try `wp core install --url=example.com --title=example --admin_user=example [email protected]`
    Then STDOUT should contain:
      """
      Success:
      """
    And the return code should be 0

I also tried using the optional wp scaffold --url flag to set a different domain, but the generated wp-tests-config.php file did not reflect this change.

What are `WP_TESTS_DIR`, `WP_CORE_DIR` and `TMPDIR`?

TMPDIR=${TMPDIR-/tmp}
TMPDIR=$(echo $TMPDIR | sed -e "s/\/$//")
WP_TESTS_DIR=${WP_TESTS_DIR-$TMPDIR/wordpress-tests-lib}
WP_CORE_DIR=${WP_CORE_DIR-$TMPDIR/wordpress/}

I couldn't track down where the value of WP_TESTS_DIR was set (and didn't look for WP_CORE_DIR).

However, it seems that there were ignored anyway, with the resultant paths only featuring $TMPDIR:

$WP_TESTS_DIR: /var/folders/0y/xxxxxxxx/T/wordpress-tests-lib
$TMPDIR: /var/folders/0y/xxxxxxxx/T
$WP_CORE_DIR: /var/folders/0y/xxxxxxxx/T/wordpress/

TMPDIR was easier to track down. It is an environmental variable, which is available on MacOS, and used by many apps to store temporary files:

OS X generates a programmatic directory stored in /private/var and defines the $TMPDIR environment variable for locating the system temporary folder.

Using Terminal.app, type echo $TMPDIR or open $TMPDIR (to open Finder on that folder).

From: Where is the temp folder?

echo $TMPDIR; # /var/folders/0y/xxxxxxxxxxxxxx/T

In PHP this is exposed with:

sys_get_temp_dir()

In WP CLI this is in wp-cli/wp-cli/php/utils.php:

Utils\get_temp_dir()

Although there is little in the way of documentation online, it’s clear that the var folder was designed to improve operating security. It was designed to improve permissions (rwxr-xr-x) over previous temp and cache locations, such as the /Library/Caches and /tmp folders. According to Apple developer information, the var folder is a reference to “per-user temporary files and caches.”

From: What is a var folder?

3. Run the tests

yarn run test

This runs gulp-modules/test.js:

const { stdout, stderr } = await exec( './vendor/bin/phpunit --configuration phpunit.dist.xml' );
console.log( stdout );
console.error( stderr );

dotherightthing/unit-testing-phpunit-cypress.md