Web Platform Tests

Update metadata

./mach wpt-update /Users/jdescottes/Development/wpt-update/

After downloading the relevant wpt-report.json files from try. Always try to take a variety of platforms (eg 1 linux 1 android, or more generally one which fails for specific conditions and a successful one).

View WPT tests differences

For local tests you can run with --debug-test for try pushes there's a button on treeherder in the bottom panel

Testing

tldr:

> mt test_path.py --setpref marionette.actors.enabled=true -vv --gecko-log -

Fission and useActors

To enable code paths guarded by MarionettePrefs.useActors pass --setpref marionette.actors.enabled=true when running a test.

We are not running with marionette.actors.enabled on CI yet, so we have to check this locally.

Enable logs

By default all logs are hidden. Pass -vv --gecko-log - to enable logging. logger.info/warn/trace/... statements in the code should now show up in the logs.

For webplatform tests, you need to pass --webdriver-arg=-vv.

Workaround missing commands

The executeScript command is not working yet with actors, but a lot of tests rely on it. They use it to gather "expected" values (example https://searchfox.org/mozilla-central/rev/30e70f2fe80c97bfbfcd975e68538cefd7f58b2a/testing/marionette/harness/marionette_harness/tests/unit/test_chrome_element_css.py#15)

For such tests, simply comment out the execute_script call and find the expected value "manually". Eg. run the test without marionette.actors.enabled, check the value, and then hardcode it in the test (or be creative).

Run wpt tests on android

./mach test TESTPATH --log-mach-level debug --webdriver-binary target/debug/geckodriver

./mach test TESTPATH --webdriver-binary target/debug/geckodriver --log-mach-level debug --webdriver-arg=-vv

Run wpt tests on safari

./mach wpt TESTPATH --product=safari --webdriver-binary=/usr/bin/safaridriver

Run wpt reftests using Marionette

./mach wpt testing/web-platform/tests/css/CSS2/colors/color-001.xht --test-types reftest  --setpref marionette.log.level=Trace

./mach wpt testing/web-platform/tests/css/css-text/writing-system/ --test-types reftest --setpref marionette.log.level=Trace --repeat 10

Other relevant test harnesses

Use debugger with marionette harness unit tests

Pass --app-arg=--jsdebugger & --app-arg=--wait-for-jsdebugger (--app-arg will forward arguments to the APP, here firefox).

mt testing/marionette/harness/marionette_harness/tests/unit/test_reftest.py --setpref marionette.actors.enabled=true -vv --gecko-log - --app-arg=--jsdebugger --app-arg=--wait-for-jsdebugger

Wait in python tests

import time

time.sleep(1)

Build and copy geckodriver

Build:

% cd testing/geckodriver
% cargo build

Copy:

% cp target/debug/geckodriver objdir.noindex/dist/Nightly.app/Contents/MacOS/

Tips for TRY pushes

mach try preset

And then ./mach try --preset remote-protocol.

Add WebProgress logs on TRY

See example at https://hg.mozilla.org/try/rev/26060f0e4765228f8ef9c3c408076abeac9bc9ae

Add the following line to testing/mozharness/scripts/web_platform_tests.py

env["MOZ_LOG"] = "BCWebProgress:5"

Profiler

./mach try fuzzy --rebuild 10 --env MOZ_PROFILER_STARTUP=1 --env MOZ_PROFILER_SHUTDOWN=/builds/worker/workspace/build/blobber_upload_dir/profile.json

Add logs to WPT tests on try

Add - --setpref=marionette.log.level=trace in taskcluster/ci/test/web-platform.yml at https://searchfox.org/mozilla-central/rev/71621bfa47a371f2b1ccfd33c704913124afb933/taskcluster/ci/test/web-platform.yml#89

Add WebProgress logs on TRY (android)

For Android, the environment above is not forwarded. Consequently you need to either set the capability via a fixture:

@pytest.mark.capabilities({
    "acceptInsecureCerts": True,
    "moz:firefoxOptions": {
        "env": {
            "MOZ_LOG": "BCWebProgress:5"
        }
    }
})

Or in geckodriver directly around https://searchfox.org/mozilla-central/source/testing/geckodriver/src/browser.rs#128

External projects

Testing CDP

clone the CRI at https://github.com/cyrus-and/chrome-remote-interface
install the CRI npm i
start Chrome /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --remote-debugging-port=9222
run the CRI bin/client.js inspect
run individual commands in the prompt, eg Page.close()

Testing Puppeteer

open /Users/juliandescottes/.cache/puppeteer/firefox
PUPPETEER_PRODUCT=firefox npm i
npm run test:firefox:bidi:headful
BINARY=/Users/juliandescottes/Development/mozilla/hg/mozilla-unified/objdir.noindex/dist/Nightly.app/Contents/MacOS/firefox npm run test:firefox:bidi

Testing Playwright

https://github.com/microsoft/playwright/blob/main/tests/bidi/README.md

To run basic tests:

npm run biditest -- --project='bidi-firefox-beta-*' browsercontext-expose-function.spec.ts:21

DEBUG=pw:browser BIDI_FFPATH='/Users/juliandescottes/Development/mozilla/hg/mozilla-unified/objdir.noindex/dist/Nightly.app/Contents/MacOS/firefox' npm run biditest -- --project='moz-firefox-*' --headed

To enable browser logs:

DEBUG=pw:browser

Testing Browsertime

node bin/browsertime.js spa.js -b firefox

juliandescottes/MARIONETTE_NOTES.md