Nightwatch aims to provide an easy-to-use wrapper around Selenium WebDriver. Even though at the time of writing it's quite outdated, compared to other current alternatives, but it's still OK for simple cases. However, trying to build a large, maintainable set of test suites for modestly sized web app uncovers quite a few shortcomings. This text aims to list them and for each one describe both the issue and some way to work around it.
Note that this text is meant to supplement Nightwatch Developer Guide, not replace it.
Aside from the specific problems listed below, a general drawback of using Nightwatch is the way it is configured. Extensions (e.g. page objects or custom commands) are fed to it through paths, with filenames being used as names of the entities. This obfuscates all dependencies and makes it hard to distribute tooling into separate packages.
The main vehicle for abstractions in Nightwatch are so-called "page objects" and their sections. A page object is meant to encapsulate API for controlling a particular web page. It can have inner sections to separately encapsulate controllers for some clearly delimited parts of the page. Page objects and sections are also where most of the issues with Nightwatch lie - all of the problems described below are in some way tied to page objects.
Nightwatch's API is separated into four groups: BDD-style assertions, regular assertions, Nightwatch commands, and WebDriver commands. While this separation may seem superficial at first glance, it's actually quite important. The important division is into the two following categories:
- high-level (simple) commands - BDD-style assertions and Nightwatch commands - these have simpler contracts and work in context of page objects
- low-level commands - regular assertions and WebDriver commands - these have contracts based on external specs (WebDriver, NodeJS assertions) and work only as globals
Low-level commands impose severe limitations, so it's best to use them only when absolutely necessary. Additionally, only the high-level commands are provided as methods on page objects and their sections.
Page object sections are just like page objects themselves except for a single important differerence -
they have an additional selector
property to target the section's container element. Selectors of
elements
defined inside a section are applied in context of section's container when passed to
high-level commands executed on the section.
When testing a web application rather than a web page, page objects are best used only as containers for sections, so that the controller APIs are restricted to the parts of the application they control and they're similarly reusable.
Nightwatch allows choosing between two "locate strategies" for element selection - css selector
and xpath
.
There's a major difference in how the choice is made between WebDriver commands and the rest of the API. Those
of the former that perform element selection accept locateStrategy
as the first argument. There are only
a few of these as the element selection is separated from other operations, which use the element IDs retrieved
by the few selection methods.
section.element(
'xpath',
'//input[name="useBetterApi"]',
({ value: { ELEMENT } }) => browser.elementIdSelected(
ELEMENT,
({ value: isSelected }) => console.log(isSelected)
)
)
The rest of the API shares a global strategy selection with a configurable default value and two global
methods for switching between the strategies - useXpath
and useCss
. The selection being both global
and private makes using this quite awkward and brittle.
BDD-style assertions support passing locateStrategy
as the second argument (though this is undocumented).
If the second argument is not supplied, they use the global strategy as well.
Page objects and sections support defining elements
to serve as aliases for selectors. The definitions
contain the locateStrategy
tied to each selector and ignore the global selection, so using elements
can alleviate the issues with it. Unfortunately the definitions have to be static, so any dynamic selection,
e.g. selecting items from large set by their labels, has to happen directly through the commands, without
elements
. Still, it's best to use elements
when possible to simplify command code.
In our codebase, we resolved the issue of global private strategy by agreeing on a common default
(css selector
) and a usingXpath
helper that switches to XPath just while running the commands passed
to it.
const selector = `//a[.="${someLabel}"]`;
section.usingXpath(
() => browser
.waitForElementVisible(selector)
.click(selector)
);
This replaces useCss
and useXpath
completely.
Element selectors should be robust enough to survive through minor changes to app layout. To this end, they need to find just the right kind and amount of specificity. The following guidelines help achieve this.
- CSS selectors should be favored over XPath. They're easier to read and allow clean selection by
class name, the most common mode of selection. When using XPath for class selection,
contains(@class, "...")
is the right tool, not exact match (@class="..."
). - Tag names should only be used when necessary and only for meaningful tags, e.g.
button
orinput
, notdiv
orspan
.'.Dialog_container button[type=submit]' // good 'div.Dialog_container h1' // bad, even the heading should be selected by a class if possible
- Exact markup structure should only be used when necessary, e.g. when component nesting is possible.
'.App_main > .HeaderLayout_body', // good, there might be another `HeaderLayout` inside '.App_main > div > .App_sidebar' // bad, needlessly specific structure
Note that these guidelines heavily depend on application markup. In our codebase, application components have unobfuscated class names with component-specific unique prefixes, making class names the ideal method of element selection.
When using XPath selectors in page object section elements
, selectors need to begin with .
to be executed
in context of the section's root element, e.g. .//button[@type=submit]
. Without that the selectors are
executed globally, with no regard to the section context. While this can be detrimental when applied
by mistake, it can be useful to break out of context when trying to select content rendered separately from
the main application body but tied in functionality to the section.
const userManagement = {
sections: {
userSettings: {
selector: '.userSettings',
sections: {
// the drop-down is semantically part of user settings but actually rendered outside the app
userRolesDropdown: {
// selector starting with '//' searches anywhere on the page, even outside the parent section
selector: '//*[contains(@class, ".Dropdown_dropdown")]',
elements: {
// selector starting with './/' searches just inside the section
adminRole: './/*[.="Admin"]'
},
// ...
}
},
// ...
}
}
};
Only the selectors of elements
defined in a page object section are applied in context of the section
(with the exception of global XPath selectors as described above). Selectors passed directly to commands,
even high-level ones, are executed globally. This makes dynamic selection in context of page object sections
impossible, leaving the whole concept of sections severely lacking. Getting around this limitation requires
some serious h4cking (see below).
Nightwatch's API can be extended through "custom commands". When supplied to Nightwatch, these are added both to the global API and to page objects and their sections. However, even when custom commands get called from page object sections, they retain the global API as their context unlike commands defined directly in page object section definitions. In other words, globally defined custom commands have no access to the page object section they are called from, so they are not extensions equivalent to existing high-level commands.
Having local and global commands combined on page object sections can lead to confusion. To avoid this, "custom" (global) commands should be used only for actions that are by their nature incompatible with the page object section context and they should always be called from the global API.
// Selenium doesn't support native drag&drop yet, so we have an implementation that requires executing
// a script in the browser. Browser script execution is by its nature a context-less action.
browser.dragAndDrop('.section-container .source', '.section-container .target');
section.waitForElementVisible('.target .source');
The provided set of element-related Nightwatch commands when compared to the WebDriver commands shows a gaping omission. The high-level commands only ever deal with single element selection. There's even a config variable for making the tests fail whenever a selector supplied to a high-level command finds more than one element. Combined with the aforementioned limitation on selector context, there are no tools provided for e.g. retrieving labels from a list of items and making assertions about the whole list. Finally, the limitation on custom command context prevents simply extending the set of high-level commands to include the mass selection tools.
The above issues combined mean that trying to architect scalable and maintainable end-to-end tests with Nightwatch (as is) is largely an exercise in futility. Getting around them is dealt with in the next section. Regardless of how the various hurdles are surmounted, there's a simple guideline to make the tests maintainable and scalable.
Tests should operate on the application in user terms through the page objects.
Page objects should completely encapsulate DOM manipulation, creating an abstraction buffer between the user requirements the tests are supposed to verify and the app implementation. This helps make the tests resilient and their guarantees clear. Proper encapsulation ensures that even large-scale refactoring of the application has mostly 1:1 impact on page objects and little to no impact on tests themselves.
Achieving a clean encapsulation of implementation-specific controller code means the following.
- The tests should only interact with the app through page objects. Custom commands can be helpful to effectuate common global actions, e.g. login, but even they should work through page objects.
- The only part of page objects or their sections considered public are the local commands.
elements
aliases are for private use inside their page object / section only.
const tests = {
// good - test operates purely in user terms
'Meaning of life should be correctly calculated'(browser) {
const { calculator } = browser.page.meaningOfLife().section;
calculator
.waitForVisible()
.startCalculation()
.waitForCalculationToComplete()
.expectResult(42);
},
// bad - test directly manipulates DOM, even though it does it through page object section
'There should be cake'() {
const { calculator } = browser.page.meaningOfLife().section;
calculator
.waitForElementVisible('@container')
.click('@startCalculation')
.waitForElementNotPresent('@calculating')
.expect.element('@result').text.to.equal('42');
}
}
In a modern web application, most of the actions are asynchronous. Oftentimes the needed time is so
small that a user doesn't even notice the delay. Unfortunately, automated tests are much faster than
human operators, so they can uncover these delays. A lot of Nightwatch commands already deals with
this by waiting for configured time before declaring an element absent. Some of them don't, however,
which can make tests using them fail unpredictably. waitForElementVisible
may be used before such
commands to deal with the issue.
The most commonly used command that doesn't wait for element to appear is click
. In addition to
the possible momentary absence of the target element, the target can also be disabled, making the
command pass without warning and leading to later failures with unclear cause. The best practice in
our codebase is using our helper waitForAndClick
instead. But as it needs to be a full-fledged
equivalent to Nightwatch commands (executed in section context), it depends on the h4cks described
below.
All's well and good as long as your test suites can just end on failure, wherever it may happen, which is the default behavior provided by Nightwatch. However, if your tests require clean-up that is executed even on failure, e.g. because they need to modify a shared environment, there are a few things to know and consider.
To even start, end_session_on_fail
needs to be turned off in Nightwatch
test settings, so that it's still possible
to communicate with the browser after a test case fail. (Not to be confused with
skip_testcases_on_fail
, which makes the test suite go on after a failed case when turned off.)
Being able to interact with the browser from the after
hook, where the clean-up needs to be
located, is a good start. Unfortunately it's also a step on the way to mysterious test suite failures,
because when a test suite fails in an after
hook, Nightwatch doesn't report it properly as
a test suite failure. This leads to Nightwatch declaring that all tests passed while returning
a failure exit code. (When a suite fails in a before
hook, it's reported as a failure of the first
test case in the suite.) Additionally, turning off end_session_on_fail
means that the session needs
to be ended manually after the clean-up is done. If the clean-up fails, the session can remain open,
leaving the browser instance hanging and the driver possibly unresponsive.
So if making do without clean-up is not an option, it should at least be implemented in a way that makes it impossible to fail. This requires using WebDriver commands directly (or their proxies described at the end of this text) to be able to look for elements without failing automatically when they're not found.
Page object functionality limitations outlined above severely hamper the effort of making a Nightwatch-based test suite maintainable. This section describes the tooling we've created to work around them.
Nightwatch offers no way to hook directly into its page object instantialization process, so the only
way to modify their functionality is through monkey-patching them after they're created. This requires
tests to no longer create page objects directly and instead access them through our patcher.
(See getPageObjects.js
.)
By default, we aim to improve two aspects of page object / section functionality:
- High-level commands that select elements should use provided selectors in context of the page object / section they're called on.
- It should be possible to add more commands to all page objects / sections to be executed in context of the page object / section they're called on.
Since the native high-level commands only work in section context when used with aliases defined
in elements
, making the same apply to raw selectors passed to them requires registering new elements
for those selectors. And as Nightwatch doesn't provide access to element instantializer, we have to
clone existing elements. (See useSelectorsInContext.js
.) This imposes the following important
restriction:
All page objects and sections need to define at least one element in elements
.
Additionally, as the current value of the global locateStrategy
is inaccessible, selectors need to
be assigned locateStrategy
automatically, based on their format. The assigner assumes that all
XPath selectors should start either with ./
or with /
. As this is just a h4ck, possibly temporary,
care should be taken not to exploit this and still write the page object section command code with
the assumption that selectors use the global locateStrategy
.
const section = {
elements: {
submit: {
selector: './/button[@type="submit"]',
locateStrategy: 'xpath'
}
},
commands: [{
submit() {
return this
// good - explicit `locateStrategy` switch
.usingXpath(
() => this.waitForElementNotPresent('.//*[contains(@class, "processing")]')
)
// good - `elements` have `locateStrategy` baked in
.waitForAndClick('@submit')
// bad - even though this works now, it'd stop if we ever removed the h4ck
.waitForElementNotPresent('.//*[contains(@class, "Dialog_container")]');
}
}]
};
There's no way to push section context to global commands, so generic local commands need to be added
to the page objects / sections during the monkey-patching. To that end, local commands should be
defined separately, unbeknownst to Nightwatch, and fed to the getPageObjects
maker. Global commands
can still be used but only for logic inherently unconnected to any page object / section context.
Now that we're able to properly extend the set of omnipresent Nightwatch commands with our own, we
can fill in the missing multi-element operation commands. Making use of the WebDriver commands, we
can implement the recursive element search ourselves and provide high-level variants of all of
the element state getters for both single- and multi-element selection. To make them fully on par
with existing Nightwatch commands, we also need to support element aliases in addition to raw selectors.
(See webDriverElementCommands.js
for details.)
section
// `elementId` and `elementIds` are low-level function for contextual element selection
// this is just an example, there exists an additional `elementsCount` command just for this
// note that the low-level command `elementIds` doesn't support aliases and unwrap result
.elementIds('.hamster', ({ value }) => expect(value.length).to.be.above(1))
// `getText` could be used as well, but that doesn't unwrap the result
.elementText('xpath', './/a[2]', linkText => expect(linkText).to.equal('Release Hamsters'))
// this multi-element operation could not be otherwise performed in context of a page object section
// note the support for aliases on high-level commands `element*` and `elements*`
.elementsAttribute('@actions', 'title', tooltips => expect(tooltips).to.include('Observe the mayhem'));