How does pytest work?

README.md

In this tutorial we're going to follow the execution flow of pytest to see how it does work. Pytest is highly extendable to the extent that not only its plugins but also its core functionalities are implemented as plugins. Generally, A program and its plugins work in the way that program starts and at some specific steps of its execution, it gives control to the plugins. Those specific steps are called hooks. The program has a registered list of plugins that once program reaches a hook, their corresponding hook implementations are called in a specific order.

For its plugin system, pytest relies on Pluggy. As per Pluggy, program should hold an instance of PluginManager which is in charge of keeping hook specifications, registering plugins and calling them. Program declares its hooks specification and adds it to the plugin manager. Then plugins can register themselves to the plugin manager.

When a hook is called, its implementations are called in LIFO order, the later an implementation is registered the sooner it gets called. If a hook implementation wants to break this rule, it could use trylast or tryfirst properties in its declaration. Return value of a hook is the result of its implementations gathered in a list, unless firstresult property is specified in the hook specification which in this case the result of the first implementation returning non-None value is returned and next implementations aren't called anymore.

Program can call hooks in three ways as follows. Note that the hook implementations call order follows the explanation above. You see these ways of calling hooks in pytest code here and there.

1. Simple. By simply doing pluginmanager.hook.HOOK_NAME(**kwargs). Hook implementations are called with kwargs being passed to them.
2. Historic. By doing pluginmanager.hook.HOOK_NAME.call_historic(result_callback, **kwargs). Not only is the hook called for registered plugins but also for the ones that will be registered in the future upon their registration with kwargs being passed to them. result_callback is then called on the result of each one if the result isn't None. Note that this way works only if the hook is declared with the property historic in the specification.
3. Extra. By doing pluginmanager.hook.HOOK_NAME.call_extra(methods, **kwargs). Calls the hook implementations with kwargs by temporarily considering the given methods as extra hook implementations.

Another property that pluggy hook implementations could use upon declaration is hookwrapper. By using this property, pluggy calls the implementation before other ones, expects it to do yield and sends back the result of other hooks to it. This way the hook implementations behaves like a wrapper of the other hook implementations.

When more than one hook implementations declare tryfirst, trylast or hookwrapper properties, those hooks themselves are called in LIFO order as well.

Pytest hooks specification is located in src/_pytest/hookspec.py and its core plugins, the ones that implement the hooks and so the pytest, are located in src/_pytest as separate modules. Those plugins are listed as follows.

essential_plugins = ( # These plugins could'nt be disabled by user
    "mark",
    "main",
    "runner",
    "fixtures",
    "helpconfig",
)
default_plugins = essential_plugins + (
    "python",
    "terminal",
    ...
)

Note that essentially any object holding hook implementations as its attributes could be registered as plugin. The config:Config object of the pytest is an example. It is not put on the list above but registers itself as a plugin before the ones above. Yet before the config, pytest's pluginmanager which overrides Pluggy's PluginManager registers itself as the first pytest's plugin!

Pytest user can also provide plugins to the program by implementing desired hooks. Those plugins are supposed to be in conftest.py files throughout the test suite. User could both implement the hooks and define fixtures in them. More on these plugins later.

The main program of pytest is essentially as small as the code below. We talked about pluginmanager. config:Config holds the configuration and options of pytest. In the third statement config registers itself as the first plugin and in the for loop all pytest internal plugins are registered in turn. So in pytest hook implementations call order, that of the config is the last. config._parse is an instance of Parser(defined in src/_pytest/config/argparsing.py) which internally uses python's argparse.ArgumentParser to parse the options.

pluginmanager = PytestPluginManager() # Subclass of `pluggy.PluginManager`

config = Config(pluginmanager, args)

# Two statements below are originally in the `Config` constructor above. We brought them here for brevity.
pluginmanager.register(config, "pytestconfig")
pluginmanager.hook.pytest_addoption.call_historic(kwargs=dict(parser=config._parser, pluginmanager=pluginmanager))

for spec in default_plugins:
    pluginmanager.import_plugin(spec)

config = pluginmanager.hook.pytest_cmdline_parse(pluginmanager=pluginmanager, args=args)
config.hook.pytest_cmdline_main(config=config) # `config.hook` is a proxy for the same `pluginmanager.hook`

Now, starting by pytest_addoption, we follow the execution flow of pytest step by step. In the hook descriptions below, we name the plugins that have implementation for them and explain the implementations that do something important. Plugins are listed in the order they're called. Note that there are some other plugins that are registered dynamically and/or conditionally during the execution of pytest. We do not cover them to reduce complexity unless they are crucial to pytest's main job.

• hook.pytest_addoption(parser: Parser, pluginmanager: PluginManager) In this hook, all plugins add their specific options to the parser. Note that this hook is declared as historic (see src/_pytest/hookspec.py) so whenever a plugin gets registered in the future, this hook will get called for that as well.

• hook.pytest_cmdline_parse(pluginmanager: PluginManager, args: List[str])

Implemented by: helpconfig(hookwrapper), config

Plugin	Implementation
config	Calls self.parse(args) and returns itself. In parse method, config parses user input arguments and calls pytest_addhooks(pluginmanager) and pytest_load_initial_conftests(self, args=args, parser=self._parser) hooks.

• hook.pytest_addhooks(pluginmanager: PluginManager)

Implemented by: -

• hook.pytest_load_initial_conftests(config: Config, args: List[str], parser: Parser)

Implemented by: python_path, warnings, legacypath, capture, config

Plugin

Implementation

config

Attempts to load user-provided conftest.py plugins from the paths given in the input arguments. For each user input path, conftest.py at its root and the ones in its first-level test* subdirectories are loaded and registered as plugins then for each one, pytest_plugin_registered hook is called. Note that conftest.py files in deeper subdirectories are not loaded now but in the test collection phase so the hooks being called in the meanwhile are not called for these plugins except the historic ones.

To explain why pytest has made such a distinction between user-provided plugins, we note that pytest hooks are basically of two categories. The ones that pertain to pytest's general behaviour and the others that pertain to pytest's behaviour in dealing with a specific part of the test suite e.g. tests located in a specific path. Hooks of the former category are scattered throughtout the pytest runtime but those of the latter take place starting from the test collection phase. Thus loading plugins implementing hooks of this category could be deferred until then for the sake of readability and performance. To this end, pytest recommends the user to put implementations of the former category in the test suite root path or root directories as a convention and somewhat enforces it by introducing this hook.

In case you ask what's the point of having conftest.py plugins in deeper directories then, the answer is to customize pytest behaviour differently in different parts of the test suite by implementing the hooks of the second category in conftest.py plugins of those parts differently. But how is this possible while we call all implementations of a hook when it's called? We see it shortly.

• hook.pytest_plugin_registered(plugin: object, pluginmanager: PluginManager) This hook is historic and there are some plugins registered later which implement this. We explain them later, but to have a clue FixtureManager(located in src/_pytest/fixtures) is the most important one.

Implemented by: legacypath

• hook.pytest_cmdline_main(config: Config) This hook is a firstresult one.

Implemented by: setupplan, setuponly, cacheprovider, python, helpconfig, main, mark

Plugin

Implementation

main

First pytest_configure hook is called. Now pytest is configured and ready to do the main job. So a session:Session object is constructed and pytest_sessionstart, pytest_collection, pytest_runtestloop hooks are called respectively. Finally pytest_configure hook is called. This implementation either raises an error or returns an exit status. Note that session registers itself as a plugin in its constructor.

• hook.pytest_configure(config: Config) This hook is a historic hook and is aimed at configuring pytest and the plugins themselves. Many of the builtin_plugins implement it, some of which are worth explaining.

Plugin	Location	Description
terminalreporter	src/_pytest/terminal.py	This plugin is responsible for preparing whatever user sees after running pytest command. In the implementation, plugin constructs a TerminalReporter and registers it as a plugin.
lfplugin	src/_pytest/cacheprovider.py	Enabled by --failed-first and --last-failed options, this plugin lets user to run only the tests which have failed in the last run or run them before other tests respectively. In the implementation, a LFPlugin object is constructed and registered as plugin.
nfplugin	src/_pytest/cacheprovider.py	Enabled by --new-first option, lets user to run only new tests, the tests that have been added after last run. In the implementation, a NFPlugin object is constructed and registered as plugin.
stepwiseplugin	src/_pytest/stepwise.py	Enabled by --stepwise or --stepwise-skip, makes pytest to stop on the first test failure and in the next run, start off by that failing test. If the --stepwise-skip option is enabled pytest stops at second failure. In the implementation, a StepwisePlugin object is constructed and registered as plugin.

• hook.pytest_sessionstart(session: Session)

Implemented by: stepwiseplugin, junitxml, fixtures, runner, terminalreporter(trylast)

Plugin	Implementation
fixtures	Constructs a FixtureManager object and assigns it to the session. FixtureManager is responsible for loading fixtures from files and maintaining them. This object registers itself as a plugin in its constructor with the name funcmanage which we refer to by later on.
runner	Constructs a SetupState object and assigns it to the session. We explain this object later.
terminalreporter	Reports the output header of pytest command which includes the line containing ...==== test session starts====....

• hook.pytest_collection(session: Session)

Implemented by: warnings(tryfirst, hookwrapper), config(hookwrapper), terminalreporter, assertion, main

Plugin	Implementation
main	Calls perform_collect method of session and returns nothing. After that, collected tests are stored in items member of the session
terminalreporter	Reports collecting ... to the output

Before going to see what happens in session's perform_collect() during pytest_collection hook, we note that pytest represents the test suite as a tree of nodes reflecting file system hierarchy and dependency between tests and their upper-level entities like classes, modules(a .py file), packages(a folder containing a __init__.py file along with other .py files) and the very session. Root of this tree is the session and its leaves are the test invocations. Note that we didn't say tests because test is the function or class method user writes and since he/she might parametrize it, there may be many invocations of it, each one a leaf in the tree. Here comes the types with which the test suite is modelled into a tree.

Class	description
Node	All nodes of the test tree inherit this class. parent, nodeid and path are its notable attributes and has reference to the session object.
Collector	Represents a non-leaf node in the tree. Collects its children through its collect method.
FSCollector	A node that has one-to-one association with an entity located in file system. Inherits Collector.
PyCollector	A node that has one-to-one association with a python object like module, class, function or method. Inherits Collector. Its associated python object is stored in obj attribute.
Item	Represents a test invocation. Has a runtest method which is implemented by subclasses. Inherits Node
Function	A node that is associated with a python function or class method. Inherits PyCollector. It has two important attributes, namely _fixtureinfo and callspec which will be explained later. _fixtureinfo:FuncFixtureInfo holds information about the fixtures this functions uses and callspec:CallSpec2 determines which values of arguments and/or fixures the function gets called with in the case it has been parametrized directly or through fixture respectively. Inherits Item.
DoctestItem	As you might know, pytest is able to run doctests located in python or text files in a test suite as well. This node has one-to-one association with a doctest. Inherits Item.
TestCaseFunction	Pytest is capable of running tests written using Python's unittest library. A node of this type is associated with a single test function written using unittest. Inherits Function.
Class	A node that has one-to-one association with a python class. Inherits PyCollector
UnitTestCase	Represents a unittest class containing TestCaseFunctions. Inherits Class.
Module	This node is associated with a python module. Inherits empty class File which itself inherits FSCollector.
DoctestTextfile	This node is associated with a .txt or .rst file containing doctests. Inherits Module.
DoctestModule	A node that is associated with a python file but is supposed to collect its doctests, not its test functions. Inherits Module.
Package	A node that has one-to-one association with a python package. Its obj attribute is the __init__.py module located in the package. Inherits Module.
Session	Associates with the root path of the test suite and test collection starts from it. Inherits FSCollector.

perform_collect method of the session object is roughly as follows:

self.items = []
rep = collect_one_node(self)
for node in rep.result:
    self.items.extend(self.genitems(node))

hook.pytest_collection_modifyitems(self, self.config, self.items)
hook.pytest_collection_finish(self)

collect_one_node(Collector) is a function located in _pytest/runner.py and does collection of the nodes that descend from a specific node in the node tree. This function is roughly as follows.

def collect_one_node(collector: Collector):
    collector.ihook.pytest_collectstart(collector=collector)
    rep: CollectReport = collector.ihook.pytest_make_collect_report(collector=collector)
    return rep

As you see, it calls two hooks pytest_collectstart(Collector) and pytest_make_collect_report(Collector) internally and returns the result of the latter. The term collector.ihook seems odd. Here is the explanation. There are two ways to call a hook in pytest. First is the general way in which all the user-provided plugins having implementation for the hook are considered. This could be done by accessing hook attribute of config or pluginmanager. But in the second way which we name fspath-sensitive hook call, conftest.py plugins not located in the ancestor directories of the fspath are ignored when the hook is called. This allows to further customize pytest behaviour as we stated earlier.

To use this way, user should access ihook attribure of the node whose fspath user wants the hook be sensitive to. The attribute is computed first time it's called simply by doing self.session.gethookproxy(self.path) and then is stored. gethookproxy method is roughly as follows. It first retrieves the conftest plugins visible to the path that is the conftest.py modules existed in its ancestor directories as well as its own directory. Then it finds out the plugins which should be ignored and a proxy object is returned accordingly. FSHookProxy is a container of Pluggys' _SubsetHookCallers that whenever is called with a new hook name, constructs a _SubsetHookCaller by calling pm.subset_hook_caller(hook_name, remove_mods) and stores it in its __dict__ attribute. PathAwareHookProxy is also a wrapper.

def gethookproxy(self, path):
    pm = self.config.pluginmanager
    my_conftestmodules = pm._getconftestmodules(path, self.config.getoption("importmode"), rootpath=self.config.rootpath)
    remove_mods = pm._conftest_plugins.difference(my_conftestmodules) # `_conftest_plugins` holds all the conftest plugins
    if remove_mods:
        proxy = PathAwareHookProxy(FSHookProxy(pm, remove_mods))
    else:
        proxy = self.config.hook
    return proxy

We note that pm._getconftestmodules(path, importmode, rootpath) is the same method being used in config's pytest_load_initial_conftest hook implementation. This method first consults plugin manager's _dirpath2confmods attribute which plays the role of a cache. If it misses, for each conftest plugin located in path's ancestors as well as in its own directory, pm._importconftest(conftestpath, importmode, rootpath) is called. _importconftest loads and registers the plugin in case it hasn't been already loaded and registered, then returns it. _getconftestmodules then puts the loaded plugins into a list, updates the _dirpath2confmods and returns the list. For each conftest plugin, pytest_plugin_registered hook is called after loading.

Let's come back to collect_one_node, starting by pytest_collectstart hook.

• hook.pytest_collectstart(collector: Collector)

Implemented by: session

Plugin	Implementation
session	Checks if pytest execution should be stopped because of interruption or failure. Either returns None or raises an exception.

• hook.pytest_make_collect_report(collector: Collector)

Implemented by: capture(hookwrapper), runner

Plugin	Implementation
runner	Among other things, calls collect() method of the collector and roughly returns its result which is of type Iterable[Union["Item", "Collector"]]

This collect() method is declared as abstract in Collector class and is defined in some of its subclasses as follows.

• Session: We saw earlier that hook pytest_collection(session), session.perform_collect() and collect_one_node(session) led to session.collect() Here we start to collect the nodes of the tree, more specifically the ones that user has designated in the pytest command input. Suppose our test suite is like below.

tests/
  package1/
    __init__.py
    conftest.py
    test_module_a.py
      ::TestClass1
        ::test_method1
      ::test_func_1
    package2/
      __init__.py
      test_module_b.py
  package3/
    __init__.py
    folder4/
      test_module_c.py
    test_module_d.py
    test_module_e.py
      ::TestClass2
        ::test_method2
      ::test_func2

Then user runs

pytest tests/package1 tests/package3/folder4 tests/package3/test_module_d.py

Then function returns two Package nodes for package1 and package2, and two Module nodes for test_module_c and test_module_d respectively. Note that when user designates a package, all of its subpackages are also collected. Also note that folder4 is not a package since it hasn't __init__.py, so its only contained module is collected as a Module. User is also able to select a class, a method, a function or even a test invocation. For example user can run the following command in which the function returns a Function node, a Class node and two other Function nodes respectively.

pytest tests/package1/test_module_a.py::test_func_1 tests/package1/test_module_a.py::TestClass1 tests/package3/test_module_e.py::TestClass2::test_method2 tests/package3/test_module_e.py::test_func2[param0]

Collection is first done using pytest_collect_file hook as follows.

• hook.pytest_collect_file(file_path: Path, parent: nodes.Collector)

Implemented by: doctest, python

Plugin	Implementation
doctest	returns a DoctestModule object if the file is a python file and doctestmodules option is enabled, otherwise it returns a DoctestTextfile object if the file is of extension txt or rst.
python	Calls pytest_pycollect_makemodule(file, parent) hook if file is a .py file otherwise returns None.

• hook.pytest_pycollect_makemodule(module_path: Path, parent) This hook is a firstresult one.

Implemented by: python

Plugin	Implementation
python	If module_path is path of a __init__.py file, it constructs and returns a Package otherwise a Module.

Using these two hooks, some Packages or Modules are constructed for each input argument, but if the argument represents a class, method or a function, some further steps is needed. If argument is a class, a function or a test invocation, collect_one_node(so a pytest_make_collect_report hook and then collect) is called on its module and the proper Class or Function node is retrieved. If argument is a method, calling collect_one_node on its class is added to the previous step. Note that for parametrized test functions in a module or a class, all of the test invocations are collected.

• Package: For each subpackage (at any level), a Package instance and for its modules Module instances is created (using pytest_collect_file hook) and returned. Since Session also collects single modules, the ones that are not located in a package but a folder, Package does not collect them. It checks them out by checking config.pluginmanager._duplicatepaths.

• Module: Among other things, calls self.session._fixturemanager.parsefactories(self). This function looks for fixtures found in the module, constructs a FixtureDef for each one and inserts them into the dictionary session._fixturemanager._arg2fixturedefs. FixtureDef holds the fixture function defined by user in its func attribute. _arg2fixturedefs is a dictionary of type Dict[str, List[FixtureDef]] which holds the collected fixtures in the test suite with the same name under that name's entry. After collecting fixtures, Module calls collect method of its parent class, PyCollector. Recall from beginning of the tutorial that pytest_cmdline_parse hook loads the user-defined plugins, namely the conftest.py files. A conftest.py is actually an ordinary module located in a package or in the root of the test suite. Within it, user can provide hook implementations and fixture functions. This file has the property that its fixtures are only accessible by the subnodes of its corresponding package. Also its hook implementations are only called fore for the subnodes of its corresponding package.

• Class: Like Module, first collects fixtures from the class then calls collect method of its parent class, PyCollector.

• PyCollector: By inspecting its associated entity (module or class), returns a list of subnodes (of class Item or Class). For each subnode, it calls pytest_pycollect_makeitem hook to make it.

• hook.pytest_pycollect_makeitem(collector: Union["Module", "Class"], name: str, obj: object) This hook is a firstresult one.

Implemented by: unittest, python(trylast)

Plugin	Implementation
unittest	if obj's class inherits unittest's TestCase, returns a UnitTestCase otherwise returns nothing.
python	if obj is a class, a Class node is constructed and returned. If obj is a function, first a MetaFunc responsible for parametrizing the function is created then a list of Functions is generated using pytest_generate_tests hook and is returned. If obj is neither a class nor a function, None is returned.

• hook.pytest_generate_tests(metafunc: MetaFunc)

Implemented by: funcmanage, python

Plugin	Implementation
funcmanage	For any of the fixtures which metafunc uses, metafunc.parametrize is called which extends the call specification list of the metafunc stored in its _calls attribute.
python	Iterates over @pytest.mark.parametrize markers of the test function or method and for each one calls metafunc.parametrize.

Going back to session.perform_collect() depicted below, now we know that collect_one_node(self) returns nodes designated by the user. Type of these nodes might be Item, Class, Module or Package among which only Item is the acutal test invocation and others need further collection. So in genitems, for an Item, pytest_itemcollected hook is called and then the item is retured as is. But for a Collector, again, collect_one_node is called and then genitems is called on its returned subnodes.

self.items = []
rep = collect_one_node(self)
for node in rep.result:
    self.items.extend(self.genitems(node))

hook.pytest_collection_modifyitems(self, self.config, self.items)
hook.pytest_collection_finish(self)

After all items are collected, pytest_collection_modifyitems and pytest_collection_finish hooks are called.

• hook.pytest_collection_modifyitems(items: List[nodes.Item], config: Config)

Implemented by: nfplugin(hookwrapper, tryfirst), lfplugin(hookwrapper, tryfirst), stepwiseplugin, funcmanage, main, mark

Plugin	Implementation
nfplugin	If enabled by --new-first option, first finds out new items, the ones that have been added after the last run, using a cache and modify items in-place to first include these items and then other items. Items in both sets are sorted in decreasing modify time of their respective test files. Finally the cache holding item ids is updated by items.
lfplugin	If enabled by --last-failed or --failed-first, modifies items in-place to include only the tests which failed in the last run. In the latter case it appends other tests to the new items as well. If no test has failed in the last run, it runs no test if --last-failed-no-failures option is "none" otherwise all of tests are run.
stepwiseplugin	If enabled by --stepwise or --stepwise-skip options, it keeps only the tests after the first or second failed test (including itself) in the last run respectievely.
funcmanage	Does an important job. Suppose some items use a fixture with a high-level scope (higher than function). Isn't better for these items to get called close to each other to lessen these two conditions? Tests setup time. The fixtures take time to setup themselves. So it's desirable to minimize the number of fixture setups. Undesired alterations in system state. The fixture might make some changes in system state which could make things unpredictable when an item not dependant on the fixture is being called. Hence, this hook implementation does a reorder of collected items to satisfy those conditions.
main	Remove items deselected in deselected config option.
mark	In the case user has specified tests with specific keywords or markers using -k and -m respectively in the input, it keeps them and remove other items from items.

To recap the execution flow of pytest, as the user calls the command, first pytest_cmdline_main hook is called. This hook then calls pytest_sessionstart, pytest_collection, pytest_runtestloop and pytest_sessionfinish hooks respectively. So far we've moved along till before the pytest_runtestloop hook. Let's go on.

• hook.pytest_runtestloop(session: Session) This hook is a firstresult one.

Implemented by: main

Plugin	Implementation
main	pytest_runtest_protocol(item, nextitem) hook is called for each item in session.items. Either raises error, or return true.

• hook.pytest_runtest_protocol(item: Item, nextitem: Optional[Item]) This hook is a firstresult one.

Implemented by: warnings(hookwrapper, tryfirst), assertion(hookwrapper, tryfirst), faulthandler(hookwrapper, trylast), unittest(hookwrapper), runner

Plugin	Implementation
runner	Among other things, it calls runtestprotocol(item, nextitem) function and returns true.

• hook.pytest_sessionfinish(session: Session, exitstatus: Union[int, ExitCode])

Implemented by: terminalreporter(hookwrapper), warnings(hookwrapper), stepwise, assertion, tmpdir, runner

Plugin	Implementation
terminalreporter	It prints some info in the case of error or keyboard interrupt or some summary statistics.
runner	Calls teardown_exact(None) on the _setupstate member of the session.

In src/_pytest/runner.py::runtestprotocol function, test item is set up, called and teared down. This function is roughly as follows. Depending on the second argument, a specific hook is called.

call_and_report(item, "setup", log) # Calls `pytest_runtest_setup` hook.
call_and_report(item, "call", log) # Calls `pytest_runtest_call` hook.
call_and_report(item, "teardown", log, nextitem) # Calls `pytest_runtest_teardown` hook.

• hook.pytest_runtest_setup( item : nodes.Item)

Implemented by: unraisableexception(hookwrapper, tryfirst), threadexception(hookwrapper, trylast), skipping(tryfirst), runner, nose(trylast)

Plugin

Implementation

runner

Among other things, calls item.session._setupstate.setup(item). session._setupstate is of type SetupState and is responsible for maintaining which nodes pytest has run their setup and are yet to tear down. This object is like a stack that when an item is introduced to it, it pushes that item's parents to the stack and calls their setup() method. Among node types, only Package and Function implement setup(). The former calls a specific user-provided __init__.py::setup_module function and the latter runs self._request._fillfixtures().

• hook.pytest_runtest_call( item : nodes.Item)

Implemented by: unraisableexception(hookwrapper, tryfirst), threadexception(hookwrapper, tryfirst), logging(hookwrapper), skipping(hookwrapper), capture(hookwrapper), runner

Plugin	Implementation
runner	Calls item.runtest()

• hook.pytest_runtest_teardown( item : nodes.Item, nextitem :nodes.Item)

Implemented by: unraisableexception(hookwrapper, tryfirst), threadexception(hookwrapper, tryfirst), logging(hookwrapper), skipping(hookwrapper), capture(hookwrapper), runner

Plugin	Implementation
runner	Calls item.session._setupstate.teardown_exact(nextitem)