How to modularize your py.test fixtures

pytest-fixture-modularization.md

Using py.test is great and the support for test fixtures is pretty awesome. However, in order to share your fixtures across your entire module, py.test suggests you define all your fixtures within one single conftest.py file. This is impractical if you have a large quantity of fixtures -- for better organization and readibility, you would much rather define your fixtures across multiple, well-named files. But how do you do that? ...No one on the internet seemed to know.

Turns out, however, you can define fixtures in individual files like this:

tests/fixtures/add.py

import pytest

@pytest.fixture
def add(x, y):
    x + y

Then you can import these fixtures in your conftest.py:

tests/conftest.py

import pytest
from fixtures.add import add

...and then you're good to test!

tests/adding_test.py

import pytest

@pytest.mark.usefixtures("add")
def test_adding(add):
    assert add(2, 3) == 5

Because of the modularity, tests will have to be run with python -m py.test instead of py.test directly.

Does pytest try to import all the fixtures from the pytest_plugins variable? I was wondering if the method mentioned in this post can be applied to resolve dependency issues that occur with so many fixtures being present in a single file, something like only the fixtures that are needed are imported... (I cant necessarily have multiple conftest files as I want to import and resuse fixtures across different directories)?

Does pytest try to import all the fixtures from the pytest_plugins variable?

Yes, every module in pytest_plugins is loaded and all fixtures in that module are made available to all tests.

glob("tests/fixtures/*.py") if "__" not in fixture

Can be replaced with

glob("tests/fixtures/[!__]*.py")

or even

glob("tests/fixtures/[!_]*.py")

to also skip filenames that start with single underscore.

Just a solution I've come across for anyone reading:

# conftest.py
here = os.path.dirname(os.path.realpath(__file__))

pytest_plugins = []

def _as_module(root: str, path: str) -> str:
    path = os.path.join(root, path)
    path = path.replace(here, "")
    path = path.replace(".py", "")
    path = path.replace(os.path.sep, ".")[1:]
    return "test." + path


for root, dirs, files in os.walk(here, topdown=True):
    dirs[:] = [d for d in dirs if d.startswith("test")]
    pytest_plugins += [_as_module(root, f) for f in files if f.endswith("fixtures.py")]

It allows you to have a fixtures file in any subdirectory

.../test
   __init__.py
   conftest.py
   /test_f1
      __init__.py
      fixtures.py
   
      /test_subf1
         __init__.py
         fixtures.py
   
      /test_subf2
         __init__.py
         endswith_fixtures.py
         extra_fixtures.py
   
   /test_f2
      __init__.py
      fixtures.py

@eddiebergman That's what the code in https://gist.github.com/peterhurford/09f7dcda0ab04b95c026c60fa49c2a68#gistcomment-3453453 (using pkgutil.walk_packages) already does.

@ikonst Ahh, I didn't know it supported the same layout with fixtures in each dir, very nice

You can simply create "local pytest plugins" which can be nothing more than Python files with fixtures, e.g.

* tests/unit/conftest.py:
  ```python
  pytest_plugins = [
     "tests.unit.fixtures.some_stuff",
  ]

• tests/unit/fixtures/some_stuff.py:

  import pytest
  
  @pytest.fixture
  def foo():
      return 'foobar'

Thanks for you idea @ikonst! Just made my conftest as package and add it as plugin!

Can you share how did you share the conftest as a package? Can that package be pip installed across other repo/projects?

There's no much difference between a conftest.py and a pytest plugin (only one I can think of, is that a top-level conftest can have a pytest_plugins section).

If you want to make a Python package with pytest fixtures, see https://docs.pytest.org/en/6.2.x/writing_plugins.html

Hello everyone, thank you for all the insightful comments.
If you use PyCharm, it seems that, for now, you won't be able to use the glob approach because of this issue, which cleans up the "unused import" needed with this approach.

My recommendation would be to use the hard-coded list and maintain the conftest.py file.
With this, you also do not need to import the fixture in your test file.

Otherwise, here's a summary of the above:

tests/conftest.py

from glob import glob
def _as_module(fixture_path: str) -> str:
    return fixture_path.replace("/", ".").replace("\\", ".").replace(".py", "")


pytest_plugins = [
    _as_module(fixture) for fixture in glob("tests/fixtures/[!_]*.py")
]

tests/fixtures/example.py

@pytest.fixture()
def example():
    return []

tests/unit/mytest.py

# If you did not hardcode pytest_plugins, you might need `from tests.fixtures.example import example`
def test_get_a_list(example):
    assert isinstance(example, list)

pytest_plugins

It worked like a charm, thank you.

Great gist on modularization of conftest, it works great, thanks!

And thanks to @ryanjdillon for the tip about init.py, it solved a mypy problem for me.

Another option would be to have multiple nested directories/packages containing your tests, and each directory can have its own conftest.py with its own fixtures, adding on to the ones provided by the conftest.py files in parent directories as written in the official doc here