Working with plugins and conftest files¶
pytest
implements all aspects of configuration, collection, running and reporting by calling well specified hooks. Virtually any Python module can be registered as a plugin. It can implement any number of hook functions (usually two or three) which all have a pytest_
prefix, making hook functions easy to distinguish and find. There are three basic location types:
- builtin plugins: loaded from pytest’s internal
_pytest
directory. - external plugins: modules discovered through setuptools entry points
- conftest.py plugins: modules auto-discovered in test directories
conftest.py: local per-directory plugins¶
local conftest.py
plugins contain directory-specific hook
implementations. Session and test running activities will
invoke all hooks defined in conftest.py
files closer to the
root of the filesystem. Example: Assume the following layout
and content of files:
a/conftest.py:
def pytest_runtest_setup(item):
# called for running each test in 'a' directory
print ("setting up", item)
a/test_sub.py:
def test_sub():
pass
test_flat.py:
def test_flat():
pass
Here is how you might run it:
py.test test_flat.py # will not show "setting up"
py.test a/test_sub.py # will show "setting up"
Note
If you have conftest.py
files which do not reside in a
python package directory (i.e. one containing an __init__.py
) then
“import conftest” can be ambiguous because there might be other
conftest.py
files as well on your PYTHONPATH or sys.path
.
It is thus good practise for projects to either put conftest.py
under a package scope or to never import anything from a
conftest.py file.
Installing External Plugins / Searching¶
Installing a plugin happens through any usual Python installation tool, for example:
pip install pytest-NAME
pip uninstall pytest-NAME
If a plugin is installed, pytest
automatically finds and integrates it,
there is no need to activate it. We have a page listing
all 3rd party plugins and their status against the latest py.test version and here is a little annotated list
for some popular plugins:
- pytest-django: write tests for django apps, using pytest integration.
- pytest-twisted: write tests for twisted apps, starting a reactor and processing deferreds from test functions.
- pytest-capturelog: to capture and assert about messages from the logging module
- pytest-cov: coverage reporting, compatible with distributed testing
- pytest-xdist: to distribute tests to CPUs and remote hosts, to run in boxed mode which allows to survive segmentation faults, to run in looponfailing mode, automatically re-running failing tests on file changes, see also xdist: pytest distributed testing plugin
- pytest-instafail: to report failures while the test run is happening.
- pytest-bdd and pytest-konira to write tests using behaviour-driven testing.
- pytest-timeout: to timeout tests based on function marks or global definitions.
- pytest-cache: to interactively re-run failing tests and help other plugins to store test run information across invocations.
- pytest-pep8:
a
--pep8
option to enable PEP8 compliance checking. - oejskit: a plugin to run javascript unittests in life browsers
To see a complete list of all plugins with their latest testing status against different py.test and Python versions, please visit plugincompat.
You may also discover more plugins through a pytest- pypi.python.org search.
Writing a plugin by looking at examples¶
If you want to write a plugin, there are many real-life examples you can copy from:
- a custom collection example plugin: A basic example for specifying tests in Yaml files
- around 20 builtin plugins which provide pytest’s own functionality
- many external plugins providing additional features
All of these plugins implement the documented well specified hooks to extend and add functionality.
You can also contribute your plugin to pytest-dev once it has some happy users other than yourself.
Making your plugin installable by others¶
If you want to make your plugin externally available, you
may define a so-called entry point for your distribution so
that pytest
finds your plugin module. Entry points are
a feature that is provided by setuptools. pytest looks up
the pytest11
entrypoint to discover its
plugins and you can thus make your plugin available by defining
it in your setuptools-invocation:
# sample ./setup.py file
from setuptools import setup
setup(
name="myproject",
packages = ['myproject']
# the following makes a plugin available to pytest
entry_points = {
'pytest11': [
'name_of_plugin = myproject.pluginmodule',
]
},
)
If a package is installed this way, pytest
will load
myproject.pluginmodule
as a plugin which can define
well specified hooks.
Plugin discovery order at tool startup¶
pytest
loads plugin modules at tool startup in the following way:
by loading all builtin plugins
by loading all plugins registered through setuptools entry points.
by pre-scanning the command line for the
-p name
option and loading the specified plugin before actual command line parsing.by loading all
conftest.py
files as inferred by the command line invocation:- if no test paths are specified use current dir as a test path
- if exists, load
conftest.py
andtest*/conftest.py
relative to the directory part of the first test path.
Note that pytest does not find
conftest.py
files in deeper nested sub directories at tool startup. It is usually a good idea to keep your conftest.py file in the top level test or project root directory.by recursively loading all plugins specified by the
pytest_plugins
variable inconftest.py
files
Requiring/Loading plugins in a test module or conftest file¶
You can require plugins in a test module or a conftest file like this:
pytest_plugins = "name1", "name2",
When the test module or conftest plugin is loaded the specified plugins will be loaded as well. You can also use dotted path like this:
pytest_plugins = "myapp.testsupport.myplugin"
which will import the specified module as a pytest
plugin.
Accessing another plugin by name¶
If a plugin wants to collaborate with code from another plugin it can obtain a reference through the plugin manager like this:
plugin = config.pluginmanager.getplugin("name_of_plugin")
If you want to look at the names of existing plugins, use
the --traceconfig
option.
Finding out which plugins are active¶
If you want to find out which plugins are active in your environment you can type:
py.test --traceconfig
and will get an extended test header which shows activated plugins and their names. It will also print local plugins aka conftest.py files when they are loaded.
Deactivating / unregistering a plugin by name¶
You can prevent plugins from loading or unregister them:
py.test -p no:NAME
This means that any subsequent try to activate/load the named plugin will it already existing. See Finding out which plugins are active for how to obtain the name of a plugin.
pytest default plugin reference¶
You can find the source code for the following plugins in the pytest repository.
_pytest.assertion |
support for presenting detailed information in failing assertions. |
_pytest.capture |
per-test stdout/stderr capturing mechanism. |
_pytest.config |
command line options, ini-file and conftest.py processing. |
_pytest.doctest |
discover and run doctests in modules and test files. |
_pytest.genscript |
generate a single-file self-contained version of pytest |
_pytest.helpconfig |
version info, help messages, tracing configuration. |
_pytest.junitxml |
report test results in JUnit-XML format, for use with Hudson and build integration servers. |
_pytest.mark |
generic mechanism for marking and selecting python functions. |
_pytest.monkeypatch |
monkeypatching and mocking functionality. |
_pytest.nose |
run test suites written for nose. |
_pytest.pastebin |
submit failure or test session information to a pastebin service. |
_pytest.pdb |
interactive debugging with PDB, the Python Debugger. |
_pytest.pytester |
(disabled by default) support for testing pytest and pytest plugins. |
_pytest.python |
Python test discovery, setup and run of test functions. |
_pytest.recwarn |
recording warnings during test function execution. |
_pytest.resultlog |
log machine-parseable test session result information in a plain |
_pytest.runner |
basic collect and runtest protocol implementations |
_pytest.main |
core implementation of testing process: init, session, runtest loop. |
_pytest.skipping |
support for skip/xfail functions and markers. |
_pytest.terminal |
terminal reporting of the full testing process. |
_pytest.tmpdir |
support for providing temporary directories to test functions. |
_pytest.unittest |
discovery and running of std-library “unittest” style tests. |
pytest hook reference¶
Hook specification and validation¶
pytest
calls hook functions to implement initialization, running,
test execution and reporting. When pytest
loads a plugin it validates
that each hook function conforms to its respective hook specification.
Each hook function name and its argument names need to match a hook
specification. However, a hook function may accept fewer parameters
by simply not specifying them. If you mistype argument names or the
hook name itself you get an error showing the available arguments.
Initialization, command line and configuration hooks¶
-
pytest_load_initial_conftests
(args, early_config, parser)[source]¶ implements the loading of initial conftest files ahead of command line option parsing.
-
pytest_cmdline_preparse
(config, args)[source]¶ (deprecated) modify command line arguments before option parsing.
-
pytest_cmdline_parse
(pluginmanager, args)[source]¶ return initialized config object, parsing the specified args.
-
pytest_namespace
()[source]¶ return dict of name->object to be made globally available in the pytest namespace. This hook is called before command line options are parsed.
-
pytest_addoption
(parser)[source]¶ register argparse-style options and ini-style config values.
This function must be implemented in a plugin and is called once at the beginning of a test run.
Parameters: parser – To add command line options, call parser.addoption(...)
. To add ini-file values callparser.addini(...)
.Options can later be accessed through the
config
object, respectively:config.getoption(name)
to retrieve the value of a command line option.config.getini(name)
to retrieve a value read from an ini-style file.
The config object is passed around on many internal objects via the
.config
attribute or can be retrieved as thepytestconfig
fixture or accessed via (deprecated)pytest.config
.
-
pytest_cmdline_main
(config)[source]¶ called for performing the main command line action. The default implementation will invoke the configure hooks and runtest_mainloop.
Generic “runtest” hooks¶
All runtest related hooks receive a pytest.Item
object.
-
pytest_runtest_protocol
(item, nextitem)[source]¶ implements the runtest_setup/call/teardown protocol for the given test item, including capturing exceptions and calling reporting hooks.
Parameters: - item – test item for which the runtest protocol is performed.
- nextitem – the scheduled-to-be-next test item (or None if this
is the end my friend). This argument is passed on to
pytest_runtest_teardown()
.
Return boolean: True if no further hook implementations should be invoked.
-
pytest_runtest_teardown
(item, nextitem)[source]¶ called after
pytest_runtest_call
.Parameters: nextitem – the scheduled-to-be-next test item (None if no further test item is scheduled). This argument can be used to perform exact teardowns, i.e. calling just enough finalizers so that nextitem only needs to call setup-functions.
-
pytest_runtest_makereport
(item, call)[source]¶ return a
_pytest.runner.TestReport
object for the givenpytest.Item
and_pytest.runner.CallInfo
.
For deeper understanding you may look at the default implementation of
these hooks in _pytest.runner
and maybe also
in _pytest.pdb
which interacts with _pytest.capture
and its input/output capturing in order to immediately drop
into interactive debugging when a test failure occurs.
The _pytest.terminal
reported specifically uses
the reporting hook to print information about a test run.
Collection hooks¶
pytest
calls the following hooks for collecting files and directories:
-
pytest_ignore_collect
(path, config)[source]¶ return True to prevent considering this path for collection. This hook is consulted for all files and directories prior to calling more specific hooks.
-
pytest_collect_directory
(path, parent)[source]¶ called before traversing a directory for collection files.
-
pytest_collect_file
(path, parent)[source]¶ return collection Node or None for the given path. Any new node needs to have the specified
parent
as a parent.
For influencing the collection of objects in Python modules you can use the following hook:
-
pytest_pycollect_makeitem
(collector, name, obj)[source]¶ return custom item/collector for a python object in a module, or None.
After collection is complete, you can modify the order of items, delete or otherwise amend the test items:
Reporting hooks¶
Session related reporting hooks:
And here is the central hook for reporting about test execution:
Debugging/Interaction hooks¶
There are few hooks which can be used for special reporting or interaction with exceptions:
Declaring new hooks¶
Plugins and conftest.py
files may declare new hooks that can then be
implemented by other plugins in order to alter behaviour or interact with
the new plugin:
-
pytest_addhooks
(pluginmanager)[source]¶ called at plugin load time to allow adding new hooks via a call to pluginmanager.registerhooks(module).
Hooks are usually declared as do-nothing functions that contain only documentation describing when the hook will be called and what return values are expected.
For an example, see newhooks.py from xdist: pytest distributed testing plugin.
Using hooks from 3rd party plugins¶
Using new hooks from plugins as explained above might be a little tricky because the standard Hook specification and validation mechanism: if you depend on a plugin that is not installed, validation will fail and the error message will not make much sense to your users.
One approach is to defer the hook implementation to a new plugin instead of declaring the hook functions directly in your plugin module, for example:
# contents of myplugin.py
class DeferPlugin(object):
"""Simple plugin to defer pytest-xdist hook functions."""
def pytest_testnodedown(self, node, error):
"""standard xdist hook function.
"""
def pytest_configure(config):
if config.pluginmanager.hasplugin('xdist'):
config.pluginmanager.register(DeferPlugin())
This has the added benefit of allowing you to conditionally install hooks depending on which plugins are installed.
hookwrapper: executing around other hooks¶
New in version 2.7: (experimental)
pytest plugins can implement hook wrappers which which wrap the execution of other hook implementations. A hook wrapper is a generator function which yields exactly once. When pytest invokes hooks it first executes hook wrappers and passes the same arguments as to the regular hooks.
At the yield point of the hook wrapper pytest will execute the next hook
implementations and return their result to the yield point in the form of
a CallOutcome
instance which encapsulates a result or
exception info. The yield point itself will thus typically not raise
exceptions (unless there are bugs).
Here is an example definition of a hook wrapper:
import pytest
@pytest.mark.hookwrapper
def pytest_pyfunc_call(pyfuncitem):
# do whatever you want before the next hook executes
outcome = yield
# outcome.excinfo may be None or a (cls, val, tb) tuple
res = outcome.get_result() # will raise if outcome was exception
# postprocess result
Note that hook wrappers don’t return results themselves, they merely perform tracing or other side effects around the actual hook implementations. If the result of the underlying hook is a mutable object, they may modify that result, however.
Reference of objects involved in hooks¶
-
class
Config
[source]¶ access to configuration values, pluginmanager and plugin hooks.
-
option
= None¶ access to command line option as attributes. (deprecated), use
getoption()
instead
-
pluginmanager
= None¶ a pluginmanager instance
-
addinivalue_line
(name, line)[source]¶ add a line to an ini-file option. The option must have been declared but might not yet be set in which case the line becomes the the first line in its value.
-
getini
(name)[source]¶ return configuration value from an ini file. If the specified name hasn’t been registered through a prior
parser.addini
call (usually from a plugin), a ValueError is raised.
-
getoption
(name, default=<NOTSET>, skip=False)[source]¶ return command line option value.
Parameters: - name – name of the option. You may also specify
the literal
--OPT
option instead of the “dest” option name. - default – default value if no option of that name exists.
- skip – if True raise pytest.skip if option does not exists or has a None value.
- name – name of the option. You may also specify
the literal
-
-
class
Parser
[source]¶ Parser for command line arguments and ini-file values.
-
getgroup
(name, description='', after=None)[source]¶ get (or create) a named option Group.
Name: name of the option group. Description: long description for –help output. After: name of other group, used for ordering –help output. The returned group object has an
addoption
method with the same signature asparser.addoption
but will be shown in the respective group in the output ofpytest. --help
.
-
addoption
(*opts, **attrs)[source]¶ register a command line option.
Opts: option names, can be short or long options. Attrs: same attributes which the add_option()
function of the argparse library accepts.After command line parsing options are available on the pytest config object via
config.option.NAME
whereNAME
is usually set by passing adest
attribute, for exampleaddoption("--long", dest="NAME", ...)
.
-
addini
(name, help, type=None, default=None)[source]¶ register an ini-file option.
Name: name of the ini-variable Type: type of the variable, can be pathlist
,args
orlinelist
.Default: default value if no ini-file option exists but is queried. The value of ini-variables can be retrieved via a call to
config.getini(name)
.
-
-
class
Node
[source]¶ base class for Collector and Item the test collection tree. Collector subclasses have children, Items are terminal nodes.
-
name
= None¶ a unique name within the scope of the parent node
-
parent
= None¶ the parent collector node.
-
config
= None¶ the pytest config object
-
session
= None¶ the session this node is part of
-
fspath
= None¶ filesystem path where this node was collected from (can be None)
-
keywords
= None¶ keywords/markers collected from all scopes
-
extra_keyword_matches
= None¶ allow adding of extra keywords to use for matching
-
ihook
¶ fspath sensitive hook proxy used to call pytest hooks
-
nodeid
¶ a ::-separated string denoting its collection tree address.
-
listchain
()[source]¶ return list of all parent collectors up to self, starting from root of collection tree.
-
add_marker
(marker)[source]¶ dynamically add a marker object to the node.
marker
can be a string or pytest.mark.* instance.
-
get_marker
(name)[source]¶ get a marker object from this node or None if the node doesn’t have a marker with that name.
-
-
class
Collector
[source]¶ Bases:
_pytest.main.Node
Collector instances create children through collect() and thus iteratively build a tree.
-
exception
CollectError
[source]¶ Bases:
Exception
an error during collection, contains a custom message.
-
exception
-
class
Item
[source]¶ Bases:
_pytest.main.Node
a basic test invocation item. Note that for a single function there might be multiple test invocation items.
-
class
Module
[source]¶ Bases:
_pytest.main.File
,_pytest.python.PyCollector
Collector for test classes and functions.
-
class
Function
[source]¶ Bases:
_pytest.python.FunctionMixin
,_pytest.main.Item
,_pytest.python.FuncargnamesCompatAttr
a Function Item is responsible for setting up and executing a Python test function.
-
function
¶ underlying python ‘function’ object
-
-
class
CallInfo
[source]¶ Result/Exception info a function invocation.
-
when
= None¶ context of invocation: one of “setup”, “call”, “teardown”, “memocollect”
-
excinfo
= None¶ None or ExceptionInfo object.
-
-
class
TestReport
[source]¶ Basic test report object (also used for setup and teardown calls if they fail).
-
nodeid
= None¶ normalized collection node id
-
location
= None¶ a (filesystempath, lineno, domaininfo) tuple indicating the actual location of a test item - it might be different from the collected one e.g. if a method is inherited from a different module.
-
keywords
= None¶ a name -> value dictionary containing all keywords and markers associated with a test invocation.
-
outcome
= None¶ test outcome, always one of “passed”, “failed”, “skipped”.
-
longrepr
= None¶ None or a failure representation.
-
when
= None¶ one of ‘setup’, ‘call’, ‘teardown’ to indicate runtest phase.
-
sections
= None¶ list of (secname, data) extra information which needs to marshallable
-
duration
= None¶ time it took to run just the test
-