A pytest plugin for automatically updating doctest outputs
APACHE-2.0 License
pytest-accept is a pytest plugin for automatically updating doctest outputs. It runs doctests, observes the generated outputs, and writes them to the doctests' documented outputs.
It's designed for a couple of audiences:
pytest-accept is decoupled from the doctests it works with — it can be used with existing doctests, and the doctests it edits are no different from normal doctests.
Here's an example of what pytest-accept does: given a file like
add.py
containing an incorrect documented output:
def add(x, y):
"""
Adds two values.
>>> add(1, 1)
3
>>> add("ab", "c")
'bac'
"""
return x + y
...running doctests using pytest and passing --accept
replaces the existing
incorrect values with correct values:
pytest --doctest-modules examples/add.py --accept
diff --git a/examples/add.py b/examples/add.py
index 10a71fd..c2c945f 100644
--- a/examples/add.py
+++ b/examples/add.py
@@ -3,10 +3,10 @@ def add(x, y):
Adds two values.
>>> add(1, 1)
- 3
+ 2
>>> add("ab", "c")
- 'bac'
+ 'abc'
"""
return x + y
This style of testing is fairly well-developed in some languages, although still doesn't receive the attention I think it deserves, and historically hasn't had good support in python.
Confusingly, it's referred to "snapshot testing" or "regression testing" or "expect testing" or "literate testing" or "acceptance testing". The best explanation I've seen on this testing style is from @yminsky in a Jane Street Blogpost. @matklad also has an excellent summary in his blog post How to Test.
pip install pytest-accept
A previous effort in assert_plugin.py
attempted to do this for assert
statements, and the file contains some notes
on the effort. The biggest problem is pytest stops on the first assert
failure
in each test, which is very limiting. (Whereas pytest can be configured to
continue on doctest failures, which this library takes advantage of.)
It's probably possible to change pytest's behavior here, but it's a significant effort on the pytest codebase.
Some alternatives:
accept(result, "abc")
,assert
tests. And one of the great elegances of pytest is itsassert
statement.Nothing ground-breaking! Some notes:
If a docstring uses escape characters such as \n
, python will interpret them
as the escape character rather than the literal. Use a raw string to have it
interpreted as a literal. e.g. this fails:
def raw_string():
"""
>>> "\n"
'\n'
"""
but succeeds with:
def raw_string():
- """
+ r"""
>>> "\n"
'\n'
Possibly pytest-accept could do more here — e.g. change the format of the docstring. But that would not be trivial to implement, and may be too invasive.
The library attempts to confirm the file hasn't changed between the start and
end of the test and won't overwrite the file where it detects there's been a
change. This can be helpful for workflows where the tests run repeatedly in
the background (e.g. using something like
watchexec) while a person is working
on the file, or when the tests take a long time, maybe because of --pdb
. To
be doubly careful, passing --accept-copy
will cause the plugin to instead
create a file named {file}.py.new
rather than overwriting the file on any
doctest failure.
--accept-copy
to be conservative.This is still fairly early, has mostly been used by me & xarray and there may be some small bugs. Let me know anything at all and I'll attempt to fix them.
It currently doesn't affect the printing of test results; the doctests will still print as failures.
Python's doctest library is imperfect:
.*
is an ellipsis ...
, which is also the syntax forpprint(x)
, which is verbose.\
is counted as one, meaning this