Welcome to testbook¶
testbook is a unit testing framework for testing code in Jupyter Notebooks.
Previous attempts at unit testing notebooks involved writing the tests in the notebook itself. However, testbook will allow for unit tests to be run against notebooks in separate test files, hence treating .ipynb
files as .py
files.
Here is an example of a unit test written using testbook
Consider the following code cell in a Jupyter Notebook:
def func(a, b):
return a + b
You would write a unit test using testbook
in a Python file as follows:
from testbook import testbook
@testbook('/path/to/notebook.ipynb', execute=True)
def test_func(tb):
func = tb.ref("func")
assert func(1, 2) == 3
Features¶
Write conventional unit tests for Jupyter Notebooks
Share kernel context across multiple tests (using pytest fixtures)
Inject code into Jupyter notebooks
Works with any unit testing library - unittest, pytest or nose
Documentation¶
Installation and Getting Started¶
testbook
is a unit testing framework for testing code in Jupyter Notebooks.
Installing testbook
¶
pip install testbook
Create your first test¶
Consider the following code cell in a Jupyter Notebook,
def foo(x):
return x + 1
Here is the unit test for it which must be written in a Python module (.py
file).
from testbook import testbook
@testbook('/path/to/notebook.ipynb', execute=True)
def test_foo(tb):
foo = tb.ref("foo")
assert foo(2) == 3
That’s it! You can now execute the test.
General workflow when using testbook to write a unit test¶
Use
testbook.testbook
as a decorator or context manager to specify the path to the Jupyter Notebook. Passingexecute=True
will execute all the cells, and passingexecute=['cell-tag-1', 'cell-tag-2']
will only execute specific cells identified by cell tags.Obtain references to objects under test using the
.ref
method.Write the test!
Usage¶
The motivation behind creating testbook was to be able to write conventional unit tests for Jupyter Notebooks.
How it works¶
Testbook achieves conventional unit tests to be written by setting up references to variables/functions/classes in the Jupyter Notebook. All interactions with these reference objects are internally “pushed down” into the kernel, which is where it gets executed.
Set up Jupyter Notebook under test¶
Decorator and context manager pattern¶
These patterns are interchangeable in most cases. If there are nested decorators on your unit test function, consider using the context manager pattern instead.
Decorator pattern
from testbook import testbook @testbook('/path/to/notebook.ipynb', execute=True) def test_func(tb): func = tb.ref("func") assert func(1, 2) == 3
Context manager pattern
from testbook import testbook def test_func(): with testbook('/path/to/notebook.ipynb', execute=True) as tb: func = tb.ref("func") assert func(1, 2) == 3
Using execute
to control which cells are executed before test¶
You may also choose to execute all or some cells:
Pass
execute=True
to execute the entire notebook before the test. In this case, it might be better to set up a module scoped pytest fixture.Pass
execute=['cell1', 'cell2']
orexecute='cell1'
to only execute the specified cell(s) before the test.Pass
execute=slice('start-cell', 'end-cell')
orexecute=range(2, 10)
to execute all cells in the specified range.
Obtain references to objects present in notebook¶
Testing functions in Jupyter Notebook¶
Consider the following code cell in a Jupyter Notebook:
def foo(name):
return f"You passed {name}!"
my_list = ['spam', 'eggs']
Reference objects to functions can be called with,
explicit JSON serializable values (like
dict
,list
,int
,float
,str
,bool
, etc)other reference objects
@testbook.testbook('/path/to/notebook.ipynb', execute=True)
def test_foo(tb):
foo = tb.ref("foo")
# passing in explicitly
assert foo(['spam', 'eggs']) == "You passed ['spam', 'eggs']!"
# passing in reference object as arg
my_list = tb.ref("my_list")
assert foo(my_list) == "You passed ['spam', 'eggs']!"
Testing function/class returning a non-serializable value¶
Consider the following code cell in a Jupyter Notebook:
class Foo:
def __init__(self):
self.name = name
def say_hello(self):
return f"Hello {self.name}!"
When Foo
is instantiated from the test, the return value will be a reference object which stores a reference to the non-serializable Foo
object.
@testbook.testbook('/path/to/notebook.ipynb', execute=True)
def test_say_hello(tb):
Foo = tb.ref("Foo")
bar = Foo("bar")
assert bar.say_hello() == "Hello bar!"
Support for patching objects¶
Use the patch
and patch_dict
contextmanager to patch out objects during unit test. Learn more about how to use patch
here.
Example usage of patch
:
def foo():
bar()
@testbook('/path/to/notebook.ipynb', execute=True)
def test_method(tb):
with tb.patch('__main__.bar') as mock_bar:
foo = tb.ref("foo")
foo()
mock_bar.assert_called_once()
Example usage of patch_dict
:
my_dict = {'hello': 'world'}
@testbook('/path/to/notebook.ipynb', execute=True)
def test_my_dict(tb):
with tb.patch('__main__.my_dict', {'hello' : 'new world'}) as mock_my_dict:
my_dict = tb.ref("my_dict")
assert my_dict == {'hello' : 'new world'}
Examples¶
Here are some common testing patterns where testbook can help.
Mocking requests library¶
Notebook:
Test:
from testbook import testbook
@testbook('/path/to/notebook.ipynb', execute=True)
def test_get_details(tb):
with tb.patch('requests.get') as mock_get:
get_details = tb.ref('get_details') # get reference to function
get_details('https://my-api.com')
mock_get.assert_called_with('https://my-api.com')
Asserting dataframe manipulations¶
Notebook:
Test:
from testbook import testbook
@testbook('/path/to/notebook.ipynb')
def test_dataframe_manipulation(tb):
tb.execute_cell('imports')
# Inject a dataframe with code
tb.inject(
"""
df = pandas.DataFrame([[1, None, 3], [4, 5, 6]], columns=['a', 'b', 'c'], dtype='float')
"""
)
# Perform manipulation
tb.execute_cell('manipulation')
# Inject assertion into notebook
tb.inject("assert len(df) == 1")
Asserting STDOUT of a cell¶
Notebook:
Test:
from testbook import testbook
@testbook('stdout.ipynb', execute=True)
def test_stdout(tb):
assert tb.cell_output_text(1) == 'hello world!'
assert 'The current time is' in tb.cell_output_text(2)
Reference¶
This part of the documentation lists the full API reference of all public classes and functions.
testbook.client module¶
-
class
testbook.client.
TestbookNotebookClient
(nb, km=None, **kw)¶ Bases:
nbclient.client.NotebookClient
-
cell_execute_result
(cell: Union[int, str]) → List[Dict[str, Any]]¶ Return the execute results of cell at a given index or with a given tag.
Each result is expressed with a dictionary for which the key is the mimetype of the data. A same result can have different representation corresponding to different mimetype.
- Parameters
- Returns
The execute results
- Return type
List[Dict[str, Any]]
- Raises
IndexError – If index is invalid
TestbookCellTagNotFoundError – If tag is not found
-
cell_output_text
(cell) → str¶ Return cell text output
-
property
cells
¶
-
execute
() → None¶ Executes all cells
-
execute_cell
(cell, **kwargs) → Union[Dict, List[Dict]]¶ Executes a cell or list of cells
-
get
(item)¶
-
inject
(code: str, args: List = None, kwargs: Dict = None, run: bool = True, before: Union[str, int, None] = None, after: Union[str, int, None] = None, pop: bool = False) → testbook.testbooknode.TestbookNode¶ Injects and executes given code block
- Parameters
code (str) – Code or function to be injected
args (iterable, optional) – tuple of arguments to be passed to the function
kwargs (dict, optional) – dict of keyword arguments to be passed to the function
run (bool, optional) – Control immediate execution after injection (default is True)
after (before,) – Inject code before or after cell
pop (bool) – Pop cell after execution (default is False)
- Returns
Injected cell
- Return type
TestbookNode
-
patch
(target, **kwargs)¶ Used as contextmanager to patch objects in the kernel
-
patch_dict
(in_dict, values=(), clear=False, **kwargs)¶ Used as contextmanager to patch dictionaries in the kernel
-
ref
(name: str) → Union[testbook.reference.TestbookObjectReference, Any]¶ Return a reference to an object in the kernel
-
value
(code: str) → Any¶ Execute given code in the kernel and return JSON serializeable result.
If the result is not JSON serializeable, it raises TestbookAttributeError. This error object will also contain an attribute called save_varname which can be used to create a reference object with
ref()
.- Parameters
code (str) – This can be any executable code that returns a value. It can be used the return the value of an object, or the output of a function call.
- Returns
- Return type
The output of the executed code
- Raises
-
testbook.exceptions module¶
-
exception
testbook.exceptions.
TestbookAttributeError
¶ Bases:
AttributeError
-
exception
testbook.exceptions.
TestbookCellTagNotFoundError
¶ Bases:
testbook.exceptions.TestbookError
Raised when cell tag is not declared in notebook
-
exception
testbook.exceptions.
TestbookExecuteResultNotFoundError
¶ Bases:
testbook.exceptions.TestbookError
Raised when there is no execute_result
-
exception
testbook.exceptions.
TestbookRuntimeError
(evalue, traceback, eclass=None)¶ Bases:
RuntimeError
-
exception
testbook.exceptions.
TestbookSerializeError
¶ Bases:
testbook.exceptions.TestbookError
Raised when output cannot be JSON serialized
Changelog¶
0.4.1¶
check for errors when
allow_errors
is true
0.4.0¶
Testbook now returns actual object for JSON serializable objects instead of reference objects. Please note that this may break tests written with prior versions.
0.3.0¶
Implemented container methods – len – iter – next – getitem – setitem – contains
Fixed testbook to work with ipykernel 5.5
0.2.6¶
Fixed Python underscore (
_
) issue
0.2.5¶
Fixed testbook decorator.
0.2.4¶
Add
cell_execute_result
toTestbookNotebookClient
Use testbook decorator with pytest fixture and marker
0.2.3¶
Accept notebook node as argument to testbook
Added support for specifying kernel with
kernel_name
kwarg
0.2.2¶
Added support for passing notebook as file-like object or path as str
0.2.1¶
Added support for
allow_errors
0.2.0¶
Changed to new package name
testbook
Supports for patch and patch_dict
Slices now supported for execute patterns
Raises TestbookRuntimeError for all exceptions that occur during cell execution
0.1.3¶
Added warning about package name change
0.1.2¶
Updated docs link in setup.py
0.1.1¶
Unpin dependencies
0.1.0¶
Initial release with basic features