chmp / ipytest Goto Github PK

View Code? Open in Web Editor NEW

311.0 5.0 18.0 751 KB

Pytest in IPython notebooks.

License: MIT License

Jupyter Notebook 33.03% Python 66.97%

jupyter-notebook pytest python ipython-notebook

ipytest's People

Contributors

Stargazers

Watchers

Forkers

alfonsorreyes abingham nyamu01b pya tamuhey ericlundberg bemm-js xhochy adess23 pmeier the-compiler bessy-mukaria xtile volodink pplonski alexandrebrown python-repository-hub ssghost

ipytest's Issues

using ipytest inside databricks

I wish to use ipytest in databricks, but I ran into an error like below.

I notice that this is something related to the databricks' mechanism: https://menziess.github.io/howto/test/code-in-databricks-notebooks/

Is it possible we can introduce ipytest in databricks?

My cells:

%%run_pytest[clean] -qq

def test_get_label_dir():
  assert get_label_dir("/test/daisy") == "daisy"

Error:

TypeError: run_cell() missing 1 required positional argument: 'raw_cell'
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<command-994187> in <module>
----> 1 get_ipython().run_cell_magic('run_pytest[clean]', '-qq', '\ndef test_get_label_dir():\n  assert get_label_dir("/test/daisy") == "daisy"\n')

/databricks/python/lib/python3.7/site-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
   2350             with self.builtin_trap:
   2351                 args = (magic_arg_s, cell)
-> 2352                 result = fn(*args, **kwargs)
   2353             return result
   2354 

</databricks/python/lib/python3.7/site-packages/decorator.py:decorator-gen-120> in run_pytest_clean(self, line, cell)

/databricks/python/lib/python3.7/site-packages/IPython/core/magic.py in <lambda>(f, *a, **k)
    185     # but it's overkill for just that one bit of state.
    186     def magic_deco(arg):
--> 187         call = lambda f, *a, **k: f(*a, **k)
    188 
    189         if callable(arg):

Add tests with nbval to ensure notebooks function

Currently upgrading ipytest requires manaul tests. Use nbval to test the notebooks automatically.

Save pytest return value into ipytest module variable to enable proper testing.

Remove dependency on defining file

Hi. I'm working on a custom pytest kernel for Jupyter Notebooks. It is simply a thin wrapper on the ipython3 kernel and uses your excellent package for all the heavy lifting.

My aim is to make using it transparent, meaning no extra code should be added by the user. I have used injection to achieve most of this goal with the sole exception of the __file__ variable. I have a very crude hack that works around this.

What I was wondering, however, was if it could be possible to remove the dependency on defining __file__ in the notebook.

In particular some way to leverage the fact that I have access to the code being passed in to do_execute() in the kernel so can essentially store all the code in a string if ipytest can digest that directly without having to parse the .ipynb file that it needs to be passed in using __file__.

Automatic assertion rewriting

As noted by in jupyter/notebook#3456 using magics for assertion rewriting is cumbersome. IPython has support for custom AST transforms.

Using IPython's AST transforms is preferable over magics.

Support async tests

Using pytest-asyncio with ipytest and a recent ipykernel version fails with:

RuntimeError: Cannot run the event loop while another loop is running

Reason: the ipython kernel keeps a persistent event loop running. The asyncio plugin tries to use its own event loop. However, event loops cannot be nested.

Added ipytest to awesome-jupyter

Hi @chmp,

JFYI, I added ipytest to awesome-jupyter.

Cool project! In particular, anything that makes converting notebook code to proper tests is awesome!

Cheers,
Markus

Add support to raise exception for non zero pytest return code

Currently pytest errors do not raise an error inside the kernel. To use ipytest with tools, such as papermill or nbval, errors should be propagated.

trying to use in notebook in AWS sagemaker "ImportError: cannot import name 'final' from '_pytest.compat' (/opt/conda/lib/python3.7/site-packages/_pytest/compat.py)"

I'm running the example code in an AWS sagemaker notebook

import ipytest

ipytest.autoconfig()

%%ipytest

# define the tests

def test_my_func():
    assert my_func(0) == 0
    assert my_func(1) == 0
    assert my_func(2) == 2
    assert my_func(3) == 2
    
    
def my_func(x):
    return x // 2 * 2

and I get this error:

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-6-858ce095dc44> in <module>
----> 1 get_ipython().run_cell_magic('ipytest', '', '\n# define the tests\n\ndef test_my_func():\n    assert my_func(0) == 0\n    assert my_func(1) == 0\n    assert my_func(2) == 2\n    assert my_func(3) == 2\n    \n    \ndef my_func(x):\n    return x // 2 * 2 \n')

/opt/conda/lib/python3.7/site-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
   2460             with self.builtin_trap:
   2461                 args = (magic_arg_s, cell)
-> 2462                 result = fn(*args, **kwargs)
   2463             return result
   2464 

/opt/conda/lib/python3.7/site-packages/ipytest/_impl.py in pytest_magic(line, cell)
     85             raise e
     86 
---> 87     run(*shlex.split(line))
     88 
     89 

/opt/conda/lib/python3.7/site-packages/ipytest/_impl.py in run(module, plugins, *args)
     45         *args,
     46         module=module,
---> 47         plugins=plugins,
     48     )
     49 

/opt/conda/lib/python3.7/site-packages/ipytest/_impl.py in run_direct(func, *args, **kwargs)
    276 
    277 def run_direct(func, *args, **kwargs):
--> 278     return func(*args, **kwargs)
    279 
    280 

/opt/conda/lib/python3.7/site-packages/ipytest/_impl.py in _run_impl(module, plugins, *args)
    135     with _prepared_env(module) as filename:
    136         full_args = _build_full_args(args, filename)
--> 137         return pytest.main(full_args, plugins=[*plugins, FixProgramNamePlugin()])
    138 
    139 

/opt/conda/lib/python3.7/site-packages/_pytest/config/__init__.py in main(args, plugins)
     71 Plugins can be any namespace, so we can't narrow it down much, but we use an
     72 alias to make the intent clear.
---> 73 
     74 Ideally this type would be provided by pluggy itself.
     75 """

/opt/conda/lib/python3.7/site-packages/_pytest/config/__init__.py in _prepareconfig(args, plugins)
    212 
    213 def directory_arg(path: str, optname: str) -> str:
--> 214     """Argparse type validator for directory arguments.
    215 
    216     :path: Path of directory.

/opt/conda/lib/python3.7/site-packages/_pytest/config/__init__.py in get_config(args, plugins)
    178 
    179 
--> 180 def console_main() -> int:
    181     """The CLI entry point of pytest.
    182 

/opt/conda/lib/python3.7/site-packages/_pytest/config/__init__.py in __init__(self, pluginmanager, invocation_params)
    733         )
    734         if self.is_blocked(modname) or self.get_plugin(modname) is not None:
--> 735             return
    736 
    737         importspec = "_pytest." + modname if modname in builtin_plugins else modname

/opt/conda/lib/python3.7/site-packages/_pytest/config/argparsing.py in <module>
     17 
     18 import _pytest._io
---> 19 from _pytest.compat import final
     20 from _pytest.config.exceptions import UsageError
     21 from _pytest.deprecated import ARGUMENT_PERCENT_DEFAULT

ImportError: cannot import name 'final' from '_pytest.compat' (/opt/conda/lib/python3.7/site-packages/_pytest/compat.py)

any ideas?

ipytest doesn't seem to work with Python 2.7

In PyPI, this package is listed as supporting Python 2, but when I pip install it with Python 2.7 and run "import ipytest", I get the following error:

File "lib\site-packages\ipytest_util.py", line 73
print(message, file=sys.stderr)
^
SyntaxError: invalid syntax

Not sure it makes sense to actually bother fixing this since Python 2.7 support will be ending in the near future, but it might make sense to officially remove support.

Add a proper signature to ipytest.config

Using ipytest.config to configure ipytest has the downside that it is hard to know the available options, as the signature uses kwargs. The signature should be fixed to contain all options.

Running with ipython 8 results in crash

Issue

Usint ipytest with latest ipython 8.0.1 results in program crash with ModuleNotFoundError.

Example

Simple test notebook

ipytest_nb.ipynb

import ipytest
ipytest.autoconfig()

def test_fail():
    assert False
ipytest.run()

IPython 7.23.1

(venv) rschulze@red: (dev *)~/development/pyshl$ ipython --TerminalIPythonApp.file_to_run=ipytest_nb.ipynb
F                                                                                            [100%]
============================================= FAILURES =============================================
____________________________________________ test_fail _____________________________________________

    def test_fail():
>       assert False
E       assert False

<ipython-input-1-ff73e06d7454>:2: AssertionError
===================================== short test summary info ======================================
FAILED tmp0n4g6iqq.py::test_fail - assert False
1 failed in 0.05s

IPython 8.0.1

(venv) rschulze@red: (dev *)~/development/pyshl$ ipython --TerminalIPythonApp.file_to_run=ipytest_nb.ipynb

ModuleNotFoundError                       Traceback (most recent call last)
~/development/pyshl/venv/lib/python3.8/site-packages/runipy/notebook_runner.py in <module>
     26         # IPython 3
---> 27         from IPython.kernel import KernelManager
     28         from IPython.nbformat import NotebookNode

ModuleNotFoundError: No module named 'IPython.kernel'

[...]

Solution

It seems that following replacements all-over the code solve the issue:

from IPython.nbformat import... -> from nbformat import...
from IPython.kernel import... -> jupyter_client import...

Probably have to be embedded in some try:... except:... for backwards compatibility.

run_cell() missing 1 required positional argument: 'raw_cell'

Hi,
I installed ipytest, and ran the following in one cell

import ipytest
ipytest.autoconfig()

which was successful. But the first of the following lines gave me an error:

%%run_pytest[clean] -qq

def test_example():
assert [1, 2, 3] == [1, 2, 3]

I also tried %%run_pytest[clean]

The error I get is this:
run_cell() missing 1 required positional argument: 'raw_cell'

with the following detail:

TypeError Traceback (most recent call last)
in
----> 1 get_ipython().run_cell_magic('run_pytest[clean]', '-qq', '\ndef test_example():\n assert [1, 2, 3] == [1, 2, 3]\n')

/databricks/python/lib/python3.7/site-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
2360 with self.builtin_trap:
2361 args = (magic_arg_s, cell)
-> 2362 result = fn(*args, **kwargs)
2363 return result
2364

</databricks/python/lib/python3.7/site-packages/decorator.py:decorator-gen-123> in run_pytest_clean(self, line, cell)

/databricks/python/lib/python3.7/site-packages/IPython/core/magic.py in (f, *a, **k)
185 # but it's overkill for just that one bit of state.
186 def magic_deco(arg):
--> 187 call = lambda f, *a, **k: f(*a, **k)
188
189 if callable(arg):

/databricks/python/lib/python3.7/site-packages/ipytest/_pytest_support.py in run_pytest_clean(self, line, cell)
258
259 clean_tests(items=main.dict)
--> 260 return self.run_pytest(line, cell)
261
262 @cell_magic

</databricks/python/lib/python3.7/site-packages/decorator.py:decorator-gen-124> in run_pytest(self, line, cell)

/databricks/python/lib/python3.7/site-packages/ipytest/_pytest_support.py in run_pytest(self, line, cell)
268
269 else:
--> 270 self.shell.run_cell(cell)
271
272 import ipytest

TypeError: run_cell() missing 1 required positional argument: 'raw_cell'

UsageError: Line magic function `%%run_pytest` not found.

I installed ipytest with pip in my virtualenv and then created a Jupyter notebook with Visual Studio Code:

import ipytest
ipytest.autoconfig()

%%run_pytest[clean] -qq 

def test_example():
    assert [1, 2, 3] == [1, 2, 3]

But I'm getting an error that the magic function isn't being found:

UsageError: Line magic function `%%run_pytest` not found.

Notes:

The run_pytest cell magic is visible in the list returned by %lsmagic.
I get the same error when I use ipython.

How to run doctest with cell magic command?

Hi, thanks for this great project!

I want to run doctest with cell magic, as follows:

%%run_pytest --doctest-modules
def foo(x):
    """
    >>> foo(2)
    4
    """
    return x*2

However, the output says no tests found:

============================= test session starts ==============================
platform darwin -- Python 3.6.9, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /Users/yohei_tamura/work/men
collected 0 items

========================= no tests ran in 0.02 seconds =========================

Upload sdists

We build the conda-forge packages for Python of sdist files, our auto-update bot is failing as the latest release only has a wheel.

Deprecation warning with pytest 5.4.1

Hi,

First of all a big thank you for implementing this package! This was the last missing bit for working productively in a notebook (together with nbdev).

For my installation of pytest 5.4.1 I get a deprecation warning during my first test with the example

%%run_pytest[clean] -qq

def test_example():
    assert [1, 2, 3] == [1, 2, 3]

full warning:

.                                                                                      [100%]
===================================== warnings summary ======================================
<conda-env-path>\lib\site-packages\ipytest\_pytest_support.py:110
  <conda-env-path>\lib\site-packages\ipytest\_pytest_support.py:110: PytestDeprecationWarning: direct construction of Module has been deprecated, please use Module.from_parent
    return Module(path=path, parent=parent, module=self.module)

-- Docs: https://docs.pytest.org/en/latest/warnings.html

This is nothing urgent or blocking but I wanted to let you know about this for future updates to this project.

Add nbmake to related projects

https://github.com/treebeardtech/nbmake

warning when running tests if have "import dash" run beforehand

If I run import dash before running any tests, the following warning will appear:
PytestAssertRewriteWarning: Module already imported so cannot be rewritten: dash

I assume this is enough to reproduce this issue. Feel free to contact if you need more information :)

Trouble with Python 3.6.6 kernel in Jupyter Notebook

Using a Python 3.6.6 kernl in Jupyter Notebook seems to affect test collection:

import ipytest.magics

__file__ = 'Benchmark.ipynb'

%%run_pytest[clean] -v

def test_demo():
    assert 2 == int('2')

============================================================== test session starts ===============================================================
platform linux -- Python 3.6.6, pytest-3.7.4, py-1.7.0, pluggy-0.8.0 -- /home/stefano/venvs/benchmark/bin/python
cachedir: .pytest_cache
rootdir: /home/stefano/Documents/notebooks, inifile:
collecting ... collected 0 items

========================================================== no tests ran in 0.02 seconds ==========================================================

When switching to a Python 2.7 kernel test are run as expected

============================================================== test session starts ===============================================================
platform linux2 -- Python 2.7.15rc1, pytest-3.7.4, py-1.6.0, pluggy-0.7.1 -- /home/stefano/venvs/test-proj/bin/python
cachedir: .pytest_cache
rootdir: /home/stefano/Documents/notebooks, inifile:
plugins: repeat-0.7.0, ordering-0.5
collecting ... collected 1 item

Benchmark.py::test_demo <- <ipython-input-5-9b87b157f532> PASSED                                                                           [100%]

Supress pytest rewrite warnings for plugins

When running ipytest multiple times, the rewrite warnings for plugins in pytest are triggered.

I raised the issue upstream: pytest-dev/pytest#5432

Depending on the repsonse, try to work around locally. My current ideas:

pytest-dev/pytest#4439
try restoring self._rewritten_names before each run

Doesn't work with Google Colab

Running this small test on Google Colab

%%run_pytest[clean] -qq

def test_example():
    assert [1, 2, 3] == [1, 2, 3]

results in exception:

/usr/local/lib/python3.6/dist-packages/pluggy/hooks.py:258: in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
/usr/local/lib/python3.6/dist-packages/pluggy/manager.py:67: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
/usr/local/lib/python3.6/dist-packages/pluggy/manager.py:61: in <lambda>
    firstresult=hook.spec_opts.get('firstresult'),
/usr/local/lib/python3.6/dist-packages/ipytest/_pytest_support.py:143: in pytest_collect_file
    parent, fspath=path.new(ext=".py"), module=self.module
/usr/local/lib/python3.6/dist-packages/ipytest/_pytest_support.py:156: in from_parent
    self = super().from_parent(parent, fspath=fspath)
E   AttributeError: 'super' object has no attribute 'from_parent'

database access with pytest-django

I'm running into some problems with Django database access. pytest-django does some trickery with the database so that it can roll back any changes made by a test to the test database. The django_db mark informs pytest that a test will be accessing the test database.

%%run_pytest -s

import pytest
import ipytest

from django.contrib.contenttypes.models import ContentType

@pytest.mark.django_db
def test_django():
    for content_type in ContentType.objects.all():
        print(content_type)

    assert True

Here's the error I'm getting:

E
==================================== ERRORS ====================================
________________________ ERROR at setup of test_django _________________________

self = <django.db.backends.postgresql.base.DatabaseWrapper object at 0x7fb4c35ec730>

    @contextmanager
    def _nodb_cursor(self):
        """
        Return a cursor from an alternative connection to be used when there is
        no need to access the main database, specifically for test db
        creation/deletion. This also prevents the production database from
        being exposed to potential child threads while (or after) the test
        database is destroyed. Refs #10868, #17786, #16969.
        """
        conn = self.__class__({**self.settings_dict, 'NAME': None}, alias=NO_DB_ALIAS)
        try:
>           with conn.cursor() as cursor:

/opt/alex/pyvenv/lib/python3.8/site-packages/django/db/backends/base/base.py:620:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

args = (<django.db.backends.postgresql.base.DatabaseWrapper object at 0x7fb4bf2930a0>,)
kwargs = {}
event_loop = <_UnixSelectorEventLoop running=True closed=False debug=False>

    @functools.wraps(func)
    def inner(*args, **kwargs):
        if not os.environ.get('DJANGO_ALLOW_ASYNC_UNSAFE'):
            # Detect a running event loop in this thread.
            try:
                event_loop = asyncio.get_event_loop()
            except RuntimeError:
                pass
            else:
                if event_loop.is_running():
>                   raise SynchronousOnlyOperation(message)
E                   django.core.exceptions.SynchronousOnlyOperation: You cannot call this from an async context - use a thread or sync_to_async.

/opt/alex/pyvenv/lib/python3.8/site-packages/django/utils/asyncio.py:24: SynchronousOnlyOperation

During handling of the above exception, another exception occurred:

request = <SubRequest '_django_db_marker' for <Function test_django>>

    @pytest.fixture(autouse=True)
    def _django_db_marker(request):
        """Implement the django_db marker, internal to pytest-django.

        This will dynamically request the ``db``, ``transactional_db`` or
        ``django_db_reset_sequences`` fixtures as required by the django_db marker.
        """
        marker = request.node.get_closest_marker("django_db")
        if marker:
            transaction, reset_sequences = validate_django_db(marker)
            if reset_sequences:
                request.getfixturevalue("django_db_reset_sequences")
            elif transaction:
                request.getfixturevalue("transactional_db")
            else:
>               request.getfixturevalue("db")

/opt/alex/pyvenv/lib/python3.8/site-packages/pytest_django/plugin.py:513:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/opt/alex/pyvenv/lib/python3.8/site-packages/pytest_django/fixtures.py:105: in django_db_setup
    db_cfg = setup_databases(
/opt/alex/pyvenv/lib/python3.8/site-packages/django/test/utils.py:170: in setup_databases
    connection.creation.create_test_db(
/opt/alex/pyvenv/lib/python3.8/site-packages/django/db/backends/base/creation.py:55: in create_test_db
    self._create_test_db(verbosity, autoclobber, keepdb)
/opt/alex/pyvenv/lib/python3.8/site-packages/django/db/backends/base/creation.py:176: in _create_test_db
    with self._nodb_cursor() as cursor:
/opt/alex/pyvenv/build/Python-3.8.3/lib/python3.8/contextlib.py:113: in __enter__
    return next(self.gen)
/opt/alex/pyvenv/lib/python3.8/site-packages/django/db/backends/postgresql/base.py:298: in _nodb_cursor
    with super()._nodb_cursor() as cursor:
/opt/alex/pyvenv/build/Python-3.8.3/lib/python3.8/contextlib.py:113: in __enter__
    return next(self.gen)
/opt/alex/pyvenv/lib/python3.8/site-packages/django/db/backends/base/base.py:623: in _nodb_cursor
    conn.close()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

args = (<django.db.backends.postgresql.base.DatabaseWrapper object at 0x7fb4bf2930a0>,)
kwargs = {}
event_loop = <_UnixSelectorEventLoop running=True closed=False debug=False>

    @functools.wraps(func)
    def inner(*args, **kwargs):
        if not os.environ.get('DJANGO_ALLOW_ASYNC_UNSAFE'):
            # Detect a running event loop in this thread.
            try:
                event_loop = asyncio.get_event_loop()
            except RuntimeError:
                pass
            else:
                if event_loop.is_running():
>                   raise SynchronousOnlyOperation(message)
E                   django.core.exceptions.SynchronousOnlyOperation: You cannot call this from an async context - use a thread or sync_to_async.

/opt/alex/pyvenv/lib/python3.8/site-packages/django/utils/asyncio.py:24: SynchronousOnlyOperation
=========================== short test summary info ============================
ERROR tmp55mtb4ih.py::test_django - django.core.exceptions.SynchronousOnlyOpe...
1 error in 0.21s

Tweak the defaults of autoconfig

Add new register mode "auto", that registers the module only for temporary filenames. With these changes async tests and doc tests will work out of the box.

Consider use the defaults:

run_in_thread=True
register_module="auto"

Errors while running the example notebook

I get the following error when I run the example notebook

==================================== ERRORS ==================================== ________________________ ERROR collecting test session _________________________ ../../.conda/envs/regression_env/lib/python3.7/site-packages/pluggy/hooks.py:286: in __call__ return self._hookexec(self, self.get_hookimpls(), kwargs) ../../.conda/envs/regression_env/lib/python3.7/site-packages/pluggy/manager.py:93: in _hookexec return self._inner_hookexec(hook, methods, kwargs) ../../.conda/envs/regression_env/lib/python3.7/site-packages/pluggy/manager.py:87: in <lambda> firstresult=hook.spec.opts.get("firstresult") if hook.spec else False, ../../.conda/envs/regression_env/lib/python3.7/site-packages/ipytest/_pytest_support.py:143: in pytest_collect_file parent, fspath=path.new(ext=".py"), module=self.module ../../.conda/envs/regression_env/lib/python3.7/site-packages/ipytest/_pytest_support.py:156: in from_parent self = super().from_parent(parent, fspath=fspath) E AttributeError: 'super' object has no attribute 'from_parent' !!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!

Cleaning tests does not affect to test classes named as Test*

If I have something like this:

cell 1

class TestClass:
    def test_1(self):
        pass

cell 2

%%run_pytest[clean] '-v'

def test_2():
    pass

Running these in order will execute both test_1 and test_2. I'd expect that only test_2 is executed. I guess this originates from here: https://github.com/chmp/ipytest/blob/master/ipytest/__init__.py#L60

Add 'how to run a specific test' to the docs

Supposedly this runs a specific test:

import ipytest
ipytest.autoconfig()

def test1():
    assert 2*2 == 5
    
ipytest.config(defopts=False)
ipytest.run('{MODULE}::test1')

(btw, is there any way to pass defopts to run directly?)

and this runs a subset of tests with a certain substring in the name:

import ipytest
ipytest.autoconfig()

def test_this1():
    assert 2*2 == 11

def test_this2():
    assert 2*2 == 12
    
def test_that1():
    assert 2*2 == 21
    
def test_that2():
    assert 2*2 == 22    
    
ipytest.run('-k', 'this')

It was not obvious to figure it out :)

I would suggest adding it to the docs.

Question about running in Jypiter notebooks

@chmp - from you comment in the issue logged in the pytest repo:
pytest-dev/pytest#2268 (comment)

In both cases, the asserts will be pytest flavoured. You can execute test_a either in the cell it was defined in or anywhere else.

I'm new to using Python / Jupyiter and found this entry, but I'm sure I'm doing something wrong because I can't get the simple examples to work. Using the first example, I get stuck after the second cell with

unknown magic command '%rewrite_asserts'
UnknownMagic: unknown magic command '%rewrite_asserts'

More generally - I'm not sure how to 'execute' the test. Whether this means just calling the function normally, since using %%run_pytest will result in the same error above, but calling the function does not return especially helpful results.

I found the Example.ipynb file and running them provides the a different error when I run ipytest.run('-qq')

ERROR: not found: /mnt/tmp/8347656836271596575
(no name '/mnt/tmp/8347656836271596575' in any of [])

For context, I'm running the jupyter notebook on and AWS EMR instance running a PySpark kernel. Relevant installed versions:

ipytest                    0.7.1  
ipython                    7.11.1 
ipython-genutils           0.2.0  
pytest                     5.3.4

Thanks for your help.

always_clean option?

Before I open another pull request, I was wondering what you think about adding an always_clean=True option flag (False by default) which makes %%run_pytest equivalent to %%run_pytest[clean]?

For some context, I'd like to start using ipytest for the pytest trainings I teach, where the behavior of re-running all previous tests isn't very helpful - instead, for demos/exercises, only the current snippet is relevant usually.

Perhaps there could also be a %%run_pytest[noclean] (or [dirty] or [continue] or whatever?) to override the default when needeed.

Missing output in pytest magics

When I run a cell using the new magics, I don't see the complete output that I expect from pytest. On the command line pytest typically prints detailed information about an assertion, but all the magic prints is E AssertionError.

For example, given this function:

def test_sorted():
    assert sorted([4, 2, 1, 6]) == [1, 2, 3, 4]

the command line shows this:

sixtynorth@TK421% pytest -qq pytests.py 
F
=============================================== FAILURES ===============================================
_____________________________________________ test_sorted ______________________________________________

    def test_sorted():
>       assert sorted([4, 2, 1, 6]) == [1, 2, 3, 4]
E       assert [1, 2, 4, 6] == [1, 2, 3, 4]
E         At index 2 diff: 4 != 3
E         Full diff:
E         - [1, 2, 4, 6]
E         ?         ---
E         + [1, 2, 3, 4]
E         ?        +++

pytests.py:2: AssertionError

but the cell magic just shows:

F
=============================================== FAILURES ===============================================
_____________________________________________ test_sorted ______________________________________________

    def test_sorted():
>       assert sorted([4, 2, 1, 6]) == [1, 2, 3, 4]
E       AssertionError

<ipython-input-3-f12e7a50e0c6>:3: AssertionError

This makes it hard to demo one of pytest's strengths, it's ability to introspect assertions and pinpoint the problem.

I looked through the code a bit, but nothing struck me as an obvious culprit. Is there something I can do to make this work with flags to run_pytest? Or is there a code change you can make to fix this?

Create a better Example notebook

Currently the example notebook is more geared towards understanding the implementaiton details, than how to use ipytest effectively.

Replace `mddocs` with `minidocs`

Replace the old docutils based documentation with a direct Markdown export. Use minidoc for this.

Return exit code from ipytest.run

Makes it easer to check for errors when running the tests. See #62

`@pytest.mark.parameterize` gives misleading error messages

The following code fails with a misleading error message:

%%run_pytest[clean]
import pytest

@pytest.mark.parameterize('test_input,expected', [("3+5", 8)])
def test_eval(test_input, expected):
    assert eval(test_input) == expected

The error message is:

E                                                                        [100%]
==================================== ERRORS ====================================
________________ ERROR at setup of test_temperature_conversion _________________
file <ipython-input-22-dcba670f399e>, line 3
  @pytest.mark.parameterize("test_input,expected",
                            [
                                (100, 212)
                            ]
                           )
  def test_temperature_conversion(test_input, expected):
E       fixture 'test_input' not found
>       available fixtures: _session_faker, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, dash_br, dash_duo, dash_process_server, dash_thread_server, dashr, dashr_server, doctest_namespace, faker, monkeypatch, no_cover, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

The error message is misleading because the cause is actually that parameterize has a different spelling to the parametrize spelling expected by pytest.

This is an easy mistake to make and, with the above error, it can be difficult to understand how to solve the problem. According to various dictionaries (e.g. https://www.merriam-webster.com/dictionary/parameterize), the verbs "parameterize" and "parametrize" are variants. A quick Google search turns up 1.97 million hits for "parameterize" versus 763,000 hits for "parametrize".

For reference, using pytest by itself (without ipytest) gives a much more useful error message:

test_eval.py:3: in <module>
    @pytest.mark.parameterize('test_input,expected', [("3+5", 8)])
E   Failed: Unknown 'parameterize' mark, did you mean 'parametrize'?

Get TypeError upon (otherwise successful) use of pytest.mark.parametrize

May be related to #39

I can run tests using pytest.mark.parametrize, but jupyter gives me a TypeError:

%%run_pytest[clean]


@pytest.mark.parametrize(
    'text,last_text_to_exclude,expected',
    [
        (
            '''Dear colleagues, thank you for taking the time to respond to this short survey. \n
            Thank you very much! - Ranks - Topics of the highest priority & interest - 
            Storage & Querying (Prepare) - Rank''',
            'Topics of the highest priority & interest - ', 'Storage & Querying (Prepare) - Rank'
        )
    ]
)
def test_extract_text_after(text, last_text_to_exclude, expected):
    text = (
        'Dear colleagues, thank you for taking the time to respond to this short survey. \n'
        'Thank you very much! - Ranks - Topics of the highest priority & interest - '
        'Storage & Querying (Prepare) - Rank'
    )
    last_text_to_exclude = 'Topics of the highest priority & interest - '
    res = extract_text_after(text, last_text_to_exclude)
    expected = 'Storage & Querying (Prepare) - Rank'
    assert res == expected
    
try:
    test_extract_text_after()
except TypeError as err:
    print(err)

Cell output

test_extract_text_after() missing 3 required positional arguments: 'text', 'last_text_to_exclude', and 'expected'
.                                                                                                                               [100%]
1 passed in 0.01s

Here are some possibly relevant version of my python environment:

ipykernel==5.5.0
ipytest==0.9.1
ipython==7.20.0
ipython-genutils==0.2.0
ipywidgets==7.6.3
...
jupyter==1.0.0
jupyter-client==6.1.11
jupyter-console==6.2.0
jupyter-core==4.7.1
jupyterlab-pygments==0.1.2
jupyterlab-widgets==1.0.0
...
pytest==6.2.2

with python Python 3.8.3

Error Executing Ipytest with Anaconda

Hi, I am trying to run ipytest on a notebook in Jupyter lab on AWS server -

import pytest
import ipytest
ipytest.autoconfig()

def my_func(x):
    return x // 2 * 2

def test_my_func():
    assert my_func(0) == 0
    assert my_func(1) == 0
    assert my_func(2) == 2
    assert my_func(3) == 2

ipytest.run('-qq')

I get error AttributeError: 'super' object has no attribute 'from_parent'. When I run the same code on a local Jupyter notebook this runs successfully.

==================================== ERRORS ====================================
________________________ ERROR collecting test session _________________________
/opt/anaconda3/lib/python3.7/site-packages/pluggy/hooks.py:286: in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
/opt/anaconda3/lib/python3.7/site-packages/pluggy/manager.py:92: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
/opt/anaconda3/lib/python3.7/site-packages/pluggy/manager.py:86: in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
ipytest/ipytest/_pytest_support.py:144: in pytest_collect_file
    parent, fspath=path.new(ext=".py"), module=self.module
ipytest/ipytest/_pytest_support.py:157: in from_parent
    self = super(pytest.Module).from_parent(parent, fspath=fspath)
E   AttributeError: 'super' object has no attribute 'from_parent'
!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!

Program name in error messages is confusing

As reported in #57:

I got an "unrecognized arguments" error from pytest once when trying to use --durations=5 --setup-show or so, but later
couldn't reproduce it - I suppose it was just a typo I didn't see at the time (unfortunately I did overwrite the original cell).

It took me a bit to realize that the error actually is coming from pytest, as it looks like this:
ERROR: usage: ipykernel_launcher.py [options] [file_or_dir] [file_or_dir] [...]
ipykernel_launcher.py: error: unrecognized arguments: --wrong
probably since sys.argv[0] is ipykernel_launcher.py rather than pytest. I guess that's not specific to the prerelease though,
and I'm not sure if there's anything that can be done about it anyways.

Example notebook not loading in GitHub.

It spins the wheel for a while and then reports Sorry, something went wrong. Reload?.
Tested with Firefox 62.0.3

Add config option to set base command line arguments

Typically, all calls to ipytest will containt the same commandline arguments. By allowing configuration of the default options, subsequent calls will be shorter.

Jupyter magics

Thanks for your work on this! I've been using ipytest for a presentation I'm working on, and it's great. I've put together a jupyter magic that lets me use it more naturally. You can see it here.

For the most part, ipytest seems to work well. One issue I've found is that ipytest.clean_tests() doesn't always seem to work. In particular, if I re-run cells with my magic in jupyter, pytest says it's running more tests than the cell contains (i.e. test functions from other cells, the contents of which should have been cleaned). You can see this by running this notebook and running the code cells twice. (Note that you'll need the magics from that repo and ipytest along with the notebook.) The real problem may lie with what I'm doing in the magic function, but I'm not sure right now.

Anyhow, what I really want to do is start a discussion with you about ipytest and magics in general. Are you interested in integrating magics into ipytest? If so, what are your thoughts on how they should work? And even if not, so you have any sense of why ipytest is displaying the behavior I'm seeing?

warning with pytest >= 6.0

The following warning appears with corresponding link also, but I don't know how to suppress it. Could you give some advice or fix it?

##vso[task.logissue type=warning;]The 'junit_family' default value will change to 'xunit2' in pytest 6.0. See:
https://docs.pytest.org/en/stable/deprecations.html#junit-family-default-value-change-to-xunit2
for more information.

deprecated direct construction of Module (py37)

Hi, thanks for this project!

/opt/conda/lib/python3.7/site-packages/_pytest/config/__init__.py:935
  /opt/conda/lib/python3.7/site-packages/_pytest/config/__init__.py:935: PytestAssertRewriteWarning: Module already imported so cannot be rewritten: celery
    self._mark_plugins_for_rewrite(hook)

/opt/conda/lib/python3.7/site-packages/ipytest/_pytest_support.py:141
  /opt/conda/lib/python3.7/site-packages/ipytest/_pytest_support.py:141: PytestDeprecationWarning: direct construction of Module has been deprecated, please use Module.from_parent
    return Module(path=path, parent=parent, module=self.module)

-- Docs: https://docs.pytest.org/en/latest/warnings.html

Pre-release shows duplicated collection

Tests are collected multipled times.

# cell 1
import ipytest
ipytest.autoconfig()

# cell 2
%%run_pytest[clean]

def test_bar():
    pass

def test_foo():
    pass

# runs 4 tests

# cell 1
import ipytest
ipytest.autoconfig(defopts=False)

# cell 2
%%run_pytest[clean]

def test_bar():
    pass

def test_foo():
    pass

# runs 2 tests

Somehow the ".py" suffix seems to interact with pytest's test collection. Can also be reproduced with 0.9.1 and register_module=True.

run test error when having python locally and notebooks in remote server

I'm not sure what caused the following issue. I have python installed and run on a local PC, and have notebooks on a remote server.

I have __file__ correctly defined and try to have a sample test using %%run_pytest[clean] -q, the following error will appear. If I run it again, it will run as usual.

ERROR: not found: <remote_path_to_notebook>\some.ipynb
(no name '<remote_path_to_notebook>\some.ipynb' in any of [])

This issue didn't appear when I also have notebooks locally.

It's just a little annoying, but it would be nice if we could fix it.

using:

notebook 6.0.3
pytest 5.4.2
ipytest 0.8.1

Latest release has an undeclared IPython depencency

See the failing CI in the conda recipe: conda-forge/ipytest-feedstock#2

Either this should be lazily imported or ipython should be listed in the setup.py.

Improve doc string for %%ipytest?

Include the help text current included in the readme and use it inside the readme.

How to run a single test without any magics?

I want to run something like ipytest.run(test_name)

Is something like this possible?

I'm teaching a class so I don't want anything fancy like magics confusing the students ...

Package not installable from sdist

Some files are missing, see

Non-user install because user site-packages disabled
Ignoring indexes: https://pypi.org/simple
Created temporary directory: /private/tmp/pip-ephem-wheel-cache-ug7br3yw
Created temporary directory: /private/tmp/pip-req-tracker-oehxwbil
Initialized build tracking at /private/tmp/pip-req-tracker-oehxwbil
Created build tracker: /private/tmp/pip-req-tracker-oehxwbil
Entered build tracker: /private/tmp/pip-req-tracker-oehxwbil
Created temporary directory: /private/tmp/pip-install-ipqf4c2a
Processing $SRC_DIR
  Created temporary directory: /private/tmp/pip-req-build-cj348h0a
  Added file://$SRC_DIR to build tracker '/private/tmp/pip-req-tracker-oehxwbil'
    Running setup.py (path:/private/tmp/pip-req-build-cj348h0a/setup.py) egg_info for package from file://$SRC_DIR
    Running command python setup.py egg_info
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/tmp/pip-req-build-cj348h0a/setup.py", line 10, in <module>
        long_description=pathlib.Path("Readme.md").read_text(),
      File "/Users/uwe/miniconda3/conda-bld/ipytest_1586412715596/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/python3.8/pathlib.py", line 1227, in read_text
        with self.open(mode='r', encoding=encoding, errors=errors) as f:
      File "/Users/uwe/miniconda3/conda-bld/ipytest_1586412715596/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/python3.8/pathlib.py", line 1213, in open
        return io.open(self, mode, buffering, encoding, errors, newline,
      File "/Users/uwe/miniconda3/conda-bld/ipytest_1586412715596/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/python3.8/pathlib.py", line 1069, in _opener
        return self._accessor.open(self, flags, mode)
    FileNotFoundError: [Errno 2] No such file or directory: 'Readme.md'
Cleaning up...
  Removing source in /private/tmp/pip-req-build-cj348h0a

Cannot find @given for pytest-bdd

When executing pytest_bdd tests, the definitions are not found. I have tried both the ipytest.run() as well as %%run_pytest[clean] but the step definitions are not identified in the file.

pytest_bdd.exceptions.StepDefinitionNotFoundError: Step definition is not found:

The following is a straightforward Gherkin feature file:

Feature: DuckDuckGo Web Browsing

  Background:
    Given the DuckDuckGo home page is displayed

  Scenario: Basic DuckDuckGo Search
    When the user searches for "panda"
    Then results are shown for "panda"

and the steps code:

import pytest_bdd
import pytest
from selenium.webdriver import Firefox
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.firefox.options import Options

options = Options()
options.headless = True

DUCKDUCKGO_HOME = 'https://duckduckgo.com/'

@pytest.fixture
def browser():
  driver = Firefox(options=options)
  driver.implicitly_wait(10)
  yield driver
  driver.quit()

@pytest_bdd.scenario('websearch.feature','Basic DuckDuckGo Search', features_base_dir='', strict_gherkin=False)
def test_websearch():
    pass

@pytest_bdd.given('the DuckDuckGo home page is displayed')
def ddg_home(browser):
    browser.get(DUCKDUCKGO_HOME)
 
@pytest_bdd.when(pytest_bdd.parsers.parse('the user searches for "{phrase}"'))
def search_phrase(browser, phrase):
    search_input = browser.find_element_by_id('search_form_input_homepage')
    search_input.send_keys(phrase + Keys.RETURN)
 
@pytest_bdd.then(pytest_bdd.parsers.parse('results are shown for "{phrase}"'))
def search_results(browser, phrase):
    links_div = browser.find_element_by_id('links')
    assert len(links_div.find_elements_by_xpath('//div')) > 0
    search_input = browser.find_element_by_id('search_form_input')
    assert search_input.get_attribute('value') == "phrase"

On the other hand, running these from the command-line using pytest works perfectly fine. Has anyone experienced this before?

Selecting individual tests in a cell

I'm looking for a way to use pytest's selection syntax together with ipytest. Consider the following cell:

%%run_pytest[clean]

class TestFoo:
    def test_bar(self):
        pass

    def test_baz(self):
        pass

class TestSpam:
    def test_ham(self):
        pass

Normally, I can for example pass $FILE::TestFoo::test_baz to select an individual test. Since I don't know the temporary file created by ipytest I can't do that here.

Would it be possible to simply prefix the temporary file in case ipytest encounters an argument starting with a double colon ::? Since there is no such pytest command, IMO it shouldn't break anything else.

TypeError: 'Config' object is not callable

import ipytest
ipytest.config(rewrite_asserts=True, magics=True)