smarie / python-pytest-cases Goto Github PK
View Code? Open in Web Editor NEWSeparate test code from test cases in pytest.
Home Page: https://smarie.github.io/python-pytest-cases/
License: BSD 3-Clause "New" or "Revised" License
Separate test code from test cases in pytest.
Home Page: https://smarie.github.io/python-pytest-cases/
License: BSD 3-Clause "New" or "Revised" License
Like in pytest parametrize. For example providing a callable instead of a template in name_template
. Also supporting an explicit name list could be great.
See https://stackoverflow.com/questions/46913043/pytest-parametrised-fixture-with-different-cases
As of v0.3.0, using @lru_cache
on case functions raises an exception.
For example
Cases: Orange, Strawberry
Test: test_fruit
Test steps (only for the pytest-steps example):
Hi, another one! :)
Example:
from pytest_cases import pytest_fixture_plus as fixture
import pytest
@pytest.fixture(params=[
pytest.param(1, id="p_a"),
pytest.param(2, id="p_b"),
])
def pyfix(request):
return request.param
@fixture
@pytest.mark.parametrize("arg1, arg2", [
pytest.param(1,2, id="f_a"),
pytest.param(3,4, id="f_b")
])
def myfix(arg1, arg2):
return arg1, arg2
@pytest.mark.parametrize("arg3, arg4", [
pytest.param(10,20, id="t_a"),
pytest.param(30,40, id="t_b")
])
def test_one(pyfix, myfix, arg3, arg4):
print(myfix)
result:
test_one.py::test_one[p_a-1-2-t_a] PASSED [ 12%]
test_one.py::test_one[p_a-1-2-t_b] PASSED [ 25%]
test_one.py::test_one[p_a-3-4-t_a] PASSED [ 37%]
test_one.py::test_one[p_a-3-4-t_b] PASSED [ 50%]
test_one.py::test_one[p_b-1-2-t_a] PASSED [ 62%]
test_one.py::test_one[p_b-1-2-t_b] PASSED [ 75%]
test_one.py::test_one[p_b-3-4-t_a] PASSED [ 87%]
test_one.py::test_one[p_b-3-4-t_b] PASSED [100%]
expected result:
test_one.py::test_one[p_a-f_a-t_a] PASSED [ 12%]
test_one.py::test_one[p_a-f_a-t_b] PASSED [ 25%]
test_one.py::test_one[p_a-f_b-t_a] PASSED [ 37%]
test_one.py::test_one[p_a-f_b-t_b] PASSED [ 50%]
test_one.py::test_one[p_b-f_a-t_a] PASSED [ 62%]
test_one.py::test_one[p_b-f_a-t_b] PASSED [ 75%]
test_one.py::test_one[p_b-f_b-t_a] PASSED [ 87%]
test_one.py::test_one[p_b-f_b-t_b] PASSED [100%]
This test fails because instead of being [1]
the node id is [a-1]
. In other words the first arg
fixture created seems to be in 'auto use' mode.
import pytest
from pytest_cases import param_fixtures
arg = param_fixtures('arg', ['a', 'b'])
@pytest.mark.parametrize("arg", [1])
def test_reference2(arg, request):
assert '[1]' in request.node.nodeid
Note that this does not happen with param_fixture
.
from pytest_cases import param_fixture, fixture_union, pytest_fixture_plus
a = param_fixture('a', ['x', 'y'])
@pytest_fixture_plus(params=[1, 2])
def b(request):
assert request.param in [1, 2] # this fails !
return request.param
c = fixture_union('c', [a, b])
def test_fixture_union(c, a):
print(c, a)
Maybe a special constant THIS_MODULE will do the trick ?
It should be easier to
See this discusssion
This seems like the missing piece. When we have this feature, combining it with param_fixture
/ param_fixtures
will enable us to reproduce any of the scenarios mentioned in
pytest-dev/pytest#349
Note that we could also provide a helper goodie @pytest_parametrize_plus(..., fixtures=[...])
to do the same but directly on a function.
According to many users (see pytest-dev/pytest#349) it would be useful to use fixture references in pytest parameter values.
Now that we support union fixtures this should be feasible.
For example we could provide a way to 'tag' the case functions with any object(s), and to 'filter' the case functions based on tag match in the @cases_data
decorator
I can explain it in the following example:
from pytest_cases import pytest_fixture_plus
import pytest
###### Using Pytest Fixture Plus ######
@pytest_fixture_plus
@pytest.mark.parametrize("arg1,arg2", [
(1, 2),
(3, 4),
])
def myfix(arg1, arg2):
return arg1, arg2
@pytest.mark.parametrize("arg1", [
1, 2,
])
def test_one(myfix, arg1):
print(myfix)
##### Pytest Normal parametrization #####
@pytest.fixture(params=[10,20])
def myfix2(request):
return request.param
@pytest.mark.parametrize("arg1, arg2", [
(1, 2),
(3, 4),
])
def test_two(arg1, arg2, myfix2):
print(arg1, arg2)
Which renders:
test_one.py::test_one[myfix__arg1Xarg20-1] PASSED [ 12%]
test_one.py::test_one[myfix__arg1Xarg20-2] PASSED [ 25%]
test_one.py::test_one[myfix__arg1Xarg21-1] PASSED [ 37%]
test_one.py::test_one[myfix__arg1Xarg21-2] PASSED [ 50%]
test_one.py::test_two[10-1-2] PASSED [ 62%]
test_one.py::test_two[10-3-4] PASSED [ 75%]
test_one.py::test_two[20-1-2] PASSED [ 87%]
test_one.py::test_two[20-3-4] PASSED [100%]
So basically the test id for test_one
, parametrized using pytest_fixture_plus
is not very pretty. It would be better to follow the same rules pytest uses for test ids in parametrization, resulting perhaps in something like this:
test_one.py::test_one[1-2-1] PASSED [ 12%]
test_one.py::test_one[1-2-2] PASSED [ 25%]
test_one.py::test_one[3-4-1] PASSED [ 37%]
test_one.py::test_one[3-4-2] PASSED [ 50%]
test_one.py::test_two[10-1-2] PASSED [ 62%]
test_one.py::test_two[10-3-4] PASSED [ 75%]
test_one.py::test_two[20-1-2] PASSED [ 87%]
test_one.py::test_two[20-3-4] PASSED [100%]
While trying to solve this : https://stackoverflow.com/questions/46909275/parametrizing-tests-depending-of-also-parametrized-values-in-pytest
I found out that my proposal with union fixtures was not working correctly.
This idea will probably eventually be part of pytest, if users find it interesting.
But for older versions of pytest and also in the meantime, we could provide a workaround and support the following syntax:
@cases_fixture(module=xxxx, scope='session')
@pytest.mark.parametrize('param', [1, 2], ids=str)
def my_fixture(case_data, param):
return case_data.get(param)
Note that fixing it would also immediately make additional @cases_data
usable, assuming we change the argument name. Indeed they are nothing but an automatized @pytest.mark.parametrize
@cases_fixture(module=xxxx, scope='session')
@cases_data(module=yyyy, case_data_argname='case_data2')
def my_fixture(case_data, case_data2):
... whatever you wish to do with the two cases ...
With pytest 3.7
../../../virtualenv/python3.5.6/lib/python3.5/site-packages/pytest_cases/plugin.py:455: in getfixtureclosure
ref_fixturenames, ref_arg2fixturedefs = fm.__class__.getfixtureclosure(fm, fixturenames, parentnode)
First of all, thanks for this neat library, it has simplified our tests quite a bit for cookiecutter-django.
I'm not sure if I missed a change in 1.8 or if it's an actual bug, but it seems that 1.8.1 is not playing nicely with pytest markers. We use markers to spread markers over multiple workers on our CI:
black
flake8
flake8
We basically have 3 test function which all use a pretty big parametrized fixture.
Until <1.8.1, the black output was running in ~8 mins. With 1.8.1, it takes ~20 mins to run and judging by the output, it looks like a lot more test cases are executed.
I tried to bump to 1.8.0 and the output is consistent with the existing behaviour and execution around ~9mins.
Am I still using the library as it's intended or is there a regression in 1.8.1?
IndexError: list index out of range
or AttributeError: 'list' object has no attribute 'replace'
can happen even with normal pytest parametrize when argnames
contains a list/tuple.
session-scoped fixtures are not guaranteed to be unique by session, and module-scoped fixtures are not guaranteed to be unique for an unique parameter, in pytest. See pytest-dev/pytest#2846
We could at least try to provide a workaround in the @cases_fixture
decorator, such as ensure_unique=True
.
Note that pytest-dev/pytest#3393 will not solve the problem because if users have two parametrized session-scoped fixtures, one of them will end-up being setup/teardown twice for each of its parameters.
Two things would be cool:
Consider the following example:
A = [1, 2]
B = [10, 20]
@pytest_fixture_plus
@pytest.mark.parametrize("a", A)
@pytest.mark.parametrize("b", B)
def my_fix(a, b):
return a + b
def test_my_fix(my_fix):
pass
The order of execution will randomly be 1-10, 1-20, 2-10, 2-20 or 10-1, 10-2, 20-1, 20-2.
Hi, this is a feature suggestion, which I believe is the missing piece of parametrization
We can define an individual parameter used for parametrization that is not either intrinsically tied to a fixture or test function like this:
@fixture(params=[1,2,3,4])
def my_parameter(request):
return request.param
@fixture
def fixture_uses_param(my_parameter):
pass
def test_uses_param(my_parameter, fixture_uses_param):
pass
The example above is bad because we are using a fixture to mask what is really a parameter. But we can solve this by doing:
def parameter(values):
@pytest.fixture(scope="session", params=params)
def parameter_fixture(request):
return request.param
return parameter_fixture
and then the test becomes:
my_parameter = parameter([1,2,3,4])
@fixture
def fixture_uses_param(my_parameter):
pass
def test_uses_param(my_parameter, fixture_uses_param):
pass
And we can use many parameters to have a cartesian product, etc etc. This still has two problems though:
parameter("my_parameter", [1,2,3,4])
instead of my_parameter = parameter([1,2,3,4])
parameter("my_parameter1, my_parameter2", [(1,2),(3,4)])
.So, the feature proposal would be to have this parametrization ability through the module body with the mentioned syntax:
parameter("my_parameter, my_other_parameter", [(1,1), (2,2), (3,3), (4,4)])
@fixture
def fixture_uses_param(my_parameter):
pass
def test_uses_param(my_parameter, my_other_parameter, fixture_uses_param):
pass
Is this even possible? I know very little about how parametrization works under the hood in pytest, so if you might be not interested in this feature, I would appreciate some pointers.
thanks!
This works:
from pytest_cases import pytest_fixture_plus
import pytest
@pytest_fixture_plus
@pytest.mark.parametrize("arg1,arg2", [
(1, 2),
(3, 4),
])
def myfix(arg1, arg2):
return arg1, arg2
def test_one(myfix):
print(myfix)
Now adding a whitespace to the parametrize arguments, after the comma:
from pytest_cases import pytest_fixture_plus
import pytest
@pytest_fixture_plus
@pytest.mark.parametrize("arg1, arg2", [ # ADDED WHITESPACE AFTER COMMA
(1, 2),
(3, 4),
])
def myfix(arg1, arg2):
return arg1, arg2
def test_one(myfix):
print(myfix)
fails with
test_one.py:8: in <module>
(3, 4),
..\venv368\lib\site-packages\decopatch\main.py:349: in new_decorator
return call_in_appropriate_mode(impl_function, dk, disambiguation_result)
..\venv368\lib\site-packages\decopatch\utils_calls.py:34: in call_in_appropriate_mode
return no_parenthesis_usage(impl_function, dk.first_arg_value)
..\venv368\lib\site-packages\decopatch\utils_calls.py:71: in no_parenthesis_usage
return decorator_function()(decorated)
..\venv368\lib\site-packages\decopatch\utils_modes.py:129: in _apply_decorator
return user_provided_applier(*args, **kwargs)
..\venv368\lib\site-packages\pytest_cases\main.py:365: in pytest_fixture_plus
new_sig = remove_signature_parameters(old_sig, *old_parameter_names)
..\venv368\lib\site-packages\makefun\main.py:739: in remove_signature_parameters
del params[param_name]
E KeyError: ' arg2'
This works fine when parametrizing a test in pytest.
Hi,
I am testing this code:
from pytest_cases import pytest_fixture_plus
import pytest
@pytest_fixture_plus(scope="module")
@pytest.mark.parametrize("arg1", [
"one", "two"
])
@pytest.mark.parametrize("arg2", [
"one", "two"
])
def myfix(arg1, arg2):
return (arg1, arg2)
def test_one(myfix):
print(myfix)
And getting this exception:
========================================================================================================================= test session starts ==========================================================================================================================
platform win32 -- Python 3.6.4, pytest-4.3.1, py-1.8.0, pluggy-0.9.0
rootdir: C:\Users\victo\PycharmProjects\testing_parameterize_in_fixtures, inifile:
collected 0 items / 1 errors
================================================================================================================================ ERRORS ================================================================================================================================
___________________________________________________________________________________________________________________ ERROR collecting test_simple.py ____________________________________________________________________________________________________________________
venv\lib\site-packages\makefun\main.py:761: in add_signature_parameters
for param in reversed(first):
E TypeError: 'generator' object is not reversible
During handling of the above exception, another exception occurred:
test_simple.py:9: in <module>
"one", "two"
venv\lib\site-packages\decopatch\utils_modes.py:129: in _apply_decorator
return user_provided_applier(*args, **kwargs)
venv\lib\site-packages\pytest_cases\main.py:368: in pytest_fixture_plus
for n in reversed(new_parameter_names)))
venv\lib\site-packages\makefun\main.py:768: in add_signature_parameters
if first.name in params:
E AttributeError: 'generator' object has no attribute 'name'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================================================================================================================= 1 error in 0.34 seconds ========================================================================================================================
This is my environment:
atomicwrites (1.3.0)
attrs (19.1.0)
colorama (0.4.1)
decopatch (1.4.1)
makefun (1.6.0)
more-itertools (6.0.0)
pip (9.0.1)
pluggy (0.9.0)
py (1.8.0)
pytest (4.3.1)
pytest-cases (1.3.1)
setuptools (28.8.0)
six (1.12.0)
Maybe I am doing something wrong?
Thanks!
as mentioned in #13 we should at least make travis run all configurations to be sure the lib works everywhere.
First of all let me thank you for the nice pytest_fixture_plus
, I am really excited about it.
The problem I am having can be seen in the following example.
This works:
from pytest_cases import pytest_fixture_plus as fixture
import pytest
@fixture
@pytest.mark.parametrize("arg1, arg2", [
(1,2),
(3,4)
])
def myfix(arg1, arg2):
return arg1, arg2
@pytest.mark.parametrize("arg3, arg4", [
pytest.param(10,20),
pytest.param(30,40)
])
def test_one(myfix, arg3, arg4):
print(myfix)
test_one.py::test_one[1-2-10-20] PASSED [ 25%]
test_one.py::test_one[1-2-30-40] PASSED [ 50%]
test_one.py::test_one[3-4-10-20] PASSED [ 75%]
test_one.py::test_one[3-4-30-40] PASSED [100%]
But if we enclose the fixture parameters using pytest.param
, it does not anymore:
from pytest_cases import pytest_fixture_plus as fixture
import pytest
@fixture
@pytest.mark.parametrize("arg1, arg2", [
pytest.param(1,2),
pytest.param(3,4)
])
def myfix(arg1, arg2):
return arg1, arg2
@pytest.mark.parametrize("arg3, arg4", [
pytest.param(10,20),
pytest.param(30,40)
])
def test_one(myfix, arg3, arg4):
print(myfix)
test_one.py:8: in <module>
pytest.param(3,4)
..\venv368\lib\site-packages\decopatch\main.py:349: in new_decorator
return call_in_appropriate_mode(impl_function, dk, disambiguation_result)
..\venv368\lib\site-packages\decopatch\utils_calls.py:34: in call_in_appropriate_mode
return no_parenthesis_usage(impl_function, dk.first_arg_value)
..\venv368\lib\site-packages\decopatch\utils_calls.py:71: in no_parenthesis_usage
return decorator_function()(decorated)
..\venv368\lib\site-packages\decopatch\utils_modes.py:129: in _apply_decorator
return user_provided_applier(*args, **kwargs)
..\venv368\lib\site-packages\pytest_cases\main.py:299: in pytest_fixture_plus
raise ValueError("Internal error - unsupported pytest parametrization+mark combination. Please "
E ValueError: Internal error - unsupported pytest parametrization+mark combination. Please report this issue
For example I would like to mark a particular case as skipped
@pytest.mark.skip("skipped because it's too easy")
def case_skipped():
return "hello"
Basically we should copy what is done in pytest-harvest. Note that the continuous integration configuration should also be updated so as to run the correct matrix of pytest version x python version
If the case function is written as
def case_generate():
yield foo_for_step1
yield foo_for_step2
then I guess a a case_data.yield()
could be used to retrieve the case data for the various steps.
Try it and add it in the documentation if it works as expected
As of today the exception can be tricky to understand :)
(...)
..\..\..\..\..\_Libs_OpenSource\python-pytest-cases\pytest_cases\main_params.py:279: in get_all_cases
for m in module:
E TypeError: 'object' object is not iterable
During handling of the above exception, another exception occurred:
(...)
C:\Miniconda3\envs\baseenv\lib\site-packages\decopatch\utils_modes.py:132: in _apply_decorator
return user_provided_applier(*new_args, **kwargs)
..\..\..\..\..\_Libs_OpenSource\python-pytest-cases\pytest_cases\main_params.py:210: in cases_data
_cases = get_all_cases(cases, module, test_func, has_tag, filter)
..\..\..\..\..\_Libs_OpenSource\python-pytest-cases\pytest_cases\main_params.py:285: in get_all_cases
_cases = extract_cases_from_module(m, has_tag=has_tag, filter=filter)
..\..\..\..\..\_Libs_OpenSource\python-pytest-cases\pytest_cases\main_params.py:344: in extract_cases_from_module
_get_case_getter_s(f, code, cases_dct)
..\..\..\..\..\_Libs_OpenSource\python-pytest-cases\pytest_cases\main_params.py:398: in _get_case_getter_s
gen_case_name = names(**gen_case_params_dct)
..\..\..\..\..\_Libs_OpenSource\python-pytest-cases\pytest_cases\main_params.py:384: in names
return _formatter.format(**params)
E IndexError: tuple index out of range
With our new replacement for the fixture closure and parametrization engine (to enable fixture unions in addition to cartesian products), we might have broken the basics. Let's make sure this is not the case.
See https://github.com/pytest-dev/pytest/blob/master/testing/acceptance_test.py#L1096
From pytest-dev/pytest#2405 that was fixed by pytest-dev/pytest#3306
But see also
gen_paramfixture__<fixturename>__<paramname>
is not a very convenient name,
and besides it does not appear next to <fixturename>
which is the decorated fixture name.
Suggested naming: <fixturename>__<paramname>
. Simple and straightforward
In current version, the pytest order optimization (reorder_items
here) is completely disabled (the original order is restored after it has been executed), so that the annoying bug concerning module-scoped fixtures order disappears.
That allows the right order to be set for example in this example.
However
For these reasons it would seem much more reasonable to only propose this "skip pytest optimized order" as an option.
In this somewhat complex example:
from pytest_cases import param_fixtures
from pytest_cases import pytest_fixture_plus as fixture
import pytest
parg1, parg2 = param_fixtures("parg1, parg2", [("a", "b"), ("c", "d")])
@fixture
@pytest.mark.parametrize("arg1, arg2", [
pytest.param(1, 2, id="f_a"),
pytest.param(3, 4, id="f_b")
])
def myfix(arg1, arg2, parg1):
return arg1, arg2, parg1
@pytest.mark.parametrize("arg3, arg4", [
pytest.param(10, 20, id="t_a"),
pytest.param(30, 40, id="t_b")
])
def test_one(myfix, arg3, arg4, parg1, parg2):
print(f"{parg1} {parg2} {myfix} {arg3} {arg4}")
The ids are ok but returned values are not when printing them:
from pytest_cases import param_fixture, fixture_union
a = param_fixture("a", [1, 2])
b = param_fixture("b", [3, 4])
c = fixture_union('c', ['a', 'b'])
def test_the_ids(c):
pass
Currently leads to the following ids:
test_fixture_union_ids.py::test_the_ids[a-1]
test_fixture_union_ids.py::test_the_ids[a-2]
test_fixture_union_ids.py::test_the_ids[b-3]
test_fixture_union_ids.py::test_the_ids[b-4]
It will maybe not be very understandable in more complex cases. Besides as of today if one tries to set the ids manually he will get an error ValueError: ids cannot be set on a union fixture
.
I propose to (1) allow ids to be set on union fixtures, and (2) propose an additional "idstyle" argument with three options:
idstyle=None
: a
, b
(current implementation): there is no way to distinguish this from other parametersidstyle='compact'
: Ua
, Ub
idstyle='explicit'
: c_is_a
, c_is_b
. This would become the default.Note that this new parameter should be available in pytest_parametrize_plus
too.
decorator
stays focused on signature-preserving scenarii (micheles/decorator#58 ), and that makes sense.
I therefore created makefun
to separate the concerns and focus solely on dynamic function generation in particular for signature modification scenarii.
We can now safely replace the my_decorator.py
hack that was pending in here, by usage of makefun
.
As of today a test function decorated with @case_data
is a great way to perform the data retrieval inside the test function body: that way, if a single dataset fails to be retrieved, we do not fail the whole test suite.
However the fact that data retrieval is done inside the test function body might be a concern if test duration is monitored: the time spend to retrieve and/or parse each dataset is not "test duration" per se. Besides if users put some caching in place on their case functions, the first test's duration will be different from subsequent tests'.
For all those reasons it might be a better choice to allow users to create "case fixtures". They would work exactly the same way as current decorator (same arguments to specify where the cases are located and how to filter them).
Documentation would need to be improved then.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.