utapyngo / pytest-unordered Goto Github PK
View Code? Open in Web Editor NEWTest equality of unordered sequences
License: MIT License
pytest-unordered's People
Contributors
Stargazers
Watchers
pytest-unordered's Issues
list vs tuples
Hi,
I recently played around with my own order ignoring pytest assert helper: https://gist.github.com/soxofaan/8e6512f765f0f0df697311c2561be57e . I then polled the pytest devs in pytest-dev/pytest#7899 if it would be interesting to create a PR about this. This turned out to be a duplicate of pytest-dev/pytest#5548 which led to pytest-unordered :)
pytest-unordered seems to do more than what I did in my prototype, except checking the type of the sequence container.
Through the design of unordered(*args), you can not express that you expect for example a list and it should fail on a tuple (or vice versa).
Documentation
I'm considering to start using pytest-unordered in some projects, but the lack of documentation makes that an unfriendly move towards my coworkers.
A quickfix could be just providing some more examples in the README
It does not work with mock.call() and dataclasses
This example does not work:
from unittest.mock import MagicMock, call
from pytest_unordered import unordered
class EmailEvent:
def __init__(self, x):
self.x = x
def __eq__(self, other):
return self.x == other.x
def test():
m = MagicMock()
m(EmailEvent("test 1"))
m(EmailEvent("test 2"))
expected = [call(EmailEvent("test 2")), call(EmailEvent("test 1"))]
assert m.call_args_list == unordered(*expected)❯ pytest t.py
================================================= test session starts =================================================
platform linux -- Python 3.10.7, pytest-7.1.3, pluggy-1.0.0
rootdir: /home/erdnaxeli
plugins: unordered-0.5.1
collected 1 item
t.py F [100%]
====================================================== FAILURES =======================================================
________________________________________________________ test _________________________________________________________
def test():
m = MagicMock()
m(EmailEvent("test 1"))
m(EmailEvent("test 2"))
expected = [call(EmailEvent("test 2")), call(EmailEvent("test 1"))]
> assert m.call_args_list == unordered(*expected)
E assert [call(<t.EmailEvent object at 0x7fbc71ef09d0>),\n call(<t.EmailEvent object at 0x7fbc71ef29b0>)] == [call(<t.EmailEvent object at 0x7fbc71ef2a40>), call(<t.EmailEvent object at 0x7fbc71ef2b00>)]
t.py:20: AssertionError
=============================================== short test summary info ===============================================
FAILED t.py::test - assert [call(<t.EmailEvent object at 0x7fbc71ef09d0>),\n call(<t.EmailEvent object at 0x7fbc71ef...
================================================== 1 failed in 0.08s ==================================================I suspect it is using is instead of ==? If I change the order of the expected list then it works.
Test failure with pytest 8.x
Hi!
After updating pytest to 8.x we are seeing test failures with pytest-unordered 0.5.2 over in nixpkgs.
============================= test session starts ==============================
platform linux -- Python 3.11.8, pytest-8.0.2, pluggy-1.4.0
rootdir: /build/source
plugins: unordered-0.5.2
collected 69 items
tests/test_unordered.py ................................................ [ 69%]
.................F... [100%]
=================================== FAILURES ===================================
_________________________________ test_replace _________________________________
pytester = <Pytester PosixPath('/build/pytest-of-nixbld/pytest-0/test_replace0')>
def test_replace(pytester: Pytester) -> None:
pytester.makepyfile(
"""
from pytest_unordered import unordered
def test_unordered():
assert [{"a": 1, "b": 2}, 2, 3] == unordered(2, 3, {"b": 2, "a": 3})
"""
)
result = pytester.runpytest()
result.assert_outcomes(failed=1, passed=0)
> result.stdout.fnmatch_lines(
[
"E One item replaced:",
"E Omitting 1 identical items, use -vv to show",
"E Differing items:",
"E {'a': 1} != {'a': 3}",
]
)
E Failed: nomatch: 'E One item replaced:'
E and: '============================= test session starts =============================='
E and: 'platform linux -- Python 3.11.8, pytest-8.0.2, pluggy-1.4.0'
E and: 'rootdir: /build/pytest-of-nixbld/pytest-0/test_replace0'
E and: 'plugins: unordered-0.5.2'
E and: 'collected 1 item'
E and: ''
E and: 'test_replace.py F [100%]'
E and: ''
E and: '=================================== FAILURES ==================================='
E and: '________________________________ test_unordered ________________________________'
E and: ''
E and: ' def test_unordered():'
E and: '> assert [{"a": 1, "b": 2}, 2, 3] == unordered(2, 3, {"b": 2, "a": 3})'
E and: ''
E and: 'test_replace.py:4: '
E and: '_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ '
E and: '/nix/store/7404nwsyl2hbqlxlinf60y1rywbrz6vw-python3.11-pytest-8.0.2/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:494: in _call_reprcompare'
E and: ' custom = util._reprcompare(ops[i], each_obj[i], each_obj[i + 1])'
E and: '/nix/store/7404nwsyl2hbqlxlinf60y1rywbrz6vw-python3.11-pytest-8.0.2/lib/python3.11/site-packages/_pytest/assertion/__init__.py:150: in callbinrepr'
E and: ' hook_result = ihook.pytest_assertrepr_compare('
E and: '/nix/store/c5g3c75vzhbrh0lpnnc1jyzii4ahwkfx-python3.11-pluggy-1.4.0/lib/python3.11/site-packages/pluggy/_hooks.py:501: in __call__'
E and: ' return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)'
E and: '/nix/store/c5g3c75vzhbrh0lpnnc1jyzii4ahwkfx-python3.11-pluggy-1.4.0/lib/python3.11/site-packages/pluggy/_manager.py:119: in _hookexec'
E and: ' return self._inner_hookexec(hook_name, methods, kwargs, firstresult)'
E and: '/nix/store/c5g3c75vzhbrh0lpnnc1jyzii4ahwkfx-python3.11-pluggy-1.4.0/lib/python3.11/site-packages/pluggy/_manager.py:473: in traced_hookexec'
E and: ' return outcome.get_result()'
E and: '/nix/store/c5g3c75vzhbrh0lpnnc1jyzii4ahwkfx-python3.11-pluggy-1.4.0/lib/python3.11/site-packages/pluggy/_manager.py:470: in <lambda>'
E and: ' lambda: oldcall(hook_name, hook_impls, caller_kwargs, firstresult)'
E and: '/build/source/pytest_unordered/__init__.py:96: in pytest_assertrepr_compare'
E and: ' result.extend(_compare_eq_any(extra_left[0], extra_right[0], verbose))'
E and: '/nix/store/7404nwsyl2hbqlxlinf60y1rywbrz6vw-python3.11-pytest-8.0.2/lib/python3.11/site-packages/_pytest/assertion/util.py:268: in _compare_eq_any'
E and: ' explanation = _compare_eq_dict(left, right, highlighter, verbose)'
E and: '_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ '
E and: ''
E and: "left = {'a': 1, 'b': 2}, right = {'a': 3, 'b': 2}, highlighter = 0, verbose = 0"
E and: ''
E and: ' def _compare_eq_dict('
E and: ' left: Mapping[Any, Any],'
E and: ' right: Mapping[Any, Any],'
E and: ' highlighter: _HighlightFunc,'
E and: ' verbose: int = 0,'
E and: ' ) -> List[str]:'
E and: ' explanation: List[str] = []'
E and: ' set_left = set(left)'
E and: ' set_right = set(right)'
E and: ' common = set_left.intersection(set_right)'
E and: ' same = {k: left[k] for k in common if left[k] == right[k]}'
E and: ' if same and verbose < 2:'
E and: ' explanation += ["Omitting %s identical items, use -vv to show" % len(same)]'
E and: ' elif same:'
E and: ' explanation += ["Common items:"]'
E and: ' explanation += highlighter(pprint.pformat(same)).splitlines()'
E and: ' diff = {k for k in common if left[k] != right[k]}'
E and: ' if diff:'
E and: ' explanation += ["Differing items:"]'
E and: ' for k in diff:'
E and: ' explanation += ['
E and: '> highlighter(saferepr({k: left[k]}))'
E and: ' + " != "'
E and: ' + highlighter(saferepr({k: right[k]}))'
E and: ' ]'
E and: "E TypeError: 'int' object is not callable"
E and: ''
E and: '/nix/store/7404nwsyl2hbqlxlinf60y1rywbrz6vw-python3.11-pytest-8.0.2/lib/python3.11/site-packages/_pytest/assertion/util.py:505: TypeError'
E and: '=========================== short test summary info ============================'
E and: "FAILED test_replace.py::test_unordered - TypeError: 'int' object is not callable"
E and: '============================== 1 failed in 0.08s ==============================='
E remains unmatched: 'E One item replaced:'
/build/source/tests/test_unordered.py:195: Failed
----------------------------- Captured stdout call -----------------------------
============================= test session starts ==============================
platform linux -- Python 3.11.8, pytest-8.0.2, pluggy-1.4.0
rootdir: /build/pytest-of-nixbld/pytest-0/test_replace0
plugins: unordered-0.5.2
collected 1 item
test_replace.py F [100%]
=================================== FAILURES ===================================
________________________________ test_unordered ________________________________
def test_unordered():
> assert [{"a": 1, "b": 2}, 2, 3] == unordered(2, 3, {"b": 2, "a": 3})
test_replace.py:4:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/nix/store/7404nwsyl2hbqlxlinf60y1rywbrz6vw-python3.11-pytest-8.0.2/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:494: in _call_reprcompare
custom = util._reprcompare(ops[i], each_obj[i], each_obj[i + 1])
/nix/store/7404nwsyl2hbqlxlinf60y1rywbrz6vw-python3.11-pytest-8.0.2/lib/python3.11/site-packages/_pytest/assertion/__init__.py:150: in callbinrepr
hook_result = ihook.pytest_assertrepr_compare(
/nix/store/c5g3c75vzhbrh0lpnnc1jyzii4ahwkfx-python3.11-pluggy-1.4.0/lib/python3.11/site-packages/pluggy/_hooks.py:501: in __call__
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
/nix/store/c5g3c75vzhbrh0lpnnc1jyzii4ahwkfx-python3.11-pluggy-1.4.0/lib/python3.11/site-packages/pluggy/_manager.py:119: in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
/nix/store/c5g3c75vzhbrh0lpnnc1jyzii4ahwkfx-python3.11-pluggy-1.4.0/lib/python3.11/site-packages/pluggy/_manager.py:473: in traced_hookexec
return outcome.get_result()
/nix/store/c5g3c75vzhbrh0lpnnc1jyzii4ahwkfx-python3.11-pluggy-1.4.0/lib/python3.11/site-packages/pluggy/_manager.py:470: in <lambda>
lambda: oldcall(hook_name, hook_impls, caller_kwargs, firstresult)
/build/source/pytest_unordered/__init__.py:96: in pytest_assertrepr_compare
result.extend(_compare_eq_any(extra_left[0], extra_right[0], verbose))
/nix/store/7404nwsyl2hbqlxlinf60y1rywbrz6vw-python3.11-pytest-8.0.2/lib/python3.11/site-packages/_pytest/assertion/util.py:268: in _compare_eq_any
explanation = _compare_eq_dict(left, right, highlighter, verbose)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
left = {'a': 1, 'b': 2}, right = {'a': 3, 'b': 2}, highlighter = 0, verbose = 0
def _compare_eq_dict(
left: Mapping[Any, Any],
right: Mapping[Any, Any],
highlighter: _HighlightFunc,
verbose: int = 0,
) -> List[str]:
explanation: List[str] = []
set_left = set(left)
set_right = set(right)
common = set_left.intersection(set_right)
same = {k: left[k] for k in common if left[k] == right[k]}
if same and verbose < 2:
explanation += ["Omitting %s identical items, use -vv to show" % len(same)]
elif same:
explanation += ["Common items:"]
explanation += highlighter(pprint.pformat(same)).splitlines()
diff = {k for k in common if left[k] != right[k]}
if diff:
explanation += ["Differing items:"]
for k in diff:
explanation += [
> highlighter(saferepr({k: left[k]}))
+ " != "
+ highlighter(saferepr({k: right[k]}))
]
E TypeError: 'int' object is not callable
/nix/store/7404nwsyl2hbqlxlinf60y1rywbrz6vw-python3.11-pytest-8.0.2/lib/python3.11/site-packages/_pytest/assertion/util.py:505: TypeError
=========================== short test summary info ============================
FAILED test_replace.py::test_unordered - TypeError: 'int' object is not callable
============================== 1 failed in 0.08s ===============================
=========================== short test summary info ============================
FAILED tests/test_unordered.py::test_replace - Failed: nomatch: 'E One item replaced:'
========================= 1 failed, 68 passed in 0.36s =========================0.4.0 breaks 1-item sequence calls with type checking
Type checking was added in #3 (fixing #2) (v0.4.0) and enabled based on the number of arguments passed to unordered()
This breaks some code that was previously working:
def pb(message):
"""
Helper for unordered comparisons of protobuf messages
Usage:
assert pb(message_obj) == pb(other_message_obj)
"""
if isinstance(message, Message):
message = MessageToDict(message)
if isinstance(message, list):
return unordered(*[pb(x) for x in message])
elif isinstance(message, dict):
return {k: pb(v) for k, v in message.items()}
else:
# primitive
return messageUsing the above, something like this causes an error (RefList & Ref are both Messages) because the list is only one-item long, so check_type ends up as True:
assert pb(resp) == pb(
RefList(
refs=[
Ref(
name="refs/heads/master",
shorthand="master",
target="0c64d8211c072a08d5fc6e6fe898cbb59fc83d16",
)
]
)
)
>>> TypeError: cannot make unordered comparisons to mapping: {'name': 'refs/heads/master', 'target': '0c64d8211c072a08d5fc6e6fe898cbb59fc83d16', 'shorthand': 'master'}I can (have) worked around it by checking for a 1-item list, but perhaps check_type could be explicitly settable via a parameter to unordered(), rather than automatically setting it based on the sequence length?
Mapping support
unittest.TestCase.assertCountEqual supported mappings.
This is especially useful for comparing two json objects. It can be implemented by first checking the keys of both are 'unsorted equal', and then deeply checking each of the keys values.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.