nengo / pytest-allclose Goto Github PK
View Code? Open in Web Editor NEWGather statistics on allclose comparisons to measure overall test accuracy
License: Other
Gather statistics on allclose comparisons to measure overall test accuracy
License: Other
While writing the announcement, it seemed to me like a command line flag would be a simpler/easier way to report RMSE values than having users edit their conftest.py
.
The standard deviation is pretty meaningless right now. Standard error is more appropriate for what we're measuring.
It's common to want to do not allclose(...)
to ensure that two arrays are different. In this case, it's typical to want record_rmse=False
and print_fail=0
. It would be nice to have e.g. an invert
option that does all this together.
I keep getting fixture 'allclose' not found
when running the test example. Do I need to add something to pytest.ini
or import allclose
somewhere?
def test_feature(allclose):
assert allclose(test_array, target_array)
E fixture 'allclose' not found
> available fixtures: anyio_backend, anyio_backend_name, anyio_backend_options, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, class_mocker, cov, dash_br, dash_duo, dash_process_server, dash_thread_server, dashjl, dashjl_server, dashr, dashr_server, diskcache_manager, doctest_namespace, hydra_restore_singletons, hydra_sweep_runner, hydra_task_runner, mocker, module_mocker, monkeypatch, no_cover, package_mocker, pytestconfig, random_training_data, record_property, record_testsuite_property, record_xml_attribute, recwarn, requests_mock, row, session_mocker, snapshot, testrun_uid, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, worker_id
> use 'pytest --fixtures [testpath]' for help on them.
This prevents coverage from being properly recorded for imports and function definitions.
The documentation (e.g. https://www.nengo.ai/pytest-allclose/#allclose-tolerances) uses nodenames like "test_file.py:test_function", but in order to work properly they need to have a double colon ("test_file.py::test_function").
It would be great to support complex types (as np.allclose
).
import numpy as np
def test_allclose_complex(allclose):
arr = np.zeros(2, dtype=complex)
allclose(arr, arr)
gives
$ pytest test_allclose_complex.py
=========================================================== test session starts ===========================================================
platform linux -- Python 3.9.6, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
PyQt5 5.14.2 -- Qt runtime 5.14.2 -- Qt compiled 5.14.2
rootdir: /home/pierre/tmp
plugins: anyio-3.3.1, nbval-0.9.6, qt-4.0.2, datadir-1.3.1, allclose-1.0.0, cov-2.12.1
collected 1 item
test_allclose_complex.py F [100%]
================================================================ FAILURES =================================================================
__________________________________________________________ test_allclose_complex __________________________________________________________
allclose = <function allclose.<locals>._allclose at 0x7effcf62b5e0>
def test_allclose_complex(allclose):
arr = np.zeros(2, dtype=complex)
> allclose(arr, arr)
test_allclose_complex.py:6:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
a = array([0.+0.j, 0.+0.j]), b = array([0.+0.j, 0.+0.j]), rtol = 1e-05, atol = 1e-08, xtol = 0, equal_nan = False, print_fail = 5
record_rmse = True
@_add_common_docs
def _allclose(
a,
b,
rtol=1e-5,
atol=1e-8,
xtol=0,
equal_nan=False,
print_fail=5,
record_rmse=True,
):
"""Checks if two arrays are close, mimicking `numpy.allclose`.
"""
if len(overrides) > 0:
override_args = overrides[min(call_count[0], len(overrides) - 1)]
atol = override_args.get("atol", atol)
rtol = override_args.get("rtol", rtol)
xtol = override_args.get("xtol", xtol)
equal_nan = override_args.get("equal_nan", equal_nan)
print_fail = override_args.get("print_fail", print_fail)
record_rmse = override_args.get("record_rmse", record_rmse)
call_count[0] += 1
a = np.atleast_1d(a)
b = np.atleast_1d(b)
rmse = _safe_rms(a - b)
if record_rmse and not np.any(np.isnan(rmse)):
request.node.user_properties.append(("rmse", rmse))
ab_rms = _safe_rms(a) + _safe_rms(b)
> rmse_relative = (2 * rmse / ab_rms) if ab_rms > 0 else np.nan
E TypeError: '>' not supported between instances of 'complex' and 'int'
../.pyenv/versions/3.9.6/lib/python3.9/site-packages/pytest_allclose/plugin.py:89: TypeError
========================================================= short test summary info =========================================================
FAILED test_allclose_complex.py::test_allclose_complex - TypeError: '>' not supported between instances of 'complex' and 'int'
============================================================ 1 failed in 0.11s ============================================================
This doesn't render when looking at e.g. https://github.com/nengo/pytest-allclose ("pytest-allclose provides the ~.allclose Pytest fixture")
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.