Failures for new `pytest-fail-slow` check in Windows CI jobs about scipy HOT 7 OPEN

I think it's just new. That test didn't register in other recent CI runs (e.g. gh-20805), but gh-20757 merged 9 hours ago and CI wasn't run in that PR after gh-20672 merged. Now in gh-20807:

================================== FAILURES ===================================
______________ TestDifferentialEvolutionSolver.test_strategy_fn _______________
[gw0] win32 -- Python 3.12.3 C:\hostedtoolcache\windows\Python\3.12.3\x64\python.exe
Test passed but took too long to run: Duration 1.9525856999998723s > 1.0s

from scipy.

I'll look into that particular test tomorrow.

from scipy.

In #20757 TestDifferentialEvolutionSolver.test_strategy_fn was amended slightly to check that a user provided strategy worked with updating='deferred'. I think the test should be run in the fast and slow test suites.

Considering the other changes in the PR the added test runs faster now (508 ms) than it would've done before the PR (755 ms). So it's not a performance issue in the code, it's just that an extra check has been added. We could cut the number of iterations down to make it run faster, but for me this is in the noise. That it takes 2s in CI is negligible. This extra test didn't fail the runtime check in the PR so any fails will be sporadic.

It's now extra work to mark this test as pytest.mark.fail_slow. I think the trial policy to have tests run under a second is going create noise from sporadic fails that'll take extra work to quieten.

from scipy.

This extra test didn't fail the runtime check in the PR so any fails will be sporadic.

No, it would have failed, but CI wasn't run on gh-20757 after the relevant fail-slow PR (gh-20672) merged.

I think the trial policy to have tests run under a second is going create noise from sporadic fails that'll take extra work to quieten.

Noted again.

But we still haven't seen real noise yet. Already some good has come out of it: the fail slow marker revealed that COBYQA was uneccessarily slow. Here, it only came up because the two PRs were happening in parallel. To lessen the burden of applying the mark to the test, I'll go ahead and submit a PR and ping you.

I'd be fine removing pytest-fail-slow it if we really do have trouble with it, though.

from scipy.

Ran across one that was missing from the original PR and fixed it in gh-20824. A few others that I might need to adjust are:

• scipy/stats/tests/test_fit.py::TestFit::test_truncweibull_min
• scipy/sparse/tests/test_base.py::Test64Bit::test_no_64
• scipy/stats/tests/test_fast_gen_inversion.py::test_error_extreme_params
• scipy/interpolate/tests/test_bsplines.py::TestSmoothingSpline::test_weighted_smoothing_spline
• scipy/sparse/tests/test_base.py::TestCSC::test_slicing_3
• scipy/stats/tests/test_resampling.py::TestPermutationTest::test_finite_precision_statistic
• scipy/optimize/tests/test_optimize.py::TestOptimizeResultAttributes::test_attributes_present
• scipy/optimize/tests/test_constraint_conversion.py::TestNewToOld::test_multiple_constraint_objects
• scipy/sparse/tests/test_base.py::Test64Bit::test_resiliency_random

But I've been monitoring the variation between runs and most tests are reasonably consistent. For instance, tests on the list that took significantly more time on a recent CI run compared to the last run of the gh-20672 are:

Tests taking significantly more time:
18.02s -> 28.94s linalg/tests/test_extending.py::test_cython
1.28s -> 2.02s sparse/linalg/_isolve/tests/test_iterative.py::test_precond_inverse[poisson2d]
0.99s -> 1.79s optimize/tests/test__dual_annealing.py::TestDualAnnealing::test_from_docstring
0.91s -> 1.18s optimize/tests/test__dual_annealing.py::TestDualAnnealing::test_bounds_class
0.62s -> 0.83s spatial/tests/test_qhull.py::TestVoronoi::test_incremental[random-3d-chunk-1]
0.43s -> 0.54s optimize/tests/test_constraint_conversion.py::TestNewToOld::test_multiple_constraint_objects
0.4s -> 0.51s linalg/tests/test_matfuncs.py::TestExpM::test_gh18086
0.35s -> 0.49s sparse/tests/test_base.py::Test64Bit::test_resiliency_all_64[TestCSC-test_slicing_3]
0.33s -> 0.45s sparse/tests/test_base.py::Test64Bit::test_resiliency_limit_10[TestDOK-test_add_sub]
0.29s -> 0.43s optimize/tests/test__shgo.py::TestShgoArguments::test_21_1_jac_true
0.28s -> 0.39s special/tests/test_support_alternative_backends.py::test_support_alternative_backends[f_name_n_args3-numpy]
0.26s -> 0.38s sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_ARPACK::test_small_sigma_sparse[complex-shape1]
0.29s -> 0.37s special/tests/test_support_alternative_backends.py::test_support_alternative_backends[f_name_n_args6-numpy]
0.28s -> 0.37s sparse/linalg/_eigen/arpack/tests/test_arpack.py::test_hermitian_modes
0.28s -> 0.36s special/tests/test_support_alternative_backends.py::test_support_alternative_backends[f_name_n_args1-numpy]
0.26s -> 0.35s stats/tests/test_continuous_basic.py::test_cont_basic[500-burr-arg7]
0.25s -> 0.35s sparse/tests/test_base.py::Test64Bit::test_no_64[TestDOK-test_setdiag_comprehensive]
0.26s -> 0.34s interpolate/tests/test_rgi.py::TestRegularGridInterpolator::test_list_input[quintic]
0.25s -> 0.33s sparse/tests/test_base.py::Test64Bit::test_resiliency_random[TestCSR-test_add_sub]
0.25s -> 0.33s interpolate/tests/test_interpolate.py::TestNdPPoly::test_simple_4d

and all of the ones that take more than 0.55s have pytest.mark.fail_slow exceptions with plenty of buffer.

from scipy.