Comments (6)
we need a pytest.ini to store the pattern that matches our test, and then we can replace the command in CI / the test scripts, both of which should be quite easy, but we need to make sure we actually run all the tests.
Then we can start using some of the fancier features gradually if we like to, but really there's nothing else we need to do. Happy to make a PR in a bit.
from mlos.
For the record, one of the goals here is to be able to get some better stats on how long tests take.
See Also #67
from mlos.
parametrizable tests, better error messages, plugins for coverage and some other plugins.
pytest is a bit magic, it parses the AST to figure out a way to give you a good error, so if you write
b = 5
assert b == 6
It tells you "assert failed 5 != 6", while unittest or just plain python would say "AssertionError(False)".
I don't recall seeing a package not using pytest recently. There used to be nosetests, but that is no longer maintained and most packages moved from nosetests to pytest.
Examples of packages that use pytest are numpy, pandas, sklearn, requests, https://github.com/microsoft/hummingbird, https://github.com/microsoft/DeepSpeed, https://github.com/microsoft/coax, ...
from mlos.
The other reason we were discussing this was in the course of troubleshooting #67 and trying to figure out which tests were taking a long time.
I had written this nasty oneliner to try and identify the seconds elapsed that each test took:
last_ts=''; ./scripts/run-python-tests.sh | logger -s -t python-test 2>&1 | grep 'python-test: test_' | while read line; do echo "$line"; ts=$(echo "$line" | egrep -o '^<13>Sep 10 [0-9:]+ ' | sed 's/^<13>//' | xargs -I{} date -d"{}" +%s); if [ -n "$last_ts" ]; then let time_took=ts-last_ts; echo $time_took; fi; last_ts=$ts; done | egrep -B1 '^[1-9][0-9]+'
<13>Sep 10 21:01:39 python-test: test_random_search_optimizer (mlos.Optimizers.ExperimentDesigner.NumericOptimizers.unit_tests.TestRandomSearchOptimizer.TestRandomSearchOptimizer) ... ok
85
--
<13>Sep 10 21:01:50 python-test: test_random_function_configs (mlos.Optimizers.ExperimentDesigner.UtilityFunctions.unit_tests.TestConfidenceBoundUtilityFunction.TestConfidenceBoundUtilityFunction) ... ok
11
--
<13>Sep 10 21:02:04 python-test: test_default_homogeneous_random_forest_model (mlos.Optimizers.RegressionModels.unit_tests.TestHomogeneousRandomForestRegressionModel.TestHomogeneousRandomForestRegressionModel) ...
14
--
<13>Sep 10 21:02:52 python-test: test_hierarchical_quadratic_cold_start (mlos.Optimizers.unit_tests.TestBayesianOptimizer.TestBayesianOptimizer) ... 9881.271554996536
12
<13>Sep 10 21:04:00 python-test: test_hierarchical_quadratic_cold_start_random_configs (mlos.Optimizers.unit_tests.TestBayesianOptimizer.TestBayesianOptimizer) ...
68
<13>Sep 10 21:04:15 python-test: test_translating_dataframe_from_categorical_hierarchical_to_discrete_flat_hypergrid (mlos.Spaces.HypergridAdapters.unit_tests.TestCategoricalToDiscreteHypergridAdapter.TestCategoricalToDiscreteHypergridAdapter) ... ok
15
However, @amueller pointed out that pytest just gives us that with a CLI option.
from mlos.
I'm still educating myself about the relative merits of these two frameworks. What features of pytest particularly take your fancy? I know you mentioned parameterizable tests in the past...
from mlos.
That sounds really neat. What does it take to migrate?
from mlos.
Related Issues (20)
- mlos_bench: expose `--merge-experiment-ids` option via CLI HOT 1
- Support log scale and quantization of numeric range of the Tunables
- Specify prior distributions for the Tunable parameters
- SMAC optimizer does not support mixed input space HOT 5
- Temporarily disable flaky markdown-link-check test
- Add tests for kv_df_to_dict and mixed integer types
- `objectives.optimization_target` column is too long to be in the primary key for MySQL
- Expand scheduler to support additional policies
- Add a storage API to allow a user to enqueue a new trial for an experiment from a notebook
- mlos_bench: grid search support HOT 1
- Integrate multi objective optimization
- Quantized tunables missing check for quantization alignment in `is_valid` check.
- Config support for pluggable Schedulers
- Tests for Scheduler classes
- Clean up the traces and other temp files after running remote commands on Azure VMs
- Increment iteration counter in `.suggest()` instead of `.register()`
- Document "known" variables in mlos_bench
- Revamp number of iterations vs. number of configs tracking HOT 1
- Get rid of deprecated functions like `datetime.utcnow()`
- mlos_bench: parameter constraints HOT 9
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mlos.