A machine learning competition in Automated Deep Learning (AutoDL), co-organized by ChaLearn, Google and 4Paradigm. Accepted at NeurIPS 2019.
Home Page: https://autodl.chalearn.org/
License: Apache License 2.0
The score displayed for "All datasets" phase is the score of dataset 1.
The issue can come from Codalab or from the Parent Scoring Program.
When trying to call the data browser on an audio dataset, inside evariste/autodl:cpu-latest docker, I get the following error:
Playing audio...
Traceback (most recent call last):
File "data_browser.py", line 357, in <module>
main()
File "data_browser.py", line 353, in main
data_browser.show_examples(num_examples=num_examples, subset=subset)
File "data_browser.py", line 306, in show_examples
self.show_an_example(subset=subset)
File "data_browser.py", line 296, in show_an_example
self.show(tensor_4d, label_confidence_pairs=label_conf_pairs)
File "data_browser.py", line 196, in show_speech
DataBrowser.play_sound(data)
File "data_browser.py", line 238, in play_sound
playsound(tmp_filepath)
File "/usr/local/lib/python3.5/dist-packages/playsound.py", line 92, in _playsoundNix
gi.require_version('Gst', '1.0')
File "/usr/lib/python3/dist-packages/gi/__init__.py", line 102, in require_version
raise ValueError('Namespace %s not available' % namespace)
ValueError: Namespace Gst not available
The WAV file is written but the playsound function fails.
Describe the bug
This issue concerns the competitions:
https://autodl.lri.fr/competitions/3
https://autodl.lri.fr/competitions/32
When one makes the same submission, there is a chance to get very long delay. Even though the algorithm, VM configuration, ingestion + scoring, datasets etc are all the same.
The Output Log of ingestion for the submission 6272:
2019-08-23 12:22:34,493 INFO ingestion.py: ************************************************
2019-08-23 12:22:34,493 INFO ingestion.py: ******** Processing dataset Apollon ********
2019-08-23 12:22:34,493 INFO ingestion.py: ************************************************
2019-08-23 12:22:34,493 INFO ingestion.py: Reading training set and test set...
2019-08-23 12:22:34,735 INFO ingestion.py: ===== Start core part of ingestion program. Version: v20190820 =====
2019-08-23 12:22:34,735 INFO ingestion.py: Creating model...
2019-08-23 12:23:22,627 INFO ingestion.py: Begin training the model...
2019-08-23 12:23:41,877 INFO ingestion.py: Finished training the model.
2019-08-23 12:23:41,877 INFO ingestion.py: Begin testing the model by making predictions on test set...
2019-08-23 12:23:46,053 INFO ingestion.py: Finished making predictions.
2019-08-23 12:23:46,204 INFO ingestion.py: [+] 1 predictions made, time spent so far 71.47 sec
2019-08-23 12:23:46,204 INFO ingestion.py: [+] Time left 1128.53 sec
We see that making the first prediction took 72 seconds.
The Output Log of ingestion for the submission 6386:
2019-08-23 14:56:01,055 INFO ingestion.py: ************************************************
2019-08-23 14:56:01,055 INFO ingestion.py: ******** Processing dataset Apollon ********
2019-08-23 14:56:01,055 INFO ingestion.py: ************************************************
2019-08-23 14:56:01,055 INFO ingestion.py: Reading training set and test set...
2019-08-23 14:56:01,299 INFO ingestion.py: ===== Start core part of ingestion program. Version: v20190820 =====
2019-08-23 14:56:01,300 INFO ingestion.py: Creating model...
2019-08-23 14:56:07,437 INFO ingestion.py: Begin training the model...
2019-08-23 14:56:25,904 INFO ingestion.py: Finished training the model.
2019-08-23 14:56:25,904 INFO ingestion.py: Begin testing the model by making predictions on test set...
2019-08-23 14:56:30,127 INFO ingestion.py: Finished making predictions.
2019-08-23 14:56:30,273 INFO ingestion.py: [+] 1 predictions made, time spent so far 28.97 sec
2019-08-23 14:56:30,273 INFO ingestion.py: [+] Time left 1171.03 sec
We see that this time making the first prediction only took 29 seconds.
The Output Log of ingestion for the submission 6506:
2019-08-23 17:41:37,020 INFO ingestion.py: ************************************************
2019-08-23 17:41:37,021 INFO ingestion.py: ******** Processing dataset Apollon ********
2019-08-23 17:41:37,021 INFO ingestion.py: ************************************************
2019-08-23 17:41:37,021 INFO ingestion.py: Reading training set and test set...
2019-08-23 17:41:37,271 INFO ingestion.py: ===== Start core part of ingestion program. Version: v20190820 =====
2019-08-23 17:41:37,271 INFO ingestion.py: Creating model...
2019-08-23 17:41:43,085 INFO ingestion.py: Begin training the model...
2019-08-23 17:42:00,111 INFO ingestion.py: Finished training the model.
2019-08-23 17:42:00,111 INFO ingestion.py: Begin testing the model by making predictions on test set...
2019-08-23 17:42:04,084 INFO ingestion.py: Finished making predictions.
2019-08-23 17:42:04,240 INFO ingestion.py: [+] 1 predictions made, time spent so far 26.97 sec
2019-08-23 17:42:04,240 INFO ingestion.py: [+] Time left 1173.03 sec
We see that this time making the first prediction only took 27 seconds.
So the normal level should be around 28 seconds. In addition, if we look at the same submission for other datasets, this delay is common.
This variance affects the final ALC score a lot since for the above 3 submissions, the ALC scores are respectively:
0.5583, 0.6963, 0.6646
Screenshots
Comparison of different learning curves:
To Reproduce
Steps to reproduce the behavior:
Expected behavior
The variance should be a lot less. The normal level standard deviation of ALC score should be less than 0.03.
Desktop (please complete the following information):
Smartphone (please complete the following information):
[Not relevant]
Additional context
None
System information
evariste/autodl?: YesDescribe the current behavior
Now everything is doing great except that so few issues are created!
Describe the expected behavior
More issues created in Issues page!
Additional info
Everybody reports bugs and other things on Issues!
The major problem is that model.py doesn't have info on the number of training examples.
In single phase competitions, auto-migration should be set to False.
Here are the correct settings:
PARENT PHASE
Results Scoring Only => YES
Auto migration => NO
If submission beats old score, put submission on leaderboard => NO
Ingestion program only during scoring => NO
Is parallel parent => YES
CHILDREN PHASES
Results Scoring Only => YES
Auto migration => NO
If submission beats old score, put submission on leaderboard => NO
Ingestion program only during scoring => YES
Is parallel parent => NO
I hope you can consider to install this package, it's quite useful
https://docs.nvidia.com/deeplearning/sdk/dali-developer-guide/docs/installation.html
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.