Include an alternative to notebook 04 that shows how to use Bayesian hyperparameter sampling.

Replace unnecessary AzureML pipeline data dependencies by using run_after().

Consider changing the experiment to use low-priority nodes.

Allow a JSON-formatted configuration file input of hyperparameters to the training script.

1. The file would contain a dict.
2. Non-hyperparameter entries in the file would be ignored.
3. Values passed on the command line would NOT override the configuration file values.

Modify the Happy Path to use AML with Hyperdrive in place of Batch AI.

By default, when starting a new notebook, jupyter prompts the user to a similar looking, but incorrect python kernel.

Should either indicated the correct kernel to select, or figure out how to get jupyter to suggest the correct env kernel.

Add Papermill parameters for estimators to notebooks 01, 03, and 04

Mat: Instead of using print statements in the scripts use Python logging. Only reason is you can add debug logging statements that people can turn on or off. Print is fine in notebooks but when running things on clusters sometimes it is nice to have a verbose mode.

https://github.com/Microsoft/MLHyperparameterTuning/blob/master/get_auth.py

Due to finding that with CICD, logging in with a service principal is sufficient to get the interactive login and proceed with workspace creation, this code should be simplified to remove the service principal information.

Currently, the testing script uses the maximum number of iterations of the trained model to score the data. Add an "early_stopping_rounds" argument to the training script so it records the best iteration on validation data found, and an argument to the testing script that controls whether that is the iteration used in scoring.

Mat: Some of the scripts that are created in the notebooks could be broken up a bit. Where you are writing a comment stating what those lines below do that means it could be a function. Makes for easier composability. Completely optional though****

Prevalence of asserts within methods instead of raising exceptions

ipywidgets unused import statement

Add failOnStderr:true to all bash steps per Dan Grecoe:

There seems to be a change to the way the bash steps fail. That is, we've noticed in one of the paths that a notebook that throws an exception does not automatically stop the build as it used to.

Mathew Salvaris pointed out that there is a flag failOnStderr that is defaulted to false. We believe this used to be true. To ensure that the paths truly are building correctly I'm asking that each of the authors go through their path and make the following changes.

The script created by the 01_Training_Script.ipynb notebook defines a variable name that is never used, which could be confusing:

# What to name the metric logged
metric_name = "logloss"

Those lines should be deleted.

Once an AML SDK LightGBM estimator is available, use it to create the best model on the cluster.

Due to a finding with getting the interactive login session (which happens prior to notebooks being run) the need for service principal information is no longer needed.

Remove parameters for such in the following two notebooks:

https://github.com/Microsoft/MLHyperparameterTuning/blob/master/02_Run_Locally.ipynb
https://github.com/Microsoft/MLHyperparameterTuning/blob/master/04_HyperDrive_Run_Recovery.ipynb

It is good to pin python and packages' version (especially when using AML SDK )

To workaround a bug with dotenv.set_env, the python-dotenv repo is referenced in environment.yml. Before release, replace this with the package python-dotenv pinned to the right version.

Add an Azure ML pipeline notebook that does hyperparameter tuning from notebooks 04 through 06.

Put all dependencies for Estimators in a single Conda dependencies file to be used in all cases.

Notebook:
https://github.com/Microsoft/MLHyperparameterTuning/blob/master/04_HyperDrive_Run_Recovery.ipynb

Issue:
Cell 5 Code:
run = get_run(exp, 'HyperDrive_Run_ID', rehydrate=True)
run

Cell 5 Comment:
Use the ID of the run to get a handle to it. That ID was printed with the run when it was submitted in the previous notebook. You can also find that ID in Azure Portal on your experiment's page. You may need to add a RunId column to the table of experiment runs.

The build system has no idea how to collect information from an output screen from the notebook and replace that inline in code.

I assume you haven't correctly set the page anchors.

Mat: There is no easy way to dev-test a set of notebooks locally. I use Makefiles but anything will do. Just something that makes it easy for someone to run one command to create environment run notebooks and tear environment down.

If the number of estimators is less than the reporting period, no metrics are logged.

Split the dataset into a train-tune-test form, likely by splitting the current 20% test sample into two 10% samples.

https://conda.io/docs/commands/env/conda-env-remove.html link in the Readme.md file, results in a page not found error.

I also prefer to not just use (here)[https://conda.io/docs/commands/env/conda-env-remove.html] as a link reference, as it is not informative on where 'here' is to a normal user.

I would prefer (remove conda environments) or something more descriptive.

Error during job execution in notebook 5 (see attached image below): "Paging async iterator protocol is not available for JobPaged"