alteryx / evalml Goto Github PK

Callbacks were implemented in #42. We should add an explanation of potential uses cases in the docs e.g integrating EvalML into other systems

Add functionality to check if any of the features are highly correlated with the target.

It can be a standalone function

import evalml

evalml.detect_label_leakage(X, y, threshold=.85)

this returns dictionary of features with correlations greater than or equal to threshold. threshold could default to .9.

{
"feature_1": .9,
"feature_2:  .95,
"feature_3": 1.0
}

By default, the automated pipeline search can make this check and raise warning. Can be turned off like this

evalml.AutoClassifer(detect_label_leakage=False)

• Start new repo
• Design metrics and architecture
• Implement for one test case

Currently we do not standardize input as pandas DataFrames or Series. FraudCost.decision_function breaks when getting passed an np.array instead.

Design for categorical encodings.

Should it go in the pipeline? Should it be part of preprocessing?

Add a parameter that allows describe_pipeline() to score on any additional objectives.

Preliminary Tests and Error Raising.

Add to readme and as a check in PRs

Expand testing architecture to encompass multiple tests

• Design testing suite
• Implement for a small number of datasets

The assertion in test_serialization fails sometimes if the two compared models actually produce the same score.

Let's organize and fix bugs!

We should add in support for regression to evalml.preprocessing.split_data

https://alteryx.quip.com/GgtAAtfgbDKf/Data-Checks-API-Design

There is a save_pipeline function but no save_classifier function

It'd be nice to be able to save classifiers or let pipelines store more info so describe_pipeline would be possible with just the saved pipeline

As we flesh out functionality for EvalML, it will be imperative to monitor performance across test datasets to ensure regressions do not occur as we continue to develop.

Metrics to consider:

• Time to run
• Performance metrics (R2/Acc/etc.)
• Variance of pipelines

Datasets to consider:
https://www.openml.org/home

Right now you have to do AutoClassifier.describe_pipeline(id), but it's more intuitive to be able to do AutoClassifier.get_pipeline(id).describe() or AutoClassifier.best_pipeline.describe()?

The main issue right now is that the results of fitting and cross validating the pipeline are on the AutoClassifier object rather than the pipeline object.

As we flesh our EvalML, it will be critical to keep up with testing and documentation.

Right now all parameters specified at one level for the whole pipeline. We should break up parameters by step of the pipeline.

This could also be used to improve the output of describe_pipeline

Currently we require users to pass in if the classification task is multiclass or binary. However, we could infer from the provided objective if the user intended for the task to be multiclass. Therefore, we should decide if we should automatically infer multiclass=True, raise an error/warning, or something else.

Currently we do not have the option to choose which scores to display for describe_pipeline(). This can be added as an additional parameter.

• Implement one-hot encoding in current fixed pipelines.

This can inform the user how much continuing to run the search process is improving pipeline performance.

A simple implementation would just be the line, but we could also superimpose a scatter plot all models searched

• Internal cross validation
  • User can provided their own
  • Warning in the output if there is a high CV variance
• Internal holdout for learning the threshold
• Make your own validation set when training the AutoClassifier
• TODO: Add check for if a feature is 95% correlated with label

Let's continue move to a bigger release with more features.

test calling all the demos.load_* methods

Automate integration test suite on potentially the following options:

• Release
• Scheduled
• ?

Run EvalML on all openml datasets and compare results

• performance, runtime, memory, usage
• also collect best hyperparameters to help EvalML in the future

I think dataset can be accessed using this api: https://docs.openml.org/

Currently, max_time can only be specified in seconds. For much larger datasets, it would be a lot easier to specify the max time in minutes, hours, or days.

Current we show a progress bar that overwrites itself. This means at the end of the search it looks something like this

Testing LogisticRegression w/ imputation + scaling: 100%|██████████| 5/5 [00:13<00:00,  2.73s/it]

We've gotten feedback from users saying they'd like to see the history of the search as it progresses.

We should update output to show each of the models being searched

Right now there is no way of telling if the Auto(*) tests actually fail as we're not raising errors when training the pipelines. To solve this we can check for NaN scores or raise errors.

With the addition of the additional_objectives parameter in #79, it would be helpful to include an example or two in the documentation.

Suggested location: https://evalml.featurelabs.com/en/latest/automl/pipeline_search.html

Allow user to set max_pipelines=None and only stop searching once max_time is hit.

If both, max_pipelines and max_time are none, error.

Right now we don't test each pipeline specifically.

We should add tests to make sure things like feature importance works for each one.

Currently model_types use a magic string. It would be better to use an enum.

We prioritized some functionality to get the first release of EvalML out the door. Here is some functionality that we should focus on including next

Right now it is

.fit(y_prob, y), but .score(y, y_prob)

We should pick one and use it consistently

Supporting Categorical Encodings will be broken down to the following steps.

1. Create preliminary test case and raise relevant errors.
2. Creating design and testing plan
3. Implement.

Functionality added in #19. It's in API ref, but we can included it more prominently in the docs as well

Allow the user to provided callbacks where they get the the training pipelines and progress during the search process

• add a PR bumping the version
• tag a release on github
• make sure all changes in are in the change log
• run integration tests
• use release tools to release to customers
• anything needed to get the documentation update (potentially)

Currently users view the score of a custom objective as a raw number. Maybe it would make it more clear if units helped define the score. For example, in the Fraud objective it would be 25324 Dollars Lost.

Let's continue to revise and make EvalML better.

Improving how the search process and results are displayed to an end user