tidymodels / tune Goto Github PK

View Code? Open in Web Editor NEW

263.0 15.0 36.0 133.38 MB

Tools for tidy parameter tuning

Home Page: https://tune.tidymodels.org

License: Other

R 100.00%

tune's Introduction

tune

Overview

The goal of tune is to facilitate hyperparameter tuning for the tidymodels packages. It relies heavily on recipes, parsnip, and dials.

Installation

Install from CRAN:

install.packages("tune", repos = "http://cran.r-project.org") #or your local mirror

or you can install the current development version using:

# install.packages("pak")
pak::pak("tidymodels/tune")

Examples

There are several package vignettes, as well as articles available at tidymodels.org, demonstrating how to use tune.

Good places to begin include:

More advanced resources available are:

Contributing

This project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

For questions and discussions about tidymodels packages, modeling, and machine learning, please post on Posit Community.
If you think you have encountered a bug, please submit an issue.
Either way, learn how to create and share a reprex (a minimal, reproducible example), to clearly communicate about your code.
Check out further details on contributing guidelines for tidymodels packages and how to get help.

tune's People

Contributors

Stargazers

Watchers

tune's Issues

Since parsnip wants to fit the whole path (and ignores the given single penalty value), we need to find an approach for using linear_reg(penalty = 10^-5, mixture = tune()) or similar. Maybe a custom update() method for glmnet models?

collector functions

collect_pred() instead of get_predictions() and create collect_metrics().

track execution time of model fits?

issue template proposal

Making a new issue for `tune`

Please follow the template below.

If the question is related at all to a specific data analysis, please include a minimal reprex (reproducible example). If you've never heard of a reprex before, start by reading "What is a reprex", and follow the advice further down that page.

Tips:

Here is a good example issue: #46
Issues without a reprex will have a lower priority than the others.
We don't want you to use confidential data; you can blind the data or simulate other data to demonstrate the issue. The functions caret::twoClassSim() or caret::SLC14_1() might be good tools to simulate data for you.
Unless the problem is explicitly about parallel processing, please run sequentially.
- Even if it about parallel processing, please make sure that it runs sequentially first.
Make liberal use of set.seed() to help reproducibility
Please check https://stackoverflow.com/ or https://community.rstudio.com/ to see if someone has already asked the same question (see: Yihui's Rule).
You might need to install these:

install.packages(c("reprex", "sessioninfo"), repos = "http://cran.r-project.org")

When are ready to file the issue, please delete the parts above this line:
< -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -->

The problem

I'm having trouble with ... or

Have you considered ...

Reproducible example

Copy your code to the clipboard and run:

reprex::reprex(si = TRUE)

use named dots interface for update

instead of a character string for the first argument.

tune_Bayes should return all resampling results

average predictions

collect_predictions() needs an average argument that will return one prediction per sample (in cases of bootstrap and repeated CV).

This should work for all types of predictions. For probabilities, the columns should be averaged and the set of probability columns should sum to one. ~~For classes, the mode should be used.~~ (see below)

change pkgdown links

The point to https://rdrr.io/ instead of the pkgdown sites

implement extract option

as a general way of returning model components (or the model itself)

good convergence criterion for Bayesian optimization

add a notes column

catalog warnings and errors here (and other stuff). When parallel process, no output comes back and if something goes wrong all you get is a message that "all models failed" (there was details in output but it fails to show).

by can't contain join column id

The problem

I'm having trouble with running a simple random forest tuning.

Reproducible example

library(tidymodels)
library(tune)
set.seed(1)

df <- tibble(
  x1 = runif(1000, 5, 10),
  x2 = runif(1000, 20, 30),
  outcome = as.factor(ifelse(x1 + x2 > 32, 1, 0))
)

rec <- recipe(df, outcome ~ .)
rf_mod <- rand_forest(mode = "classification", trees = tune()) %>% 
  set_engine("randomForest")
cv_splits <- vfold_cv(df, v = 4)
grid <- expand.grid(trees = c(100, 200))

tune_grid(rec, model = rf_mod, rs = cv_splits, grid = grid)

#> by can't contain join column id which is missing from RHS

more specific tunable methods

This would help with engine arguments in parsnip. For example, if we had

boost_tree(mode = "classification") %>% 
  set_engine("C5.0", rules = TRUE)

then the user would need to create their own grid. For common engine parameters, there should be something a tunable.boost_tree() method that would insert a dials reference when someone chooses to optimize rules.

ers

use summarize instead of estimate

functions for choosing best parameter

Functions to be run on the grid search results such as pick_best() etc.

add post-processing

Once workflows is ready.

optimize iterators

In cases where there are a lot of workers and a fast pre-processor, collapse the for loops and parallel process over all combinations. Also, if there is a single resample, we should iterate over the parameters.

tune_grid() should be able to have good heuristics on what the best approach would be given the tuning grid. In the future, we should have more workflows and have quarterback() be able to more dynamically determine the appropriate items to iterate over.

turn off parallel processing for some packages

we should black list some engines from parallel processing (rJava and keras, notably).

From #6

make grid and Bayes functions S3?

Right now they only accept workflows but maybe they would except recipes, formulas, and a model in case no post-processing is needed. We can still use a workflow under the hood

more explicit seed setting for tune_grid

Perhaps take the caret::train() route and have a vector of seeds as a control argument and add a lot of with_seed() to the different modules.

getting_started.Rmd won't build

The problem

getting_started.Rmd fails to build. It appears to be an issue in tune_Bayes, but I'm not yet able to understand what causes the issue.

Reproducible example

(run everything in the getting_started.Rmd before line 254)

ctrl <- Bayes_control(verbose = TRUE)
set.seed(8154)
knn_search <- tune_Bayes(knn_wflow, rs = cv_splits, initial = 5, iter = 20,
                         param_info = knn_param, control = ctrl)

❯  Generating a set of 5 initial parameter results
✓ Initialization complete

Error: All of the models failed.
In addition: Warning message:
All models failed in tune_grid().

> sessioninfo::session_info()
─ Session info ──────────────────────────────────────────────────────────────────────
 setting  value                       
 version  R version 3.6.1 (2019-07-05)
 os       CentOS Linux 7 (Core)       
 system   x86_64, linux-gnu           
 ui       RStudio                     
 language (EN)                        
 collate  en_US.UTF-8                 
 ctype    en_US.UTF-8                 
 tz       America/Chicago             
 date     2019-10-09                  

─ Packages ──────────────────────────────────────────────────────────────────────────
 package       * version    date       lib source                             
 AmesHousing   * 0.0.3      2017-12-17 [1] CRAN (R 3.6.1)                     
 assertthat      0.2.1      2019-03-21 [2] CRAN (R 3.5.2)                     
 backports       1.1.5      2019-10-02 [2] CRAN (R 3.6.1)                     
 base64enc       0.1-3      2015-07-28 [2] CRAN (R 3.5.1)                     
 bayesplot       1.7.0      2019-05-23 [2] CRAN (R 3.5.2)                     
 boot            1.3-22     2019-04-02 [2] CRAN (R 3.6.1)                     
 broom         * 0.5.2      2019-04-07 [2] CRAN (R 3.5.2)                     
 callr           3.3.1      2019-07-18 [2] CRAN (R 3.5.2)                     
 class           7.3-15     2019-01-01 [2] CRAN (R 3.6.1)                     
 cli             1.9.9.9000 2019-10-09 [1] Github (r-lib/cli@ad6410a)         
 clisymbols      1.2.0      2017-05-21 [2] CRAN (R 3.5.1)                     
 codetools       0.2-16     2018-12-24 [2] CRAN (R 3.6.1)                     
 colorspace      1.4-1      2019-03-18 [2] CRAN (R 3.5.2)                     
 colourpicker    1.0        2017-09-27 [2] CRAN (R 3.5.1)                     
 crayon          1.3.4      2017-09-16 [2] CRAN (R 3.5.1)                     
 crosstalk       1.0.0      2016-12-21 [2] CRAN (R 3.5.1)                     
 curl            4.2        2019-09-24 [2] CRAN (R 3.6.1)                     
 desc            1.2.0      2018-05-01 [2] CRAN (R 3.5.1)                     
 devtools        2.1.0      2019-07-06 [2] CRAN (R 3.5.2)                     
 dials         * 0.0.3      2019-10-01 [2] CRAN (R 3.6.1)                     
 DiceDesign      1.8-1      2019-07-31 [2] CRAN (R 3.6.1)                     
 digest          0.6.21     2019-09-20 [2] CRAN (R 3.6.1)                     
 dplyr         * 0.8.3      2019-07-04 [2] CRAN (R 3.5.2)                     
 DT              0.8        2019-08-07 [2] CRAN (R 3.5.2)                     
 dygraphs        1.1.1.6    2018-07-11 [2] CRAN (R 3.5.1)                     
 evaluate        0.14       2019-05-28 [2] CRAN (R 3.5.2)                     
 fansi           0.4.0      2018-10-05 [2] CRAN (R 3.5.1)                     
 foreach         1.4.7      2019-07-27 [2] CRAN (R 3.5.2)                     
 fs              1.3.1      2019-05-06 [2] CRAN (R 3.5.2)                     
 furrr           0.1.0      2018-05-16 [2] CRAN (R 3.5.1)                     
 future          1.14.0     2019-07-02 [2] CRAN (R 3.5.2)                     
 generics        0.0.2      2018-11-29 [2] CRAN (R 3.5.1)                     
 ggplot2       * 3.2.1      2019-08-10 [2] CRAN (R 3.5.2)                     
 ggridges        0.5.1      2018-09-27 [2] CRAN (R 3.5.1)                     
 git2r           0.26.1     2019-06-29 [2] CRAN (R 3.5.2)                     
 globals         0.12.4     2018-10-11 [2] CRAN (R 3.5.1)                     
 glue            1.3.1      2019-03-12 [2] CRAN (R 3.5.2)                     
 gower           0.2.1      2019-05-14 [2] CRAN (R 3.5.2)                     
 GPfit           1.0-8      2019-02-08 [1] CRAN (R 3.6.1)                     
 gridExtra       2.3        2017-09-09 [2] CRAN (R 3.5.1)                     
 gtable          0.3.0      2019-03-25 [2] CRAN (R 3.5.2)                     
 gtools          3.8.1      2018-06-26 [2] CRAN (R 3.5.1)                     
 hms             0.5.1      2019-08-23 [2] CRAN (R 3.5.2)                     
 htmltools       0.3.6      2017-04-28 [2] CRAN (R 3.5.1)                     
 htmlwidgets     1.3        2018-09-30 [2] CRAN (R 3.5.1)                     
 httpuv          1.5.1      2019-04-05 [2] CRAN (R 3.5.2)                     
 igraph          1.2.4.1    2019-04-22 [2] CRAN (R 3.5.2)                     
 infer         * 0.4.0.1    2019-04-22 [2] CRAN (R 3.5.2)                     
 inline          0.3.15     2018-05-18 [2] CRAN (R 3.5.1)                     
 ipred           0.9-9      2019-04-28 [2] CRAN (R 3.5.2)                     
 iterators       1.0.12     2019-07-26 [2] CRAN (R 3.5.2)                     
 janeaustenr     0.1.5      2017-06-10 [2] CRAN (R 3.5.1)                     
 knitr           1.24       2019-08-08 [2] CRAN (R 3.5.2)                     
 labeling        0.3        2014-08-23 [2] CRAN (R 3.5.1)                     
 later           0.8.0      2019-02-11 [2] CRAN (R 3.5.2)                     
 lattice         0.20-38    2018-11-04 [2] CRAN (R 3.6.1)                     
 lava            1.6.6      2019-08-01 [2] CRAN (R 3.5.2)                     
 lazyeval        0.2.2      2019-03-15 [2] CRAN (R 3.5.2)                     
 lhs             1.0.1      2019-02-03 [1] CRAN (R 3.6.1)                     
 lifecycle       0.1.0      2019-08-01 [2] CRAN (R 3.6.1)                     
 listenv         0.7.0      2018-01-21 [2] CRAN (R 3.5.1)                     
 lme4            1.1-21     2019-03-05 [2] CRAN (R 3.5.2)                     
 loo             2.1.0      2019-03-13 [2] CRAN (R 3.5.2)                     
 lubridate       1.7.4      2018-04-11 [2] CRAN (R 3.5.1)                     
 magrittr        1.5        2014-11-22 [2] CRAN (R 3.5.1)                     
 markdown        1.1        2019-08-07 [2] CRAN (R 3.5.2)                     
 MASS            7.3-51.4   2019-03-31 [2] CRAN (R 3.6.1)                     
 Matrix          1.2-17     2019-03-22 [2] CRAN (R 3.6.1)                     
 matrixStats     0.54.0     2018-07-23 [2] CRAN (R 3.5.1)                     
 memoise         1.1.0      2017-04-21 [2] CRAN (R 3.5.1)                     
 mgcv            1.8-28     2019-03-21 [2] CRAN (R 3.6.1)                     
 mime            0.7        2019-06-11 [2] CRAN (R 3.5.2)                     
 miniUI          0.1.1.1    2018-05-18 [2] CRAN (R 3.5.1)                     
 minqa           1.2.4      2014-10-09 [2] CRAN (R 3.5.1)                     
 munsell         0.5.0      2018-06-12 [2] CRAN (R 3.5.1)                     
 nlme            3.1-140    2019-05-12 [2] CRAN (R 3.6.1)                     
 nloptr          1.2.1      2018-10-03 [2] CRAN (R 3.5.1)                     
 nnet            7.3-12     2016-02-02 [2] CRAN (R 3.6.1)                     
 parsnip       * 0.0.3.9001 2019-10-09 [1] Github (tidymodels/parsnip@ae42617)
 pillar          1.4.2      2019-06-29 [2] CRAN (R 3.5.2)                     
 pkgbuild        1.0.4      2019-08-05 [2] CRAN (R 3.5.2)                     
 pkgconfig       2.0.3      2019-09-22 [2] CRAN (R 3.6.1)                     
 pkgload         1.0.2      2018-10-29 [2] CRAN (R 3.5.1)                     
 plyr            1.8.4      2016-06-08 [2] CRAN (R 3.5.1)                     
 prettyunits     1.0.2      2015-07-13 [2] CRAN (R 3.5.1)                     
 pROC            1.15.3     2019-07-21 [2] CRAN (R 3.5.2)                     
 processx        3.4.1      2019-07-18 [2] CRAN (R 3.5.2)                     
 prodlim         2018.04.18 2018-04-18 [2] CRAN (R 3.5.1)                     
 progress        1.2.2      2019-05-16 [2] CRAN (R 3.5.2)                     
 promises        1.0.1      2018-04-13 [2] CRAN (R 3.5.1)                     
 ps              1.3.0      2018-12-21 [2] CRAN (R 3.5.1)                     
 purrr         * 0.3.2      2019-03-15 [2] CRAN (R 3.5.2)                     
 R6              2.4.0      2019-02-14 [2] CRAN (R 3.5.2)                     
 Rcpp            1.0.2      2019-07-25 [2] CRAN (R 3.5.2)                     
 recipes       * 0.1.7      2019-09-15 [1] CRAN (R 3.6.1)                     
 remotes         2.1.0      2019-06-24 [2] CRAN (R 3.5.2)                     
 reshape2        1.4.3      2017-12-11 [2] CRAN (R 3.5.1)                     
 rlang           0.4.0      2019-06-25 [2] CRAN (R 3.6.1)                     
 rmarkdown       1.14       2019-07-12 [2] CRAN (R 3.5.2)                     
 rpart           4.1-15     2019-04-12 [2] CRAN (R 3.6.1)                     
 rprojroot       1.3-2      2018-01-03 [2] CRAN (R 3.5.1)                     
 rsample       * 0.0.5      2019-07-12 [2] CRAN (R 3.5.2)                     
 rsconnect       0.8.15     2019-07-22 [2] CRAN (R 3.5.2)                     
 rstan           2.19.2     2019-07-09 [2] CRAN (R 3.5.2)                     
 rstanarm        2.18.2     2018-11-10 [2] CRAN (R 3.5.1)                     
 rstantools      1.5.1      2018-08-22 [2] CRAN (R 3.5.1)                     
 rstudioapi      0.10       2019-03-19 [2] CRAN (R 3.5.2)                     
 scales        * 1.0.0      2018-08-09 [2] CRAN (R 3.5.1)                     
 sessioninfo     1.1.1      2018-11-05 [2] CRAN (R 3.5.1)                     
 shiny           1.3.2      2019-04-22 [2] CRAN (R 3.5.2)                     
 shinyjs         1.0        2018-01-08 [2] CRAN (R 3.5.1)                     
 shinystan       2.5.0      2018-05-01 [2] CRAN (R 3.5.1)                     
 shinythemes     1.1.2      2018-11-06 [2] CRAN (R 3.5.1)                     
 SnowballC       0.6.0      2019-01-15 [2] CRAN (R 3.5.1)                     
 StanHeaders     2.18.1-10  2019-06-14 [2] CRAN (R 3.5.2)                     
 stringi         1.4.3      2019-03-12 [2] CRAN (R 3.5.2)                     
 stringr         1.4.0      2019-02-10 [2] CRAN (R 3.5.2)                     
 survival        2.44-1.1   2019-04-01 [2] CRAN (R 3.6.1)                     
 testthat        2.2.1      2019-07-25 [2] CRAN (R 3.5.2)                     
 threejs         0.3.1      2017-08-13 [2] CRAN (R 3.5.1)                     
 tibble        * 2.1.3      2019-06-06 [2] CRAN (R 3.5.2)                     
 tidymodels    * 0.0.2      2018-11-27 [2] CRAN (R 3.6.1)                     
 tidyposterior   0.0.2      2018-11-15 [2] CRAN (R 3.5.1)                     
 tidypredict     0.4.2      2019-07-15 [2] CRAN (R 3.5.2)                     
 tidyr         * 1.0.0      2019-09-11 [1] CRAN (R 3.6.1)                     
 tidyselect      0.2.5      2018-10-11 [2] CRAN (R 3.5.1)                     
 tidytext        0.2.2      2019-07-29 [2] CRAN (R 3.5.2)                     
 timeDate        3043.102   2018-02-21 [2] CRAN (R 3.5.1)                     
 tokenizers      0.2.1      2018-03-29 [2] CRAN (R 3.5.1)                     
 tune          * 0.0.0.9001 2019-10-09 [1] local                              
 usethis         1.5.1      2019-07-04 [2] CRAN (R 3.5.2)                     
 utf8            1.1.4      2018-05-24 [2] CRAN (R 3.5.1)                     
 vctrs           0.2.0      2019-07-05 [2] CRAN (R 3.6.1)                     
 whisker         0.3-2      2013-04-28 [2] CRAN (R 3.5.1)                     
 withr           2.1.2      2018-03-15 [2] CRAN (R 3.5.1)                     
 xfun            0.8        2019-06-25 [2] CRAN (R 3.5.2)                     
 xtable          1.8-4      2019-04-21 [2] CRAN (R 3.5.2)                     
 xts             0.11-2     2018-11-05 [2] CRAN (R 3.5.1)                     
 yaml            2.2.0      2018-07-25 [2] CRAN (R 3.5.1)                     
 yardstick     * 0.0.4      2019-08-26 [1] CRAN (R 3.6.1)                     
 zeallot         0.1.0      2018-01-28 [2] CRAN (R 3.5.1)                     
 zoo             1.8-6      2019-05-28 [2] CRAN (R 3.5.2)

resampling with no tuning

There should be a code path to just measure performance via resampling without model tuning.

test min_grid with cases where the name and ID are different

Since min_grid() was written before the current version of parameter sets.

Go back to estimate instead of summari[sz]e

It was abusing the generic too much. @DavisVaughan can say that he told me so.

option for saving all assessment set predictions

maybe cal the argument augment or save_pred?

getting installation error

Hi,

trying to install tune

devtools::install_github("tidymodels/tune")

This then asks:

These packages have more recent versions available.
Which would you like to update?

1: All                                          
2: CRAN packages only                           
3: None                                         
4: parsnip (0.0.3.9001 -> ae42617a9...) [GitHub]
5: cli     (1.1.0      -> ad6410aee...) [GitHub]

Enter one or more numbers, or an empty line to skip updates:
1
parsnip   (0.0.3.9001 -> ae42617a9...) [GitHub]
cli       (1.1.0      -> ad6410aee...) [GitHub]
backports (NA         -> 1.1.5       ) [CRAN]

Getting the error:

Error: Failed to install 'tune' from GitHub:
  (converted from warning) cannot remove prior installation of package ‘backports’

catch warnings from estimate_perf

so we don't get a huge number of

A correlation computation is required, but estimate is constant and has 0 standard deviation, resulting in a divide by 0 error. NA will be returned.

at the end

use new cli version

pre-defined alerts/danger/etc and paragraphs and containers.

update type requirements for new yardstick version

update pred_type() and perf_info() to get the directionality from each metric object

candidate sets and duplicate parameters

The grid that Bayesian optimization uses to evaluate candidate sets with semi-random but could end up choosing a set of existing parameters, especially if they are integers or qualitative.

We should do an anti_join() on them to filter existing solutions out and exit if there are no additional values to check.

404 Error on Bayesian Optimization of Classification Model article

On the website, I get a 404 Page Not Found error when I click on the Bayesian Optimization of Classification Model article:
https://tidymodels.github.io/tune/articles/extras/articles/extras/classification.html

Suggestion: tuning whether to include specific steps or not.

For example, the step_* recipe methods could have a "use_[stepname]" attribute. If this one is set to tune(), the tuning is performed both with and without that step.

load namespaces

Using foreach(.packages) fully attaches the packages. We should load their namespace inside the worker sessions. However, as we've learned form caret, this is fragile across parallel backends (topepo/caret#1017)

Split from #6

min_grid breaks

This is more of a parsnip issue but the problem shows up here.

If a model has a fixed parameter, min_grid() breaks

library(tidymodels)
#> ── Attaching packages ────────────────────────────────────────────────────────────────────── tidymodels 0.0.2 ──
#> ✔ broom     0.5.2          ✔ purrr     0.3.2     
#> ✔ dials     0.0.2.9001     ✔ recipes   0.1.6.9000
#> ✔ dplyr     0.8.3          ✔ rsample   0.0.5     
#> ✔ ggplot2   3.2.0          ✔ tibble    2.1.3     
#> ✔ infer     0.4.0.1        ✔ yardstick 0.0.3     
#> ✔ parsnip   0.0.3.9000
#> ── Conflicts ───────────────────────────────────────────────────────────────────────── tidymodels_conflicts() ──
#> ✖ purrr::discard()  masks scales::discard()
#> ✖ dplyr::filter()   masks stats::filter()
#> ✖ dplyr::lag()      masks stats::lag()
#> ✖ ggplot2::margin() masks dials::margin()
#> ✖ dials::offset()   masks stats::offset()
#> ✖ recipes::step()   masks stats::step()

mod <- linear_reg(penalty = tune(), mixture = 1)

grd <- tibble(penalty = 1:5)

min_grid(mod, grd)
#> Error: Result must have length 4, not 0

^{Created on 2019-08-16 by the reprex package (v0.2.1)}

loosen the function argument requirements for acquisition functions

They should be able to take more parameters as long as they have defaults

parallel processing

For now, this should probably work for the outer resampling grid.

For tune_rec() and tune_mod(), we might be able to collapse the resampling and parameter loops but this should wait until post-processing is worked out.

We should also disable parallel processing if any engine = "keras".

Also also, there should be a control option to opt-out of parallel processing even if it is available.

All models failed in tune_grid()

The problem

I'm having trouble reproducing the examples in the grid search article https://tidymodels.github.io/tune/articles/grid.html (I think, similar to #59)

Running the code in that article top to bottom, I encounter Warning: All models failed in tune_grid(). when running the grid search, and don't get the output presented in the article.

I should note that I've only used the wider set of tidymodels packages a few times, so I may be missing something more fundamental here, but at the moment I'm not clear why the code in the article doesn't produce the output shown.

Advanced apologies if this isn't enough to reproduce the issue on your end, please let me know what other details I can provide.

Reproducible example

library(mlbench)
data(Ionosphere)

library(tidymodels)
#> Registered S3 method overwritten by 'xts':
#>   method     from
#>   as.zoo.xts zoo
#> -- Attaching packages -------------------------------------------------------------------------------------------------------- tidymodels 0.0.3 --
#> v broom     0.5.2          v purrr     0.3.2     
#> v dials     0.0.3          v recipes   0.1.7     
#> v dplyr     0.8.3          v rsample   0.0.5     
#> v ggplot2   3.2.1          v tibble    2.1.3     
#> v infer     0.5.0          v yardstick 0.0.4     
#> v parsnip   0.0.3.9001
#> -- Conflicts ----------------------------------------------------------------------------------------------------------- tidymodels_conflicts() --
#> x purrr::discard()  masks scales::discard()
#> x dplyr::filter()   masks stats::filter()
#> x dplyr::lag()      masks stats::lag()
#> x ggplot2::margin() masks dials::margin()
#> x dials::offset()   masks stats::offset()
#> x recipes::step()   masks stats::step()
library(tune)

Ionosphere <- Ionosphere %>% select(-V2)

svm_mod <- svm_rbf(cost = tune(), rbf_sigma = tune()) %>%
  set_mode("classification") %>%
  set_engine("kernlab")


iono_rec <-
  recipe(Class ~ ., data = Ionosphere)  %>%
  # In case V1 is has a single value sampled
  step_zv(all_predictors()) %>% 
  # convert it to a dummy variable
  step_dummy(V1) %>%
  # Scale it the same as the others
  step_range(matches("V1_"))

set.seed(4943)
iono_rs <- bootstraps(Ionosphere, times = 30)

roc_vals <- metric_set(roc_auc)

ctrl <- grid_control(verbose = FALSE)

set.seed(35)
grid_form <- tune_grid(Class ~ ., model = svm_mod, rs = iono_rs, perf = roc_vals, control = ctrl)
#> Warning: All models failed in tune_grid().
grid_form
#> # Bootstrap sampling 
#> # A tibble: 30 x 3
#>    splits            id          .metrics
#>  * <list>            <chr>       <list>  
#>  1 <split [351/120]> Bootstrap01 <NULL>  
#>  2 <split [351/130]> Bootstrap02 <NULL>  
#>  3 <split [351/137]> Bootstrap03 <NULL>  
#>  4 <split [351/141]> Bootstrap04 <NULL>  
#>  5 <split [351/131]> Bootstrap05 <NULL>  
#>  6 <split [351/131]> Bootstrap06 <NULL>  
#>  7 <split [351/127]> Bootstrap07 <NULL>  
#>  8 <split [351/123]> Bootstrap08 <NULL>  
#>  9 <split [351/131]> Bootstrap09 <NULL>  
#> 10 <split [351/117]> Bootstrap10 <NULL>  
#> # ... with 20 more rows

^{Created on 2019-10-09 by the reprex package (v0.3.0)}

And session info for reference:

- Session info ---------------------------------------------------------------------------------------------------------------------------------
 setting  value                       
 version  R version 3.6.1 (2019-07-05)
 os       Windows 10 x64              
 system   x86_64, mingw32             
 ui       RStudio                     
 language (EN)                        
 collate  English_United States.1252  
 ctype    English_United States.1252  
 tz       America/New_York            
 date     2019-10-09                  

- Packages -------------------------------------------------------------------------------------------------------------------------------------
 package       * version    date       lib source        
 assertthat      0.2.1      2019-03-21 [1] CRAN (R 3.6.1)
 backports       1.1.5      2019-10-02 [1] CRAN (R 3.6.1)
 base64enc       0.1-3      2015-07-28 [1] CRAN (R 3.6.0)
 bayesplot       1.7.0      2019-05-23 [1] CRAN (R 3.6.1)
 boot            1.3-23     2019-07-05 [1] CRAN (R 3.6.1)
 broom         * 0.5.2      2019-04-07 [1] CRAN (R 3.6.1)
 callr           3.3.2      2019-09-22 [1] CRAN (R 3.6.1)
 class           7.3-15     2019-01-01 [1] CRAN (R 3.6.1)
 cli             1.9.9.9000 2019-10-09 [1] local         
 clipr           0.7.0      2019-07-23 [1] CRAN (R 3.6.1)
 codetools       0.2-16     2018-12-24 [1] CRAN (R 3.6.1)
 colorspace      1.4-1      2019-03-18 [1] CRAN (R 3.6.1)
 colourpicker    1.0        2017-09-27 [1] CRAN (R 3.6.1)
 crayon          1.3.4      2017-09-16 [1] CRAN (R 3.6.1)
 crosstalk       1.0.0      2016-12-21 [1] CRAN (R 3.6.1)
 dials         * 0.0.3      2019-10-01 [1] CRAN (R 3.6.1)
 DiceDesign      1.8-1      2019-07-31 [1] CRAN (R 3.6.1)
 digest          0.6.21     2019-09-20 [1] CRAN (R 3.6.1)
 dplyr         * 0.8.3      2019-07-04 [1] CRAN (R 3.6.1)
 DT              0.9        2019-09-17 [1] CRAN (R 3.6.1)
 dygraphs        1.1.1.6    2018-07-11 [1] CRAN (R 3.6.1)
 evaluate        0.14       2019-05-28 [1] CRAN (R 3.6.1)
 fansi           0.4.0      2018-10-05 [1] CRAN (R 3.6.1)
 foreach         1.4.7      2019-07-27 [1] CRAN (R 3.6.1)
 fs              1.3.1      2019-05-06 [1] CRAN (R 3.6.1)
 furrr           0.1.0      2018-05-16 [1] CRAN (R 3.6.1)
 future          1.14.0     2019-07-02 [1] CRAN (R 3.6.1)
 generics        0.0.2      2018-11-29 [1] CRAN (R 3.6.1)
 ggplot2       * 3.2.1      2019-08-10 [1] CRAN (R 3.6.1)
 ggridges        0.5.1      2018-09-27 [1] CRAN (R 3.6.1)
 globals         0.12.4     2018-10-11 [1] CRAN (R 3.6.0)
 glue            1.3.1      2019-03-12 [1] CRAN (R 3.6.1)
 gower           0.2.1      2019-05-14 [1] CRAN (R 3.6.0)
 GPfit           1.0-8      2019-02-08 [1] CRAN (R 3.6.1)
 gridExtra       2.3        2017-09-09 [1] CRAN (R 3.6.1)
 gtable          0.3.0      2019-03-25 [1] CRAN (R 3.6.1)
 gtools          3.8.1      2018-06-26 [1] CRAN (R 3.6.0)
 hms             0.5.1      2019-08-23 [1] CRAN (R 3.6.1)
 htmltools       0.4.0      2019-10-04 [1] CRAN (R 3.6.1)
 htmlwidgets     1.5.1      2019-10-08 [1] CRAN (R 3.6.1)
 httpuv          1.5.2      2019-09-11 [1] CRAN (R 3.6.1)
 igraph          1.2.4.1    2019-04-22 [1] CRAN (R 3.6.1)
 infer         * 0.5.0      2019-09-27 [1] CRAN (R 3.6.1)
 inline          0.3.15     2018-05-18 [1] CRAN (R 3.6.1)
 ipred           0.9-9      2019-04-28 [1] CRAN (R 3.6.1)
 iterators       1.0.12     2019-07-26 [1] CRAN (R 3.6.1)
 janeaustenr     0.1.5      2017-06-10 [1] CRAN (R 3.6.1)
 knitr           1.25       2019-09-18 [1] CRAN (R 3.6.1)
 later           1.0.0      2019-10-04 [1] CRAN (R 3.6.1)
 lattice         0.20-38    2018-11-04 [1] CRAN (R 3.6.1)
 lava            1.6.6      2019-08-01 [1] CRAN (R 3.6.1)
 lazyeval        0.2.2      2019-03-15 [1] CRAN (R 3.6.1)
 lhs             1.0.1      2019-02-03 [1] CRAN (R 3.6.1)
 lifecycle       0.1.0      2019-08-01 [1] CRAN (R 3.6.1)
 listenv         0.7.0      2018-01-21 [1] CRAN (R 3.6.1)
 lme4            1.1-21     2019-03-05 [1] CRAN (R 3.6.1)
 loo             2.1.0      2019-03-13 [1] CRAN (R 3.6.1)
 lubridate       1.7.4      2018-04-11 [1] CRAN (R 3.6.1)
 magrittr        1.5        2014-11-22 [1] CRAN (R 3.6.1)
 markdown        1.1        2019-08-07 [1] CRAN (R 3.6.1)
 MASS            7.3-51.4   2019-03-31 [1] CRAN (R 3.6.1)
 Matrix          1.2-17     2019-03-22 [1] CRAN (R 3.6.1)
 matrixStats     0.55.0     2019-09-07 [1] CRAN (R 3.6.1)
 mime            0.7        2019-06-11 [1] CRAN (R 3.6.0)
 miniUI          0.1.1.1    2018-05-18 [1] CRAN (R 3.6.1)
 minqa           1.2.4      2014-10-09 [1] CRAN (R 3.6.1)
 mlbench       * 2.1-1      2012-07-10 [1] CRAN (R 3.6.1)
 munsell         0.5.0      2018-06-12 [1] CRAN (R 3.6.1)
 nlme            3.1-141    2019-08-01 [1] CRAN (R 3.6.1)
 nloptr          1.2.1      2018-10-03 [1] CRAN (R 3.6.1)
 nnet            7.3-12     2016-02-02 [1] CRAN (R 3.6.1)
 packrat         0.5.0      2018-11-14 [1] CRAN (R 3.6.1)
 parsnip       * 0.0.3.9001 2019-10-09 [1] local         
 pillar          1.4.2      2019-06-29 [1] CRAN (R 3.6.1)
 pkgbuild        1.0.5      2019-08-26 [1] CRAN (R 3.6.1)
 pkgconfig       2.0.3      2019-09-22 [1] CRAN (R 3.6.1)
 plyr            1.8.4      2016-06-08 [1] CRAN (R 3.6.1)
 prettyunits     1.0.2      2015-07-13 [1] CRAN (R 3.6.1)
 pROC            1.15.3     2019-07-21 [1] CRAN (R 3.6.1)
 processx        3.4.1      2019-07-18 [1] CRAN (R 3.6.1)
 prodlim         2018.04.18 2018-04-18 [1] CRAN (R 3.6.1)
 progress        1.2.2      2019-05-16 [1] CRAN (R 3.6.1)
 promises        1.1.0      2019-10-04 [1] CRAN (R 3.6.1)
 ps              1.3.0      2018-12-21 [1] CRAN (R 3.6.1)
 purrr         * 0.3.2      2019-03-15 [1] CRAN (R 3.6.1)
 R6              2.4.0      2019-02-14 [1] CRAN (R 3.6.1)
 Rcpp            1.0.2      2019-07-25 [1] CRAN (R 3.6.1)
 recipes       * 0.1.7      2019-09-15 [1] CRAN (R 3.6.1)
 reprex          0.3.0      2019-05-16 [1] CRAN (R 3.6.1)
 reshape2        1.4.3      2017-12-11 [1] CRAN (R 3.6.1)
 rlang           0.4.0      2019-06-25 [1] CRAN (R 3.6.1)
 rmarkdown       1.16       2019-10-01 [1] CRAN (R 3.6.1)
 rpart           4.1-15     2019-04-12 [1] CRAN (R 3.6.1)
 rsample       * 0.0.5      2019-07-12 [1] CRAN (R 3.6.1)
 rsconnect       0.8.15     2019-07-22 [1] CRAN (R 3.6.1)
 rstan           2.19.2     2019-07-09 [1] CRAN (R 3.6.1)
 rstanarm        2.19.2     2019-10-03 [1] CRAN (R 3.6.1)
 rstantools      2.0.0      2019-09-15 [1] CRAN (R 3.6.1)
 rstudioapi      0.10       2019-03-19 [1] CRAN (R 3.6.1)
 scales        * 1.0.0      2018-08-09 [1] CRAN (R 3.6.1)
 sessioninfo     1.1.1      2018-11-05 [1] CRAN (R 3.6.1)
 shiny           1.3.2      2019-04-22 [1] CRAN (R 3.6.1)
 shinyjs         1.0        2018-01-08 [1] CRAN (R 3.6.1)
 shinystan       2.5.0      2018-05-01 [1] CRAN (R 3.6.1)
 shinythemes     1.1.2      2018-11-06 [1] CRAN (R 3.6.1)
 SnowballC       0.6.0      2019-01-15 [1] CRAN (R 3.6.0)
 StanHeaders     2.19.0     2019-09-07 [1] CRAN (R 3.6.1)
 stringi         1.4.3      2019-03-12 [1] CRAN (R 3.6.0)
 stringr         1.4.0      2019-02-10 [1] CRAN (R 3.6.1)
 survival        2.44-1.1   2019-04-01 [1] CRAN (R 3.6.1)
 threejs         0.3.1      2017-08-13 [1] CRAN (R 3.6.1)
 tibble        * 2.1.3      2019-06-06 [1] CRAN (R 3.6.1)
 tidymodels    * 0.0.3      2019-10-04 [1] CRAN (R 3.6.1)
 tidyposterior   0.0.2      2018-11-15 [1] CRAN (R 3.6.1)
 tidypredict     0.4.3      2019-09-03 [1] CRAN (R 3.6.1)
 tidyr         * 1.0.0      2019-09-11 [1] CRAN (R 3.6.1)
 tidyselect      0.2.5      2018-10-11 [1] CRAN (R 3.6.1)
 tidytext        0.2.2      2019-07-29 [1] CRAN (R 3.6.1)
 timeDate        3043.102   2018-02-21 [1] CRAN (R 3.6.0)
 tokenizers      0.2.1      2018-03-29 [1] CRAN (R 3.6.1)
 tune          * 0.0.0.9001 2019-10-09 [1] local         
 utf8            1.1.4      2018-05-24 [1] CRAN (R 3.6.1)
 vctrs           0.2.0      2019-07-05 [1] CRAN (R 3.6.1)
 whisker         0.4        2019-08-28 [1] CRAN (R 3.6.1)
 withr           2.1.2      2018-03-15 [1] CRAN (R 3.6.1)
 xfun            0.10       2019-10-01 [1] CRAN (R 3.6.1)
 xtable          1.8-4      2019-04-21 [1] CRAN (R 3.6.1)
 xts             0.11-2     2018-11-05 [1] CRAN (R 3.6.1)
 yardstick     * 0.0.4      2019-08-26 [1] CRAN (R 3.6.1)
 zeallot         0.1.0      2018-01-28 [1] CRAN (R 3.6.1)
 zoo             1.8-6      2019-05-28 [1] CRAN (R 3.6.1)

[1] C:/Users/jamesleach/Documents/R/R-3.6.1/library

(I'm on a corporate machine at the moment, hence the local installs for tune, cli, and parsnip as installing directly from GitHub is problematic).

there should be a method of getting best estimator and best score

As we do tunning like Bayesian hyper-parameter tuning or a simple grid search, there should be a method to get the best tuned estimates e.g.

# define model_spec
spec_xgboost <- boost_tree(mode = "classification", 
                           tree_depth = tune(),
                           trees = tune(),
                           learn_rate = tune(),
                           mtry = tune(),
                           min_n = tune()) %>% set_engine("xgboost")

## The workflow should have a recipe or a formula attached to it.
xgb_wf <- workflow() %>%
            add_model(spec_xgboost) %>%
          add_formula(Class ~ .)

# final grid
grid <- xgb_wf %>%
  param_set() %>%
  update(mtry = mtry(c(1,20)))%>%
  grid_max_entropy(size = 20)

grid_results <- tune_grid(xgb_wf, rs = folds, grid = grid,
                          perf = metric_set(roc_auc),
                          control = grid_control(verbose = TRUE))

get_best_score(grid_results)
get_best_params(grid_results)

tb_results <- tune_Bayes(ames_wflow,rs = cv_splits,initial = initial_grid,perf = metric_set(rmse),
                   objective = exp_improve(foo),
                   iter = 20,
                   control = Bayes_control(verbose = TRUE,uncertain = 10,extract = num_leaves))

get_best_score(tb_results)
get_best_params(tb_results)

I called it get_best_params() and get_best_score(). These set of best estimate params should then could be directly passed to the model fit to build the final model

apologies if its already being implemented.

Regards,

example issue

The problem

I'm having trouble running grid search to optimize the nonlinearity in two predictors. I'm using a recipe to specify the tuning parameters. The error below is confusing since I'm not using any ID annotations for the parameters:

Error: There are duplicate id values listed in tune(): 'deg_free', 'degree'.

Reproducible example

library(tidymodels)
#> ── Attaching packages ──────────────────────────────────────── tidymodels 0.0.2 ──
#> ✔ broom     0.5.2          ✔ purrr     0.3.2     
#> ✔ dials     0.0.3.9000     ✔ recipes   0.1.7     
#> ✔ dplyr     0.8.3          ✔ rsample   0.0.5     
#> ✔ ggplot2   3.2.1          ✔ tibble    2.1.3     
#> ✔ infer     0.4.0.1        ✔ yardstick 0.0.4     
#> ✔ parsnip   0.0.3.9001
#> ── Conflicts ─────────────────────────────────────────── tidymodels_conflicts() ──
#> ✖ purrr::discard()  masks scales::discard()
#> ✖ dplyr::filter()   masks stats::filter()
#> ✖ dplyr::lag()      masks stats::lag()
#> ✖ ggplot2::margin() masks dials::margin()
#> ✖ dials::offset()   masks stats::offset()
#> ✖ recipes::step()   masks stats::step()
library(tune)
options (width = 100)

set.seed(7898)
data_folds <- vfold_cv(mtcars, repeats = 2)

spline_rec <-
  recipe(mpg ~ ., data = mtcars) %>%
  step_normalize(all_predictors()) %>% 
  step_bs(disp, degree = tune(), deg_free = tune()) %>%
  step_bs(wt, degree = tune(), deg_free = tune())

lm_model <-
  linear_reg(mode = "regression") %>%
  set_engine("lm")

cars_res <- tune_grid(spline_rec, lm_model, rs = data_folds)
#> Error: There are duplicate `id` values listed in `tune()`: 'deg_free', 'degree'.

^{Created on 2019-10-01 by the reprex package (v0.2.1)}

Session info

devtools::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────────────────────────
#>  setting  value                       
#>  version  R version 3.6.0 (2019-04-26)
#>  os       macOS High Sierra 10.13.6   
#>  system   x86_64, darwin15.6.0        
#>  ui       X11                         
#>  language (EN)                        
#>  collate  en_US.UTF-8                 
#>  ctype    en_US.UTF-8                 
#>  tz       America/New_York            
#>  date     2019-10-01                  
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────────────────────────
#>  package       * version    date       lib source                             
#>  assertthat      0.2.1      2019-03-21 [1] CRAN (R 3.6.0)                     
#>  backports       1.1.4      2019-04-10 [1] CRAN (R 3.6.0)                     
#>  base64enc       0.1-3      2015-07-28 [1] CRAN (R 3.6.0)                     
#>  bayesplot       1.6.0      2018-08-02 [1] CRAN (R 3.6.0)                     
#>  boot            1.3-23     2019-07-05 [1] CRAN (R 3.6.0)                     
#>  broom         * 0.5.2      2019-04-07 [1] CRAN (R 3.6.0)                     
#>  callr           3.3.2      2019-09-22 [1] CRAN (R 3.6.0)                     
#>  class           7.3-15     2019-01-01 [1] CRAN (R 3.6.0)                     
#>  cli             1.9.9.9000 2019-09-25 [1] Github (r-lib/cli@ad6410a)         
#>  codetools       0.2-16     2018-12-24 [1] CRAN (R 3.6.0)                     
#>  colorspace      1.4-1      2019-03-18 [1] CRAN (R 3.6.0)                     
#>  colourpicker    1.0        2017-09-27 [1] CRAN (R 3.6.0)                     
#>  crayon          1.3.4      2017-09-16 [1] CRAN (R 3.6.0)                     
#>  crosstalk       1.0.0      2016-12-21 [1] CRAN (R 3.6.0)                     
#>  desc            1.2.0      2018-05-01 [1] CRAN (R 3.6.0)                     
#>  devtools        2.2.0      2019-09-07 [1] CRAN (R 3.6.0)                     
#>  dials         * 0.0.3.9000 2019-10-01 [1] local                              
#>  DiceDesign      1.8-1      2019-07-31 [1] CRAN (R 3.6.0)                     
#>  digest          0.6.21     2019-09-20 [1] CRAN (R 3.6.0)                     
#>  dplyr         * 0.8.3      2019-07-04 [1] CRAN (R 3.6.0)                     
#>  DT              0.9        2019-09-17 [1] CRAN (R 3.6.0)                     
#>  dygraphs        1.1.1.6    2018-07-11 [1] CRAN (R 3.6.0)                     
#>  ellipsis        0.2.0.9000 2019-10-01 [1] Github (r-lib/ellipsis@dc23a8c)    
#>  evaluate        0.14       2019-05-28 [1] CRAN (R 3.6.0)                     
#>  fansi           0.4.0      2018-10-05 [1] CRAN (R 3.6.0)                     
#>  foreach         1.4.7      2019-07-27 [1] CRAN (R 3.6.0)                     
#>  fs              1.3.1      2019-05-06 [1] CRAN (R 3.6.0)                     
#>  furrr           0.1.0      2018-05-16 [1] CRAN (R 3.6.0)                     
#>  future          1.14.0     2019-07-02 [1] CRAN (R 3.6.0)                     
#>  generics        0.0.2      2018-11-29 [1] CRAN (R 3.6.0)                     
#>  ggplot2       * 3.2.1      2019-08-10 [1] CRAN (R 3.6.0)                     
#>  ggridges        0.5.1      2018-09-27 [1] CRAN (R 3.6.0)                     
#>  globals         0.12.4     2018-10-11 [1] CRAN (R 3.6.0)                     
#>  glue            1.3.1      2019-03-12 [1] CRAN (R 3.6.0)                     
#>  gower           0.2.1      2019-05-14 [1] CRAN (R 3.6.0)                     
#>  GPfit           1.0-8      2019-02-08 [1] CRAN (R 3.6.0)                     
#>  gridExtra       2.3        2017-09-09 [1] CRAN (R 3.6.0)                     
#>  gtable          0.3.0      2019-03-25 [1] CRAN (R 3.6.0)                     
#>  gtools          3.8.1      2018-06-26 [1] CRAN (R 3.6.0)                     
#>  highr           0.8        2019-03-20 [1] CRAN (R 3.6.0)                     
#>  hms             0.5.1      2019-08-23 [1] CRAN (R 3.6.0)                     
#>  htmltools       0.3.6      2017-04-28 [1] CRAN (R 3.6.0)                     
#>  htmlwidgets     1.3        2018-09-30 [1] CRAN (R 3.6.0)                     
#>  httpuv          1.5.1      2019-04-05 [1] CRAN (R 3.6.0)                     
#>  igraph          1.2.4.1    2019-04-22 [1] CRAN (R 3.6.0)                     
#>  infer         * 0.4.0.1    2019-04-22 [1] CRAN (R 3.6.0)                     
#>  inline          0.3.15     2018-05-18 [1] CRAN (R 3.6.0)                     
#>  ipred           0.9-9      2019-04-28 [1] CRAN (R 3.6.0)                     
#>  iterators       1.0.12     2019-07-26 [1] CRAN (R 3.6.0)                     
#>  janeaustenr     0.1.5      2017-06-10 [1] CRAN (R 3.6.0)                     
#>  knitr           1.25       2019-09-18 [1] CRAN (R 3.6.0)                     
#>  later           0.8.0      2019-02-11 [1] CRAN (R 3.6.0)                     
#>  lattice         0.20-38    2018-11-04 [1] CRAN (R 3.6.0)                     
#>  lava            1.6.6      2019-08-01 [1] CRAN (R 3.6.0)                     
#>  lazyeval        0.2.2      2019-03-15 [1] CRAN (R 3.6.0)                     
#>  lhs             1.0.1      2019-02-03 [1] CRAN (R 3.6.0)                     
#>  lifecycle       0.1.0      2019-08-01 [1] CRAN (R 3.6.0)                     
#>  listenv         0.7.0      2018-01-21 [1] CRAN (R 3.6.0)                     
#>  lme4            1.1-21     2019-03-05 [1] CRAN (R 3.6.0)                     
#>  loo             2.1.0      2019-03-13 [1] CRAN (R 3.6.0)                     
#>  lubridate       1.7.4      2018-04-11 [1] CRAN (R 3.6.0)                     
#>  magrittr        1.5        2014-11-22 [1] CRAN (R 3.6.0)                     
#>  markdown        1.1        2019-08-07 [1] CRAN (R 3.6.0)                     
#>  MASS            7.3-51.4   2019-03-31 [1] CRAN (R 3.6.0)                     
#>  Matrix          1.2-17     2019-03-22 [1] CRAN (R 3.6.0)                     
#>  matrixStats     0.54.0     2018-07-23 [1] CRAN (R 3.6.0)                     
#>  memoise         1.1.0      2017-04-21 [1] CRAN (R 3.6.0)                     
#>  mime            0.7        2019-06-11 [1] CRAN (R 3.6.0)                     
#>  miniUI          0.1.1.1    2018-05-18 [1] CRAN (R 3.6.0)                     
#>  minqa           1.2.4      2014-10-09 [1] CRAN (R 3.6.0)                     
#>  munsell         0.5.0      2018-06-12 [1] CRAN (R 3.6.0)                     
#>  nlme            3.1-139    2019-04-09 [1] CRAN (R 3.6.0)                     
#>  nloptr          1.2.1      2018-10-03 [1] CRAN (R 3.6.0)                     
#>  nnet            7.3-12     2016-02-02 [1] CRAN (R 3.6.0)                     
#>  parsnip       * 0.0.3.9001 2019-10-01 [1] Github (tidymodels/parsnip@ae42617)
#>  pillar          1.4.2      2019-06-29 [1] CRAN (R 3.6.0)                     
#>  pkgbuild        1.0.5      2019-08-26 [1] CRAN (R 3.6.0)                     
#>  pkgconfig       2.0.3      2019-09-22 [1] CRAN (R 3.6.0)                     
#>  pkgload         1.0.2      2018-10-29 [1] CRAN (R 3.6.0)                     
#>  plyr            1.8.4      2016-06-08 [1] CRAN (R 3.6.0)                     
#>  prettyunits     1.0.2      2015-07-13 [1] CRAN (R 3.6.0)                     
#>  pROC            1.15.3     2019-07-21 [1] CRAN (R 3.6.0)                     
#>  processx        3.4.1      2019-07-18 [1] CRAN (R 3.6.0)                     
#>  prodlim         2018.04.18 2018-04-18 [1] CRAN (R 3.6.0)                     
#>  progress        1.2.2      2019-05-16 [1] CRAN (R 3.6.0)                     
#>  promises        1.0.1      2018-04-13 [1] CRAN (R 3.6.0)                     
#>  ps              1.3.0      2018-12-21 [1] CRAN (R 3.6.0)                     
#>  purrr         * 0.3.2      2019-03-15 [1] CRAN (R 3.6.0)                     
#>  R6              2.4.0      2019-02-14 [1] CRAN (R 3.6.0)                     
#>  Rcpp            1.0.2      2019-07-25 [1] CRAN (R 3.6.0)                     
#>  recipes       * 0.1.7      2019-09-13 [1] local                              
#>  remotes         2.1.0.9000 2019-10-01 [1] Github (r-lib/remotes@2f1a040)     
#>  reshape2        1.4.3      2017-12-11 [1] CRAN (R 3.6.0)                     
#>  rlang           0.4.0      2019-06-25 [1] CRAN (R 3.6.0)                     
#>  rmarkdown       1.15       2019-08-21 [1] CRAN (R 3.6.0)                     
#>  rpart           4.1-15     2019-04-12 [1] CRAN (R 3.6.0)                     
#>  rprojroot       1.3-2      2018-01-03 [1] CRAN (R 3.6.0)                     
#>  rsample       * 0.0.5      2019-07-12 [1] CRAN (R 3.6.0)                     
#>  rsconnect       0.8.13     2019-01-10 [1] CRAN (R 3.6.0)                     
#>  rstan           2.19.2     2019-07-09 [1] CRAN (R 3.6.0)                     
#>  rstanarm        2.18.2     2018-11-10 [1] CRAN (R 3.6.0)                     
#>  rstantools      1.5.1      2018-08-22 [1] CRAN (R 3.6.0)                     
#>  rstudioapi      0.10       2019-03-19 [1] CRAN (R 3.6.0)                     
#>  scales        * 1.0.0      2018-08-09 [1] CRAN (R 3.6.0)                     
#>  sessioninfo     1.1.1      2018-11-05 [1] CRAN (R 3.6.0)                     
#>  shiny           1.3.2      2019-04-22 [1] CRAN (R 3.6.0)                     
#>  shinyjs         1.0        2018-01-08 [1] CRAN (R 3.6.0)                     
#>  shinystan       2.5.0      2018-05-01 [1] CRAN (R 3.6.0)                     
#>  shinythemes     1.1.2      2018-11-06 [1] CRAN (R 3.6.0)                     
#>  SnowballC       0.6.0      2019-01-15 [1] CRAN (R 3.6.0)                     
#>  StanHeaders     2.18.1     2019-01-28 [1] CRAN (R 3.6.0)                     
#>  stringi         1.4.3      2019-03-12 [1] CRAN (R 3.6.0)                     
#>  stringr         1.4.0      2019-02-10 [1] CRAN (R 3.6.0)                     
#>  survival        2.44-1.1   2019-04-01 [1] CRAN (R 3.6.0)                     
#>  testthat        2.2.1      2019-07-25 [1] CRAN (R 3.6.0)                     
#>  threejs         0.3.1      2017-08-13 [1] CRAN (R 3.6.0)                     
#>  tibble        * 2.1.3      2019-06-06 [1] CRAN (R 3.6.0)                     
#>  tidymodels    * 0.0.2      2019-07-03 [1] local                              
#>  tidyposterior   0.0.2      2018-11-15 [1] CRAN (R 3.6.0)                     
#>  tidypredict     0.4.2.9000 2019-09-03 [1] local                              
#>  tidyr         * 1.0.0      2019-09-11 [1] CRAN (R 3.6.0)                     
#>  tidyselect      0.2.5      2018-10-11 [1] CRAN (R 3.6.0)                     
#>  tidytext        0.2.2      2019-07-29 [1] CRAN (R 3.6.0)                     
#>  timeDate        3043.102   2018-02-21 [1] CRAN (R 3.6.0)                     
#>  tokenizers      0.2.1      2018-03-29 [1] CRAN (R 3.6.0)                     
#>  tune          * 0.0.0.9001 2019-10-01 [1] local                              
#>  usethis         1.5.1.9000 2019-09-17 [1] Github (r-lib/usethis@a2342b8)     
#>  vctrs           0.2.0      2019-07-05 [1] CRAN (R 3.6.0)                     
#>  withr           2.1.2      2018-03-15 [1] CRAN (R 3.6.0)                     
#>  xfun            0.9        2019-08-21 [1] CRAN (R 3.6.0)                     
#>  xtable          1.8-4      2019-04-21 [1] CRAN (R 3.6.0)                     
#>  xts             0.12-1     2019-07-17 [1] Github (joshuaulrich/xts@566ac0d)  
#>  yaml            2.2.0      2018-07-25 [1] CRAN (R 3.6.0)                     
#>  yardstick     * 0.0.4      2019-08-26 [1] CRAN (R 3.6.0)                     
#>  zeallot         0.1.0      2018-01-28 [1] CRAN (R 3.6.0)                     
#>  zoo             1.8-6      2019-05-28 [1] CRAN (R 3.6.0)                     
#> 
#> [1] /Library/Frameworks/R.framework/Versions/3.6/Resources/library

better printing when the acquisition function has a dynamic argument

Printing the value at each iteration would be helpful.

write a developer's guide to tune

Describe S3 methods (e.g. tunable) and other aspects of working across dials, parsnip, recipes, and tune.

glmnet fails with one penalty value

library(tidymodels)
#> ── Attaching packages ─────────────────────────────────────────────────────────────── tidymodels 0.0.2 ──
#> ✔ broom     0.5.2          ✔ purrr     0.3.2     
#> ✔ dials     0.0.2.9001     ✔ recipes   0.1.6.9000
#> ✔ dplyr     0.8.3          ✔ rsample   0.0.5     
#> ✔ ggplot2   3.2.1          ✔ tibble    2.1.3     
#> ✔ infer     0.4.0.1        ✔ yardstick 0.0.3.9000
#> ✔ parsnip   0.0.3.9001
#> ── Conflicts ────────────────────────────────────────────────────────────────── tidymodels_conflicts() ──
#> ✖ purrr::discard()  masks scales::discard()
#> ✖ dplyr::filter()   masks stats::filter()
#> ✖ dplyr::lag()      masks stats::lag()
#> ✖ ggplot2::margin() masks dials::margin()
#> ✖ dials::offset()   masks stats::offset()
#> ✖ recipes::step()   masks stats::step()
library(tune)
library(workflows)
#> 
#> Attaching package: 'workflows'
#> The following object is masked from 'package:purrr':
#> 
#>     has_element
library(glmnet)
#> Loading required package: Matrix
#> 
#> Attaching package: 'Matrix'
#> The following object is masked from 'package:tidyr':
#> 
#>     expand
#> Loading required package: foreach
#> 
#> Attaching package: 'foreach'
#> The following objects are masked from 'package:purrr':
#> 
#>     accumulate, when
#> Loaded glmnet 2.0-16
library(dplyr)


rec_no_tune_1 <-
  recipe(mpg ~ ., data = mtcars) %>%
  step_normalize(all_predictors())

glmn_mod <- linear_reg(mixture = tune(), penalty = .1) %>% set_engine("glmnet")

set.seed(363)
mt_folds <- vfold_cv(mtcars, v = 5)

workflow() %>%
  add_recipe(rec_no_tune_1) %>%
  add_model(glmn_mod) %>%
  tune_grid(mt_folds)
#> 
#> Attaching package: 'crayon'
#> The following object is masked from 'package:ggplot2':
#> 
#>     %+%
#> 
#> Attaching package: 'rlang'
#> The following object is masked from 'package:crayon':
#> 
#>     chr
#> The following objects are masked from 'package:purrr':
#> 
#>     %@%, as_function, flatten, flatten_chr, flatten_dbl,
#>     flatten_int, flatten_lgl, flatten_raw, invoke, list_along,
#>     modify, prepend, splice
#> `by` can't contain join column `id` which is missing from RHS

^{Created on 2019-09-04 by the reprex package (v0.2.1)}

Allow for other options (e.g. range) to be specified in tunable

Something like:

tunable.step_other <- function(x, ...) {
  tibble::tibble(
    name = "threshold",
    call_info = list(
      list(pkg = "dials", fun = "threshold", range = (0:1)/10)
    ),
    source = "recipe",
    component = "step_other",
    component_id = x$id
  )
}

Here is my workflow:

load("Data/okc.RData") # data comes from Max's aml-training repo (https://github.com/tidymodels/aml-training/tree/master/two%20day/Materials/Data)

folds <- vfold_cv(okc_train, strata = "Class")

spec_xgboost <- boost_tree(mode = "classification") %>% set_engine("xgboost")

#check which parameters are tunable
spec_xgboost %>% tunable()

# redefine the model_spec
spec_xgboost <- boost_tree(mode = "classification", 
                           tree_depth = tune(),
                           trees = tune(),
                           learn_rate = tune(),
                           mtry = tune(),
                           min_n = tune()) %>% set_engine("xgboost")

# check if these are really tunable (look at the col=tunable)
spec_xgboost %>% tune_args()

xgb_wf <- workflow() %>%
            add_model(spec_xgboost) 

grid <- xgb_wf %>%
  param_set() %>%
  update(mtry = mtry(c(1,20)))%>%
  grid_max_entropy(size = 20)

grid_results <- tune_grid(xgb_wf, rs = folds, grid = grid,
                          control = grid_control(verbose = TRUE))

here is the error:

Error: `tune_rec & !tune_model ~ rlang::call2("tune_rec", !!!args)`, `tune_rec & tune_model ~ rlang::call2("tune_rec_and_mod", !!!args)`, `has_form & tune_model ~ rlang::call2("tune_mod_with_formula", 
    !!!args)`, `!tune_rec & tune_model ~ rlang::call2("tune_mod_with_recipe", 
    !!!args)` must be length 0 or one, not 6

Here is the traceback:

> rlang::last_error()
<error>
message: `tune_rec & !tune_model ~ rlang::call2("tune_rec", !!!args)`, `tune_rec & tune_model ~ rlang::call2("tune_rec_and_mod", !!!args)`, `has_form & tune_model ~ rlang::call2("tune_mod_with_formula", 
    !!!args)`, `!tune_rec & tune_model ~ rlang::call2("tune_mod_with_recipe", 
    !!!args)` must be length 0 or one, not 6
class:   `rlang_error`
backtrace:
 1. tune::tune_grid(xgb_wf, rs = folds, grid = grid)
 2. tune:::tune_grid.workflow(xgb_wf, rs = folds, grid = grid)
 3. tune:::tune_grid_workflow(...)
 4. tune:::quarterback(object)
 5. dplyr::case_when(...)
 6. dplyr:::validate_case_when_length(query, value, fs)
 7. dplyr:::bad_calls(...)
 8. dplyr:::glubort(fmt_calls(calls), ..., .envir = .envir)
Call `rlang::last_trace()` to see the full backtrace
> rlang::last_trace()
    x
 1. +-tune::tune_grid(xgb_wf, rs = folds, grid = grid)
 2. \-tune:::tune_grid.workflow(xgb_wf, rs = folds, grid = grid)
 3.   \-tune:::tune_grid_workflow(...)
 4.     \-tune:::quarterback(object)
 5.       \-dplyr::case_when(...)
 6.         \-dplyr:::validate_case_when_length(query, value, fs)
 7.           \-dplyr:::bad_calls(...)
 8.             \-dplyr:::glubort(fmt_calls(calls), ..., .envir = .envir)

here is the session info:

> sessionInfo()
R version 3.6.1 (2019-07-05)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 8.1 x64 (build 9600)

Matrix products: default

locale:
[1] LC_COLLATE=English_United Kingdom.1252  LC_CTYPE=English_United Kingdom.1252   
[3] LC_MONETARY=English_United Kingdom.1252 LC_NUMERIC=C                           
[5] LC_TIME=English_United Kingdom.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] pROC_1.15.3        naniar_0.4.2       caret_6.0-84       lattice_0.20-38    tune_0.0.0.9001    yardstick_0.0.4   
 [7] tibble_2.1.3       rsample_0.0.5      tidyr_1.0.0        recipes_0.1.7      purrr_0.3.2        parsnip_0.0.3.9001
[13] infer_0.5.0        ggplot2_3.2.1      dplyr_0.8.3        dials_0.0.3        scales_1.0.0       broom_0.5.2       
[19] tidymodels_0.0.3  

loaded via a namespace (and not attached):
  [1] backports_1.1.5     tidytext_0.2.2      plyr_1.8.4          igraph_1.2.4.1      lazyeval_0.2.2     
  [6] splines_3.6.1       crosstalk_1.0.0     listenv_0.7.0       SnowballC_0.6.0     rstantools_2.0.0   
 [11] inline_0.3.15       digest_0.6.21       foreach_1.4.7       htmltools_0.3.6     rsconnect_0.8.15   
 [16] fansi_0.4.0         magrittr_1.5        globals_0.12.4      gower_0.2.1         matrixStats_0.55.0 
 [21] xts_0.11-2          prettyunits_1.0.2   colorspace_1.4-1    xfun_0.9            callr_3.3.2        
 [26] crayon_1.3.4        lme4_1.1-21         zeallot_0.1.0       survival_2.44-1.1   zoo_1.8-6          
 [31] iterators_1.0.12    glue_1.3.1          gtable_0.3.0        ipred_0.9-9         pkgbuild_1.0.5     
 [36] rstan_2.19.2        miniUI_0.1.1.1      Rcpp_1.0.2          xtable_1.8-4        progress_1.2.2     
 [41] GPfit_1.0-8         stats4_3.6.1        lava_1.6.6          StanHeaders_2.19.0  prodlim_2018.04.18 
 [46] DT_0.9              htmlwidgets_1.3     threejs_0.3.1       ellipsis_0.3.0      pkgconfig_2.0.3    
 [51] loo_2.1.0           nnet_7.3-12         utf8_1.1.4          labeling_0.3        tidyselect_0.2.5   
 [56] rlang_0.4.0         DiceDesign_1.8-1    reshape2_1.4.3      later_0.8.0         munsell_0.5.0      
 [61] tools_3.6.1         xgboost_0.90.0.2    cli_1.9.9.9000      generics_0.0.2      ggridges_0.5.1     
 [66] stringr_1.4.0       ModelMetrics_1.2.2  processx_3.4.1      knitr_1.25          packrat_0.5.0      
 [71] visdat_0.5.3        future_1.14.0       nlme_3.1-141        mime_0.7            rstanarm_2.18.2    
 [76] tokenizers_0.2.1    compiler_3.6.1      bayesplot_1.7.0     shinythemes_1.1.2   rstudioapi_0.10    
 [81] testthat_2.2.1      tidyposterior_0.0.2 lhs_1.0.1           stringi_1.4.3       ps_1.3.0           
 [86] desc_1.2.0          Matrix_1.2-17       nloptr_1.2.1        markdown_1.1        shinyjs_1.0        
 [91] vctrs_0.2.0         pillar_1.4.2        lifecycle_0.1.0     furrr_0.1.0         data.table_1.12.2  
 [96] httpuv_1.5.2        R6_2.4.0            promises_1.0.1      gridExtra_2.3       janeaustenr_0.1.5  
[101] codetools_0.2-16    pkgload_1.0.2       boot_1.3-23         colourpicker_1.0    MASS_7.3-51.4      
[106] gtools_3.8.1        assertthat_0.2.1    rprojroot_1.3-2     withr_2.1.2         shinystan_2.5.0    
[111] parallel_3.6.1      hms_0.5.1           grid_3.6.1          rpart_4.1-15        timeDate_3043.102  
[116] class_7.3-15        minqa_1.2.4         tidypredict_0.4.3   shiny_1.3.2         lubridate_1.7.4    
[121] base64enc_0.1-3     dygraphs_1.1.1.6

use on.exit to save current results

for both tuning functions

use tunable in tune_args.step

tune_args.step() has a long white-list. Instead of this, tunable() should be run on the step to get the list of arguments that should be evaluated.

tidymodels / tune Goto Github PK

tune's Introduction

tune

Overview

Installation

Examples

Contributing

tune's People

Contributors

Stargazers

Watchers

Forkers

tune's Issues

Making a new issue for tune

The problem

Reproducible example

The problem

Reproducible example

The problem

Reproducible example

The problem

Reproducible example

The problem

Reproducible example

Recommend Projects

Recommend Topics

Recommend Org

Making a new issue for `tune`