Comments (9)
Thanks so much for your discussion! 🙌 I am cleaning up older issues.
This looks like it has been wrapped up and/or superseded by new features. Feel free to add on or open a new issue if not!
from rsample.
The idea is to use a constructor for new rset
objects to ensure uniformity so it's basically doing what it was designed to (see section 14.3.1 here for the logic).
If you are implementing a different resampling method, what attributes do you need it add? It contains the attributes that are required for the methods that I've implemented but I don't consider those to be everything that might ever be needed.
from rsample.
I have a function that uses internal logic to first select one or more columns from the input (by type, in this case), and then computes on those to generate train/test indices for each split. Given what I'd seen the included rsample
function do, it seemed like I should add the split_cols
to the attributes, to allow for introspection of how the splits were made. It seems to fit a kind of implicit ethos of the included functions I'm starting to pick up on.
I don't think I need to add this attribute, as I don't currently plan on having downstream code consume that bit of metadata, but in the abstract I can imagine wanting to do so in the future.
Would you mind explaining the logic behind the explicit whitelist for attributes? Is there harm in retaining "unexpected" attributes?
from rsample.
Given what I'd seen the included rsample function do, it seemed like I should add the
split_cols
to the attributes, to allow for introspection of how the splits were made.
So you mean the strata? That's a good idea.
You could probably overload the class:
> tmp <- vfold_cv(mtcars, strata = "cyl")
> class(tmp)
[1] "vfold_cv" "rset" "tbl_df" "tbl" "data.frame"
> class(tmp) <- c("new_class", class(tmp))
and make a specific rset
method for your new class (e.g. like the vfold_cv
class does above).
Would you mind explaining the logic behind the explicit whitelist for attributes? Is there harm in retaining "unexpected" attributes?
It wasn't really done to be restrictive in the way that you have encountered. It's just a good method for 1) forcing you to think out the object structure and plan for what you will eventually need and 2) ensure that any objects of that class are consistent with the structure.
from rsample.
I hadn't been thinking of them as strata, but I suppose they are, so I'll stick my split cols in the strata
attribute, thanks!
Not to worry, I did make my own function and do overload the class via the subclass
arg from rsample:::new_rset
(p.s. seems odd to specify "rset" subclass rather than that being automatic). It's a long function with a lot of arg validation, but I'll include it behind the fold below if you're feeling curious; it's for selective backtesting of financial panel data with arbitrary-but-enforced forecast horizons.
my function
(terrible name, I know)#' Resample a dataset into time-aware, gap-respecting train/test pairs.
#'
#' A custom implementation similar to [rsample::rolling_origin()], but more
#' respectful of time, and including as many observation in both sets as
#' possible without violating the boundary.
#'
#' For each split, rows _ending strictly before_ the respective element of
#' `dates` will be in the `analysis` set, and rows _beginning on or after_ the
#' date will be in the `assessment` set. Rows in-between will be dropped ("the
#' gap"). Rows for which this relation can't be determined (`NA`'s) will be
#' dropped, and rows with any empty values in date columns will not be in
#' analysis sets, but may be in assessment sets. Comparisons will be made using
#' the per-row minima and maxima across
#' all `Date`-type columns after applying any selectors.
#'
#' @param tbl (`tbl_df`) a dataset to resample, with at least one `Date`-valued
#' column
#' @param split_dates (`Date`) a vector of prediction dates to use for resampling.
#' Will be coerced to `Date` with [lubridate::as_date()] if necessary.
#' @param ... selectors to apply with [dplyr::select()] to filter before
#' identifying date-values columns, e.g. to ignore certain columns. All
#' columns will still appear in results.
#'
#' @return (`vc_resample`, `rset`, `tbl_df`) with `length(dates)` rows and the
#' following columns:
#' - `splits` (`rsplit`): individual splits
#' - `id` (`character`): labels
#' The `dates` are stored in an attribute of the same name.
#'
#' @export
#'
#' @examples
#' library(dplyr)
#'
#' tbl <- tibble(
#' date = rep(seq(as.Date("2018-01-01"), by = 1, length.out = 6), each = 2),
#' ticker = rep(c("AAPL", "IWV"), length.out = length(date)),
#' lead_date = rep(lead(unique(date), 2), each = n_distinct(ticker)),
#' value = 1:12
#' )
#'
#' tbl
#'
#' split_dates <- c("2018-01-04", "2018-01-05", "2018-01-06")
#'
#' vc_resample(tbl, split_dates) # use date and lead_date
#' vc_resample(tbl, split_dates, date) # don't use lead_date
#' vc_resample(tbl, split_dates, -date) # only use lead_date
vc_resample <- function(tbl, split_dates, ...) {
checkmate::assert_tibble(tbl, min.rows = 1L, min.cols = 1L)
if (!lubridate::is.Date(split_dates)) split_dates <- lubridate::as_date(split_dates)
checkmate::assert_date(split_dates, any.missing = FALSE, min.len = 1L, unique = TRUE)
# apply selections
selectors <- rlang::enquos(...)
if (rlang::is_empty(selectors)) {
date_cols <- tbl
} else {
date_cols <- dplyr::select(tbl, !!!selectors)
}
# ensure at least one is a Date
checkmate::assert_true(any(vapply(date_cols, lubridate::is.Date, logical(1))))
# isolate columns to split by
date_cols <- purrr::keep(date_cols, lubridate::is.Date)
# find min and max dates
max_dates <- purrr::lift_dl(pmax)(date_cols, na.rm = TRUE)
min_dates <- purrr::lift_dl(pmin)(date_cols, na.rm = TRUE)
# ensure no splits will be empty
checkmate::assert_true(all(split_dates > min(max_dates, na.rm = TRUE)))
checkmate::assert_true(all(split_dates <= max(min_dates, na.rm = TRUE)))
# create splits
splits <- map(
split_dates,
~rsample:::rsplit(
tbl,
which((max_dates < .x) & complete.cases(date_cols)), # no NA's in analysis
which(min_dates >=.x))
)
# pack splits into rset
rsample:::new_rset(
splits = splits,
ids = paste0("Slice", gsub(" ", "0", format(seq_along(split_dates)))),
attrib = list(strata = names(date_cols)), # just changed
subclass = c("vc_resample", "rset")
) %>%
dplyr::mutate(split_date = split_dates)
}
I get your point (1) about enforcing a structure, and ensuring that certain attributes exist is expected (as you do already), but having a whitelist enforced at the parent-class level seems highly unusual and of unclear benefit. If a downstream rset
consumer can't handle an extra attribute, I'd think that's an issue with the consumer, not the producer.
That being said, if you'd still like to keep the attribute whitelist, I have two related suggestions:
- enforce at creation (
new_rset
) rather than manipulation (mutate.rset
) - warn or error for unsupported names rather than silently dropping attributes
Per Hadley's S3 chapter linked above, the validation could happen in a new validate_rset()
, called on the result of new_rset()
inside a new rset()
function. This gets at your point (2) above, and would also partially address #40.
A nice side effect of not enforcing the whitelist in the rset
methods for dplyr
verbs (or dropping the whitelist entirely) is that you'd soon be able to drop most of dplyr-compat.R
, since if I'm following the web of issues correctly, all dplyr
verbs will soon retain attributes (tidyverse/dplyr#3429).
I'd be glad to work up a PR if any of the above sounds good to you.
In any case, thanks for engaging with me here, and thanks for these sweet tidymodels packages!
from rsample.
Funny timing, Davis submitted an issue that appears to do exactly that type of resampling already: #42
from rsample.
Haha yup, saw that. Didn't even occur to me to just nest()
first, but while that would fix the panel/group aspect, I still need the multi-column date_cols
logic to properly respect the "gap" caused by the forecasting horizon (date
vs lead_date
), and more control over which splits to make (split_dates
).
I suppose I could compute outcomes post-resampling (so horizon doesn't matter yet), like in a recipe
chain, and have that implicitly drop observations where the future value isn't available? Hmm...
from rsample.
Haha yup, saw that. Didn't even occur to me to just
nest()
first, but while that would fix the panel/group aspect, I still need the multi-columndate_cols
logic to properly respect the "gap" caused by the forecasting horizon (date
vslead_date
), and more control over which splits to make (split_dates
).
Maybe contact Davis about that. If there's some aspect of the current approach that is lacking, he would have a better sense of that than I would (since I don't work in time series all that much).
from rsample.
This issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex https://reprex.tidyverse.org) and link to this issue.
from rsample.
Related Issues (20)
- Sorting of strata in training data from initial_split
- Grouped resampling breaks with non-missing `strata = NULL` HOT 1
- `inner_split()`: keep everything inside of `split_args` or not? HOT 1
- inner_split(): better labels
- inner_split(): S3 method to retrive splitting arguments HOT 3
- inner_split(): no initial_split() arguments HOT 2
- inner_split(): global prop argument
- Update naming for elements relating to potato set
- audit for backticked package names
- Use cli errors for `R/bootci.R`
- Use cli errors for `R/caret.R`
- Use cli errors in `R/initial_validation_split.R`
- Use cli errors in `R/labels.R`
- Use cli errors in `R/make_groups.R`, `R/mc.R`, `R/nest.R`
- Use cli errors in `R/misc.R`
- Use cli errors in `R/permutations.R`, `R/reg_intervals.R`
- Use cli errors in `R/rset.R`
- Use cli errors in `R/rsplit.R` HOT 1
- Use cli errors in `R/slide.R`
- Use cli errors in `R/tidy.R`, `R/validation_set.R`, `R/vfold.R`
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rsample.