Comments (2)
I realize that I've made this issue more complicated than needed, and performance wasn't the right framing for understanding the problem. Closing in favor of #509
from pointblank.
Actually, the performance problem scales with the size of the data frame too (so you can ignore the convoluted reprex above). This is because hashing the (in most cases, global) environment will often also hash the table (and everything else) that's also available in the environment.
With that in mind, a newer, more minimal reprex on a 1-million row dataframe (using CRAN version this time, but the problem also exists on dev):
library(pointblank)
big_table <- small_table[sample(nrow(small_table), 1e6, replace = TRUE),]
agent <- create_agent(big_table)
profvis::profvis({
agent %>%
col_vals_gt(
columns = c("c", "d", "e"),
value = vars(a),
preconditions = . %>%
dplyr::select(a, c, d, e)
)
})
The slowdown is specifically being driven by hashing the vars()
quosure (sha1.formula()
) and the preconditions
function (sha1.function()
). Both methods hash the attached environment by default:
And of course, the problem also scales with the number of validation steps added to the agent:
system.time({
agent %>%
col_vals_gt(
columns = c,
value = vars(a),
preconditions = . %>%
dplyr::select(a, c)
) %>%
col_vals_gt(
columns = d,
value = vars(a),
preconditions = . %>%
dplyr::select(a, d)
) %>%
col_vals_gt(
columns = e,
value = vars(a),
preconditions = . %>%
dplyr::select(a, e)
)
})
#> user system elapsed
#> 2.07 0.39 6.72
Resolving value
and preconditions
to string before hashing resolves the performance issue (I'm testing this on a local branch):
devtools::install_github("yjunechoe/pointblank@no_env_hash")
library(pointblank)
big_table <- small_table[sample(nrow(small_table), 1e6, replace = TRUE),]
agent <- create_agent(big_table)
system.time({
agent %>%
col_vals_gt(
columns = c,
value = vars(a),
preconditions = . %>%
dplyr::select(a, c)
) %>%
col_vals_gt(
columns = d,
value = vars(a),
preconditions = . %>%
dplyr::select(a, d)
) %>%
col_vals_gt(
columns = e,
value = vars(a),
preconditions = . %>%
dplyr::select(a, e)
)
})
#> user system elapsed
#> 0.04 0.02 0.14
The "resolve to string before hashing" solution currently fails the multiagent tests (but passes all others) and it will break backwards compatibility as is. But since the multiagent has all the info that was used to generate the original hash, I wonder if we can (optionally) have the hash for each validation step be re-generated by the multiagent for the purposes of alignment (which I believe would still guarantee correct alignment for older agents)?
from pointblank.
Related Issues (20)
- Regex for Microsoft SQL Server HOT 1
- Integrate `scan_data()` elements into validation reports
- A possible switch to tidyeval for `values` and `left`/`right` arguments HOT 1
- Better handling for bad column selections HOT 3
- Support `where()` predicates for tidyselect in `info_columns()` HOT 2
- Validation steps sometimes do not align in the multiagent HOT 3
- Show labels in console output when using `interrogate()`
- col_exists shows no fail when column missing HOT 1
- `col_vals_equal()` doesn't match two `NA` values HOT 3
- `ggplot2::ggsave()` error with `pointblank::scan_data()` HOT 5
- Arithmetic overflow error converting expression to data type int.
- `quarto render --to html` makes some html tags visible in the validation report HOT 3
- Release pointblank 0.12.0 HOT 5
- Vignette for dynamic patterns in validation functions
- pointblank and tidyselect 1.2.1 HOT 1
- Conflicting validation report when using YAML in {targets} compared to interactive mode HOT 1
- Column names with spaces causes incorrect validation error HOT 1
- Release pointblank 0.12.1
- Bigquery with bigrquery . unable to find method for dbGetQuery
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pointblank.