Comments (19)
bayesfactor_inclusion
-
text
- only results. -
text_full
- Give full explanation about model averaging + posterior probability. -
table
- basicallyas.data.frame
-
table_full
- same astable
?? - docs...
- tests...
-
bayesfactor_models
-
text
- method + denominator + {best + worst model} -
text_full
- with BF computation method. -
table
- basicallyas.data.frame
-
table_full
- give also "BF01 compared to best model` column. - docs...
- tests...
-
bayesfactor_parameters
-
text
- what null (once), BF for each, by name. -
text_full
- same astext
?? -
table
- basicallyas.data.frame
+ row for null and side -
table_full
- same astable
?? - docs...
- tests...
-
bayesfactor_restricted
-
text
- ??? -
text_full
- ??? -
table
- basicallyas.data.frame
-
table_full
- same astable
?? - docs...
- tests...
-
Notes to self:
- Should use
interpret_bf
internally fortext
, with argrules
.
from report.
@DominiqueMakowski did I miss anything? Any ideas for the "???"?
Any guidelines on how text
and text_full
/ table
and table_full
should differ?
Is there a timeline on this?
How are BFs dealt with currently in the estimate
methods (etc)?
from report.
Just so I know I'm in the right direction, is this what you have in mind? (I'm afraid I haven't been paying close enough attention to report
...)
library(bayestestR)
library(report)
mo0 <- lm(Sepal.Length ~ 1, data = iris)
mo1 <- lm(Sepal.Length ~ Species, data = iris)
mo2 <- lm(Sepal.Length ~ Species + Petal.Length, data = iris)
mo3 <- lm(Sepal.Length ~ Species * Petal.Length, data = iris)
BFmodels <- bayesfactor_models(mo1, mo2, mo3, denominator = mo0)
inc_bf <- bayesfactor_inclusion(BFmodels, prior_odds = c(1,2,3), match_models = TRUE)
bf_report <- report(inc_bf)
to_table(bf_report) # same for both full = FALSE / TRUE
#> Terms | Pr(prior) | Pr(posterior) | Inclusion BF
#> ----------------------------------------------------------------------------
#> Species | 0.429 | 0.946 | 3.895e+55
#> Petal.Length | 0.286 | 0.946 | 6.891e+26
#> Species:Petal.Length | 0.429 | 0.054 | 0.038
#> | | |
#> Across matched models only, | | |
#> with custom prior odds (1, 2, 3). | | |
to_text(bf_report, full = FALSE)
We found extreme evidence (BF > 999) in favour of including Species; extreme evidence (BF > 999) in favour of including Petal.Length; strong evidence (BF = 0.04) against including Species:Petal.Length.
to_text(bf_report, full = TRUE)
Bayesian model averaging (BMA) was used to obtain the average evidence for each predictor. Since each model has a prior probability (here we used subjective prior odds of 1, 2, 3), it is possible to sum the prior probability of all models that include a predictor of interest (the prior inclusion probability), and of all models that do not include that predictor (the prior exclusion probability). After the data are observed, we can similarly consider the sums of the posterior modelsβ probabilities to obtain the posterior inclusion probability and the posterior exclusion probability. The change from prior to posterior inclusion odds is the Inclusion Bayes factor. For each predictor, averaging was done only across models that did not include any interactions with that predictor; additionally, for each interaction predictor, averaging was done only across models that contained the main effect from which the interaction predictor was comprised. This was done to prevent Inclusion Bayes factors from being contaminated with non-relevant evidence (see Mathot, 2017). We found extreme evidence (BF > 999) in favour of including Species; extreme evidence (BF > 999) in favour of including Petal.Length; strong evidence (BF = 0.04) against including Species:Petal.Length.
from report.
I was mainly thinking about this case:
report(insight::download_model("stanreg_lm_1"), effsize = "cohen1988", standardize = "refit", test = c("pd", "bf"))
Like implementing the reporting of BFs in the parameters.
Long story short, the table in to_table
and to_fulltable
(for models) come directly from the parameters table (with one additional column from performance). Then, the text processor picks up whatever indices are present in this table and puts it into text. This is done in the text_parameters.R
file. I didn't want to specifically assign this to you it was more like hey we should brainstorm on how to do it best
I wasn't even thinking about reports for BF inclusion and models yet, but that's awesome!!!
from report.
And I believe it will be super useful!
from report.
Wow, then I totally overshot this, hu? π
Not sure I follow exactly - what would you like to add to report.stanreg
? What is currently missing (to add to the to-do list above)? And will writing report.bayesfactor_parameters
help?
I also can't seem to run your example (even after updating all dependencies...):
report::report(insight::download_model("stanreg_lm_1"), effsize = "cohen1988", standardize = "refit", test = c("pd", "bf"))
#> Registered S3 method overwritten by 'xts':
#> method from
#> as.zoo.xts zoo
#> Computation of Bayes factors: sampling priors, please wait...
#> Loading required namespace: logspline
#> Error in UseMethod("format_value"): no applicable method for 'format_value' applied to an object of class "NULL"
Created on 2019-08-15 by the reprex package (v0.3.0)
from report.
I also can't seem to run your example (even after updating all dependencies...):
have you updated the latest github's performance?
report::report(insight::download_model("stanreg_lm_1"), test = c("pd", "bf"))
#> Registered S3 method overwritten by 'xts':
#> method from
#> as.zoo.xts zoo
#> Computation of Bayes factors: sampling priors, please wait...
#> Loading required namespace: logspline
#> We fitted a Bayesian linear model to predict mpg with wt. The model's explanatory power is substantial (R2's median = 0.75, adj. R2 = 0.71). The model's intercept is at 37.25. Within this model:
#>
#> - The effect of wt has a probability of 100% of being negative and can be considered as large (median = -5.34, 89% CI [-6.23, -4.46], std. median = -0.87).
Created on 2019-08-15 by the reprex package (v0.3.0)
Currently, the text doesn't mention the BFs that are present in the table. I'd like to neatly integrate it with the current textual output... cf. the discussion on the best reporting sentence in the Bayesian indices paper
from report.
Ah I see. I think for BF reporting could be put the in the parentheses, as I mostly see done in the psych papers I read, and reported as:
rep_bf <- function(bf, ...){
ori_bf <- bf
dir <- ifelse(log(bf) < 0,"BF01", "BF10")
bf[bf < 1] <- 1 / bf[bf < 1]
paste0(
parameters::format_bf(bf, name = dir),
", considered ",
report::interpret_bf(ori_bf, include_value = FALSE, ...),
" an effect."
)
}
rep_bf(exp(-4:4)) # is vectorized
#> [1] "BF01 = 54.60, considered very strong evidence against an effect."
#> [2] "BF01 = 20.09, considered strong evidence against an effect."
#> [3] "BF01 = 7.39, considered moderate evidence against an effect."
#> [4] "BF01 = 2.72, considered anecdotal evidence against an effect."
#> [5] "BF10 = 1.00, considered anecdotal evidence in favour of an effect."
#> [6] "BF10 = 2.72, considered anecdotal evidence in favour of an effect."
#> [7] "BF10 = 7.39, considered moderate evidence in favour of an effect."
#> [8] "BF10 = 20.09, considered strong evidence in favour of an effect."
#> [9] "BF10 = 54.60, considered very strong evidence in favour of an effect."
Created on 2019-08-15 by the reprex package (v0.3.0)
If you agree, feel free to add this in the appropriate place - I don't seem to understand the coding structure of the .report_regressions
function/s...
from report.
the structure is a bit convoluted indeed (but still simpler than it was ^^). Basically, there are primitives in text_parameters that are called in text_parameters_regression, which is used in report_regression (which is just an internal that can be applied to all regressions to avoid having the same code everywhere). Let me try to provide a start
from report.
- pd and ROPE and BF
There is moderate evidence (BF = 0.29) in favour of an absence of effect of X, which has a probability of 90.14% of being negative (Median = -0.03, 89% CI [-0.05, 0.01]), and can be considered as very small (Std. Median = -0.09) and not-significant (6.42% in ROPE).
- pd and ROPE
The effect of X has a probability of 90.14% of being negative (Median = -0.03, 89% CI [-0.05, 0.01]) and can be considered as very small (Std. Median = -0.09) and not-significant (6.42% in ROPE).
- pd and BF
There is moderate evidence (BF = 0.29) in favour of an absence of effect of X, which has a probability of 90.14% of being negative (Median = -0.03, 89% CI [-0.05, 0.01]) and can be considered as very small (Std. Median = -0.09).
- ROPE and BF
There is moderate evidence (BF = 0.29) in favour of an absence of effect of X (Median = -0.03, 89% CI [-0.05, 0.01]) which can be considered as very small (Std. Median = -0.09) and not-significant (6.42% in ROPE).
- Only pd
The effect of X has a probability of 90.14% of being negative (Median = -0.03, 89% CI [-0.05, 0.01]) and can be considered as very small (Std. Median = -0.09).
- Only ROPE
The effect of X (Median = -0.03, 89% CI [-0.05, 0.01]) can be considered as very small (Std. Median = -0.09) and not-significant (6.42% in ROPE).
- Only BF
There is moderate evidence (BF = 0.29) in favour of an absence of effect of X (Median = -0.03, 89% CI [-0.05, 0.01]) which can be considered as very small (Std. Median = -0.09).
from report.
I would change "in favour of an absence of effect of X" simply to "against an effect of X", and have BF01 / BF10 accordingly to also reflect that more explicitly (or just allow values smaller than 1).
from report.
agreed
from report.
The tricky part is to implement these different cases in a not-horrible way
from report.
from report.
Currently, the assemblage is made here:
report/R/text_parameters_regression.R
Lines 46 to 53 in d8212bc
Which calls this:
Lines 345 to 352 in d8212bc
Which get its content, for instance for significance, from here:
Lines 99 to 125 in d8212bc
from report.
Okay... I think I figured it out(?)
I found the best (easiest?) place to put it was inside the parentheses, next to the ROPE - both moved to the end of the perentheses. So it looks like this:
library(report)
mod <- insight::download_model("stanreg_lm_1")
#> Registered S3 method overwritten by 'xts':
#> method from
#> as.zoo.xts zoo
report(mod, test = c("pd", "bf","ROPE"))
#> Computation of Bayes factors: sampling priors, please wait...
#> Loading required namespace: logspline
We fitted a Bayesian linear model to predict mpg with wt. The model's explanatory power is substantial (R2's median = 0.75, adj. R2 = 0.71). The model's intercept is at 37.25. Within this model:
- The effect of wt has a probability of 100% of being negative and can be considered as large and significant (median = -5.34, 89% CI [-6.23, -4.46], 0% in ROPE, std. median = -0.87, BF10 > 999, considered extreme evidence in favour of the effect).
Created on 2019-08-16 by the reprex package (v0.3.0)
What do you think?
from report.
(currently BF is not in .text_parameters_significance
... seemed too clumsy to write something like:
The effect of wt has a probability of 100% of being negative and can be considered as large, significant and strongly not supported by the data (median = -5.34, 89% CI [-6.23, -4.46], 0% in ROPE, std. median = -0.87, BF01 > 999, considered extreme against in favour of the effect).
from report.
I am not a fan of the interpretation within the parenthesis :)
overall I think it's a good path to try as much as possible to dissociate indices and their interpretation, rather than cramming all interpretation on one side, and all values within brackets, which can lead to confusion about which conclusion is based on what number
let me see if what can I do here
from report.
This is not in because it's currently only sexist, but we'll need to re-add that to allow more flexibility
from report.
Related Issues (20)
- Unclear reporting HOT 1
- The model's explanatory power is "substantial" HOT 1
- What's the best way to provide appropriate attribution/citation? HOT 2
- Support models of class `gamm` HOT 2
- emmeans and beta regression support
- `report_participants()` should set age as numeric, accept more choices for gender
- report fails when model formulat built with stats::reformulate
- oneway.test: `Error in paste0(out$interpretation, " (", out$statistics, ")"): object 'out' not found`
- Add support for `kruskal.test()`
- Error: bad 'data': object 'data_std' not found HOT 3
- What is the expected behaviour for report(estimate_contrasts(model))?
- To-do: Clean-up names in outputs (`airquality$Month` instead of `as.factor(airquality$Month)`) HOT 1
- Why do the standardized beta values and CIs of a glm poisson regression model not differ from the unstandardized ones? HOT 8
- New CRAN release? HOT 1
- When using stats::t.test, the report() and report_table() function output displays "95 % CI" even if, say, conf.level = 0.975 HOT 1
- CRAN submission revedep check failed (*** Strong rev. depends ***: easystats SqueakR) HOT 7
- report does not work with BayesFactor models
- report_sample(): add indices names in caption instead of table HOT 1
- support for quantile regression
- Report Summary for Time Series Model Stats
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from report.