Random effects capturing whole variance about projpred HOT 5 CLOSED

I'm following up on the email as well.

Nonetheless, for other interested people that are suffering from similar effects, one possible solution to this, if the underlying issue is that the random effects are soaking the entire variance, is to restrict the search to select first the population effects and only later project the random effects. You can do this by passing a custom search_terms. I'm putting a vignette together that showcases how to do this for different cases and should be up soonish.

from projpred.

So, there are several things to point out here. On the first hand, random effects models are indeed more complex and therefore usually need more posterior draws to be projected. We have changed the default on the develop branch but it's not yet pushed into CRAN. You can pass different ndraws and ndraws_pred arguments to varsel and cv_varsel to see how that affects the performance. Most likely something like ndraws = 20 and ndraws_pred = 100 should work better. There is no fixed rule but increasing the number of draws would always increase the accuracy of the projections.

That said, there are indeed models where the group structure can potentially explain the whole variance, we are aware of this and we will inspect this phenomenon further down the road and see if we find a robust way of addressing it. Thanks for reporting!

from projpred.

Thanks, @AlejandroCatalina. I've tried increasing to ndraws = 50 and ndraws_pred = 400, but still see the same effect.

In the meantime I'll just present the full model in my work, but will keep an eye on projpred development. If you have any other suggestions I'd be grateful; and if you need an example dataset for this phenomenon then please let me know.

Thanks very much for your help.

from projpred.

I see! Another thing that comes to mind is to inspect the search path term by term. That is, you can look at the returned object vsel$search_path$sub_fits, which is a list of nterms containing the incremental projections. I expect some projections to behave alright and break at some point. That would give you more information regarding which group parameters are more problematic. A hard limit for ndraws_pred is the number of draws in your posterior, so an exact projection can be achieved by projecting all 4000 (or whatever) draws, albeit extremely expensive. I'm not saying you should project all of them but it can be another way of debugging what's going on.

Nonetheless, we'll keep working on this and hopefully find a good way of balancing things out. You can send me the model (a .RDS object with the rstanarm fit should suffice I think?) and I can play with it a bit to see if there's anything that we haven't realised yet.

from projpred.

Thanks @AlejandroCatalina.

• I've tried increasing ndraws_pred to half (5000) the number of draws, but the result is the same. I can increase to the full number of draws, but suspect from what I've seen so far there'll be no difference.
• I'm afraid I can't work out how to tell if the projections are breaking. I see that vsel$search_path$sub_fits contains a list of ndraws model fits, with the terms incrementally added. Should I be comparing the fit of those models each time a new term is added: i.e. compare the mean AIC across all ndraws models for each new term (each item in the vsel$search_path$sub_fits list)? Or should I be using some other metric to compare the fits?

I'll email you a link to the model .RDS file.

Thanks again for your help - it's much appreciated. I'm really keen to use projpred for this if I can!

from projpred.