issue with weights and posteriors comparing v0.9.5.3 and v2.1.1 about dynesty HOT 12 CLOSED

Ok.
There are possibly many issues in play here, but
if you put bootstrap=0 when initializing the nested sampler things go much faster and without issues.
I think there is something going on here with bootstrap. The bootstraping of bounds will naturally lead to lower acceptance rate, but it is unclear why it slows down things so much and lead to issues...

from dynesty.

Thanks, setting bootstrap=0 helps a bit. I was able to make the trace plot at least for v2.1.1; attaching that side by side with v0.9.5.3.

It still looks like there is something odd about the posterior samples in the new version, but there are some other hyperparameteres you can advise on?

output from v0.9.5.3, looks like well behaved posteriors:

output from v2.1.1, very spikey/narrow posteriors

from dynesty.

Thanks. Could you please make a test case that consists solely of

# load the data 
# import module with the likelihood function

dns = DynnestedSampler(module.like,module.prior_transform ,...)
dns.run_nested()
# make a plot or ideally compute some number that indicate there is an issue or not

that would illustrate the problem

So I can easily diagnose the issue/run without digging into the source codes of your files.

Also from running some tests over the weekend one of my impressions is the following. You have here a thin/curved and heavy tailed posterior. In this particular case the ellipsoidal bounding will have particularly tough time. It doesn't really matter for non-uniform samplers, but uniform sampler must ensure that the boundary fully enclose the L>L* region. In this case this leads to large bootstrap enlarge factors.
There could be other issues out there, but at least one thing that come to mind I think we should implement is a warning if bootstrap enlarge factors are too large.

The suggestion of setting bootstrap=0 is not a real solution, it was just testing what's playing a role (and the previous behaviour default behaviour of dynesty was exactly using bootstrap=0, but that does lead to biased posteriors).

from dynesty.

OK, makes sense!

I've updated the example scripts so that it follows the structure you request; i.e., uncovering the call to dynesty.DynamicNestedSampler in the top level script.

It will also automatically make plots; and it will check if the posteriors seem too narrow, by calculating the standard deviation of a few parameters. But, this isn't fool proof because if there are multiple modes the standard deviation is artificially inflated. FWIW, tests for 0.9.5.3 suggest that there shouldn't be multiple modes in most cases. So, looking at the plots is best (I'll try to think of something more clever).

And, of course: I'm very happy to edit or modify the example script if there are issues or something that would make testing this easier.

from dynesty.

Thanks, but it doesn't work as it is:

(pyenv310) skoposov@milkyway:~/curwork/SN_analysis_debug$ python analyze_SN_curved_PL_dnest.py 
Traceback (most recent call last):
  File "/home/skoposov/curwork/SN_analysis_debug/analyze_SN_curved_PL_dnest.py", line 14, in <module>
    import SN_model_fits.result_params as result_params
ModuleNotFoundError: No module named 'SN_model_fits.result_params'

also right now your runs are not deterministic, as you are not passing the rstate to the nested sampler.

from dynesty.

Sorry, I forgot to push the results.params.py file---that should run now?

from dynesty.

yes, thanks it works now. I'll try to take a look at it in next few days

from dynesty.

I have now identified one oversight that was made in #286
Basically when ellipsoids are being split the splitting was not aggressive enough (less agressive than in the past) and that lead to lower efficiencies.
I have a tentative fix in this branch
https://github.com/joshspeagle/dynesty/tree/splitting_fix3
But I'll need to think about this a bit more.

It is possible that there are some other issues as well, but broadly I think the main issues were

• when your posterior is heavily non-gaussian (curved + heavy tailed) then the default bootstrap leads to bounds that are much larger than the L>L* therefore making sampling very ineficient
• on top of that somehow this problem in particular was affected by less agressive split of ellipsoids leading again to non-efficient sampling.

Overall one of the takes from this is that

• uniform sampling is very sensitive to the boundary. In general sampling like rslice/rwalk are not affected by these issues (they have their own ones)

from dynesty.

Okay, after some investigation:

Final last two points

• The key problem in your code was 'maxcall' limit. That forced the sampling to interrupt earlier and thus give incomplete posterior
• Yes, the default mode is now to bootstrap (because that gives a more correct posteror). That leads to slower sampling thus leading you to hit the maxcall boundary

In the #443

• added the warning if bootstrap factor is large (i.e. if the ellipsoidal approximation is bad)
• added a warning if the code stops too early due to maxcall/maxiter

Using the #443 your analyze_ script with bootstrap=0 removed and maxcall limit removed runs fine (although not fast and producing a few warnings about inefficient sampling), but gives correct results
I'll be closing the issue soon.

from dynesty.

OK great!

If I'm understanding correctly, you are saying that the low efficiency is caused by the specifics of the likelihood combined the way the ellipsoidal approximation works?

And the fix for more efficient sampling is to use rwalk or rslice to sample?

I'm also still wondering if there is any configuration that would reproduce the same behavior as v0.9.5.3? (i.e., using the default sampling and giving a similar efficieny)

from dynesty.

Low efficiency is indeed caused by posterior shape and the ellipsoidal approximation not being ideal here.

If you use bootstrap=0 you get behaviour similar to the one you had in 0.9.5.3 -- keep in mind that will lead to potentially biased posteriors (as bootstrap ensures that ellipsoidal boundary is encompassing L>L* )

Without that the options are

• do not force bootstrap=0 the sampling will be slow/(slower)
• use rslice/rwalk samplers
• Use more live-points may help
• Reparametrize the problem so the posterior contours are more ellipsoidal in shape and/or do not have fat tails.

Also you do need to remove the maxcall limit (with the limit in place there is no guarantee that your posterior will be properly sampled); now you'll see a warning in the end if you use maxcall and the fit stops early. (alternatively you can keep the maxcall in place, but then you'll need to check if the effective number of samples you get is sufficient)

from dynesty.

OK, sounds great! Thanks for your help and looking into this.

from dynesty.