Comments (8)
Hi,
I'll start with this example that shows that the uniform sampler does utilise all the cores if you have a heavy to compute function:
import numpy as np
import pytest
import dynesty
import multiprocessing as mp
import dynesty.pool as dypool
nlive = 1000
printing = True
ndim = 10
gau_s = 0.01
def loglike_gau(x):
for i in range(1000000):
1 + 1
return (-0.5 * np.log(2 * np.pi) * ndim - np.log(gau_s) * ndim -
0.5 * np.sum((x - 0.5)**2) / gau_s**2)
def prior_transform_gau(x):
return x
LOGZ_TRUTH_GAU = 0
LOGZ_TRUTH_EGG = 235.856
def test_pool_samplers():
# this is to test how the samplers are dealing with queue_size>1
rstate = np.random.default_rng(2)
with mp.Pool(10) as pool:
sampler = dynesty.NestedSampler(loglike_gau,
prior_transform_gau,
ndim,
nlive=nlive,
sample='unif',
pool=pool,
queue_size=10,
rstate=rstate,
first_update={
'min_eff': 90,
'min_ncall': 10
})
sampler.run_nested(print_progress=printing)
assert (abs(LOGZ_TRUTH_GAU - sampler.results['logz'][-1]) <
5. * sampler.results['logzerr'][-1])
if __name__ == '__main__':
test_pool_samplers()
Regarding what's happening in your case. First there is not enough info at the moment to diagnose (one needs a reproduceable example).
But the key difference of uniform sampling is that the proposal of points within ellipsoids is done in a single thread. And only later the likelihoods are being evaluated in parallel.
A possibility for what you're seeing is that the bounding ellipsoid you get for your problem is extreme (i.e. very elongated, much longer than the length of the cube), in that case most of the proposed points within the ellipsoid will be outside the cube and the code will try again etc, etc and that will all be done in a single thread, before the likelihood function is evaluated.
Another possibility is that your function is just too heavy to pickle and that dominates the overheads.
You can test the first hypothesis by lowering the threshold for the warning here
dynesty/py/dynesty/nestedsamplers.py
Line 705 in 2d49cf9
The second hypothesis you can test by using dynesty.pool.Pool which eliminates the pickling overhead.
finally, if you cannot share a reproduceable example, you should at least share the exact way you call dynesty (with all the information, such as nlive, ndim etc. etc ) and ideally all the output from the problematic unif run.
from dynesty.
Hello,
Thank you for your help. I will address you questions below.
- Here is how I call
dynesty
with "over-subscription" (ndim=7
,npoints=1000
):
with Pool() as MP_pool:
sampler = dynesty.DynamicNestedSampler(Chi2_Tot,Prior,ndim,bound='multi',
sample='unif',
first_update={'min_eff':0.50},
update_interval=500.5,
bootstrap=50,enlarge=1.0,
pool = MP_pool,queue_size=1600)
sampler.run_nested(n_effective=10000,dlogz_init=0.05,nlive_init=npoints,
nlive_batch=500)
- I do not think that pickling is an issue as
dynesty
works efficiently in parallel with other samplers (seerslice
examples in my opening post). Moreover,emcee
and my tests where I evaluate likelihood in parallel usingmap
of multiprocessing pool also work fine and with high parallel efficiency. The former requires enough (>1000) walkers and the latter requires settingchunksize
to at least100
to be really parallel efficient, but I suppose it is due to the fact that the likelihood takes only ~1 ms to evaluate and multiprocessing pool inevitably introduces some overhead. - I think that very elongated ellipsoids might be indeed the cause of the issue. The posterior distributions of some of the parameters have very heavy tails going to the limits of the model parameters' ranges. If this is the case, is there any way to circumvent the issue? I really prefer the
unif
sampler as it is more robust and does not require to estimate the proper number ofwalks
orslices
.
I will send a working example privately.
from dynesty.
Hi,
I don't quite agree with your comment number 2. The reason is that when you use rslice or rwalk with say walks=50, it means that each thread will execute at least 50 likelihood calls per one pickle. When using the uniform sampler it'll be always 1 call per 1 pickle, because of that rslice/rwalk will look better in terms of parallelisation.
But certainly option 3 is quite possible.
And I think there are are a few possible improvements in that area. One of which is doing the ellipsoidal sampling in parallel (it will have an overhead of sending over the ellipsoidal bounds), the other one is detecting the crazy ellipsoids and maybe reshaping them somewhat
from dynesty.
An update on my side.
I had a bit of time to look into this. And a couple of points
- If you make your likelihood function computationally heavier, the parallelization works as it should, so it's clearly just an inefficiency that is present for fast likelihood functions.
- It seems there are some issues (see #427), but maybe other ones that lead to inefficient sampling in some cases. I'm still investigating this.
from dynesty.
Hello,
Thanks for the update.
If fast likelihood functions cause the sampler to lose parallel efficiency, perhaps adding an option to vectorize them (i.e., evaluate likelihood at more than one point per call) will help? For example, this is implemented in JohannesBuchner/UltraNest and seems to be efficient if the number of requested points per one call is big enough. If the likelihood does not support vectorization itself, it can be easily "wrapped" by mapping:
def Chi2_Tot_Vect(points):
return np.array(list(map(Chi2_Tot,points)))
from dynesty.
I think the #427 addresses some of the issues of updating bounds, so I am reasonably convinced that now things are functioning as they should be.
But it is clear that in this case
- the ellipsoidal sampling in your problem is inefficient in the end (I don't have time to investigate that, but I assume that must be related to the shape of the posterior)
- If the likelihood is very fast the parallelisation of the uniform sampler is not very efficient. That has various solutions, but that's a long-term project.
from dynesty.
I'm closing this issue for the time being.
The different parallelization scheme for unif sampler would be still be good, but I don't think there is a bug per se.
from dynesty.
Hello,
Sorry for not replying earlier.
Yes, I agree, it is not a bug, just a specific feature of implementation.
Thank you for your help.
from dynesty.
Related Issues (20)
- Recover partial chains from the dynesty.save checkpoint file
- An issue about dynesty posterior HOT 6
- live point's likelihood not valid HOT 6
- Documentation incorrect for Pool helper object HOT 1
- Importing Nested Sampling chains from file to plot HOT 2
- Using dynesty with npdim HOT 8
- Questions about dyplot.cornerplot HOT 4
- Questions about DynamicNestedSampler setting HOT 4
- Periodic parameters improvement by rotating them
- Ellipsoid check failed HOT 2
- tqdm example HOT 4
- Discretised prior in Dynesty HOT 7
- Loglstar and logZ values HOT 1
- Importance weight PDF shape HOT 4
- How to access posteriors? HOT 1
- Doubt on sampling methods HOT 6
- Run-to-run instability of logZ HOT 8
- Using weighted samples HOT 3
- blob issues when add_live HOT 4
- log means log_e or log_10? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dynesty.