dfm / corner.py Goto Github PK
View Code? Open in Web Editor NEWMake some beautiful corner plots
Home Page: http://corner.readthedocs.io
License: BSD 2-Clause "Simplified" License
Make some beautiful corner plots
Home Page: http://corner.readthedocs.io
License: BSD 2-Clause "Simplified" License
Hi,
Not too much of an issue here, more of an additional feature.
The idea is to multiply the 1d and 2d histograms curves/surfaces with the respective prior of each parameter, so that the full "posterior distribution" is plotted - not just the likelihood of the raw samples.
It seemed that the "weights" arguments served another purpose, so I hacked something together. It actually holds in a tiny amount of lines.
Only one new input argument "priors" is needed. It is a N-list of (callable, dict) (for N parameters). The callable argument being the function to call with the dict as *_kwargs:
prior_prob = callable(x, *_dict).
The only drawback is the loss of the parameter "histtype" (hist_kwargs["histtype"]) for the 1d histograms, since they are now always plotted with ax.plot(...) whether they have smoothing or not.
Cheers,
G. Schworer @ceyzeriat
Would it be easy to change title_fmt
to take an array of formats so that each parameter could get its own? On a related note, the default .2f
could be changed to give 2 significant figures in the uncertainty, rather than a fixed decimal place.
Is there any way to get dashed (or other styles) histograms? I have not been able to get it working with standard matplotlib arguments
Please rotate bottom right histogram 90 degrees and remove axis labels so that x and y axis layouts are the same.
In an email on 2013-11-25, Tom Loredo (Cornell) suggested:
Two suggestions:
- There does not seem to be a universal terminology for these plots,
but the terms I see most often are "pairs plots" and "scatterplot
matrices" (I'd never seen "corner plot" or "triangle plot" before).
I suggest putting these in your README.rst so people
will more easily find your solution via a web search.- One option I have in mine is to plot the posterior samples colored
according to time in the chain, and transparent. I use a color map
from blue to green to red, or something like that. It provides a
graphical 2-D convergence diagnostic: if everything is mixed, the
predominant color is brownish; but if the chain gets stuck anywhere,
spots of a specific color stand out. In particular, if you don't
discard enough "burn in," there is often a blue region sitting away
from the brownish region. Probably too busy for a final plot, but
helpful when you're exploring/tuning. My statistician collaborators
at Duke like this and have started using it.
Thanks Tom!
When I include, for example, quantiles=[0.16, 0.5, 0.84]
in a call to corner.corner I receive the following error:
ValueError Traceback (most recent call last)
/astro/mark/casa/casa-stable-4.7.74/lib/python2.7/casapy.py in <module>()
1
2
----> 3
4
5
/astro/mark/.casa/lib/python2.7/site-packages/corner/corner.pyc in corner(xs, bins, range, weights, color, smooth, smooth1d, labels, label_kwargs, show_titles, title_fmt, title_kwargs, truths, truth_color, scale_hist, quantiles, verbose, fig, max_n_ticks, top_ticks, use_math_text, hist_kwargs, **hist2d_kwargs)
254 # Plot quantiles if wanted.
255 if len(quantiles) > 0:
--> 256 qvalues = quantile(x, quantiles, weights=weights)
257 for q in qvalues:
258 ax.axvline(q, ls="dashed", color=color)
/astro/mark/.casa/lib/python2.7/site-packages/corner/corner.pyc in quantile(x, q, weights)
411
412 if weights is None:
--> 413 return np.percentile(x, 100.0 * q)
414 else:
415 weights = np.atleast_1d(weights)
/astro/mark/casa/casa-stable-4.7.74/lib/python2.7/site-packages/numpy/lib/function_base.pyc in percentile(a, q, axis, out, overwrite_input)
3077 a = np.asarray(a)
3078
-> 3079 if q == 0:
3080 return a.min(axis=axis, out=out)
3081 elif q == 100:
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
I've recently moved institution and so am working on a fresh set-up and initially thought that I hadn't set something up correctly. But then it occurred to me to check if there had been any recent changes to corner. Sure enough, some changes were made 9 days ago including this:
- return np.percentile(x, [100. * qi for qi in q])
+ return np.percentile(x, 100.0 * q)
I suspect this is the source of the error. Was there a good reason for this change?
And the reason is fairly simple: np.hist* neither (as per: http://stackoverflow.com/questions/3610040/how-to-create-the-histogram-of-an-array-with-masked-values-in-numpy )
The fix is quite simple and I applied it to my fork of the code, as I couldn't commit to a new branch on dfm/triangle.py. Not sure how to merge that back in master if you are interested.
Btw, maskedarrays (np.ma) are quite a powerful tool, especially to exclude (hide) data out of prior range without actually throwing it away, or focus on a specific region, etc.
Cheers,
Guillaume
Hey @dfm -
pip install --upgrade traiangle_plot
gives them this new power?Thanks!
pip install triangle gives me error b/c I didn't have Cython.Distutils
after installing Cython, seems to work.
Hello,
I have two corner plots that I overlay using the fig=fig
keyword. It is only with two parameters and I wanted to add a legend in the empty corner. One plots the contours blue and the other one plots red. Is there a simple way to do this with corner.py?
Thank you,
Kara
Right now, it hard sets the limits to the range of the last set of samples that are plotted. This isn't very useful. You can hack this by using the extent
keyword but it would probably be good to default to something better... like only letting the limits get bigger. Eh? ๐
1 - Is there any quick way to do this with corner? if not, then its unclear how to use this with weights, what is the shape of weights! because in the documents its written [nsmaples,] isn't should be [nsamples,ndim] as the same shape of x the sample array?
2 - whenever I try using weights, it gives: TypeError: hist() got multiple values for keyword argument 'weights'
Thanks in advance for your help.
The help text (corner.py, line 65) says
show_titles : bool
Displays a title above each 1-D histogram showing the 0.5 quantile
with the upper and lower errors supplied by the quantiles argument.
but actually the errors in the titles are not affected by the quantiles argument (corner.py, line 269) but instead are hard-coded:
q_16, q_50, q_84 = quantile(x, [0.16, 0.5, 0.84],
weights=weights)
q_m, q_p = q_50-q_16, q_84-q_50
# Format the quantile display.
fmt = "{{0:{0}}}".format(title_fmt).format
title = r"${{{0}}}_{{-{1}}}^{{+{2}}}$"
title = title.format(fmt(q_50), fmt(q_m), fmt(q_p))
Shouldn't the the array passed to quantile use the quantiles argument for the lower and upper bounds? If not, shouldn't the help text say that the errors are hard-coded to the 68% confidence region?
Please write an option for overplotting multiple chains on one corner plot (in different colors and/or line styles), with the largest chain extent setting the plot parameters. While you're at it, make colors/line styles options in triangle.corner.
I should be able to set duplicate=True
and have it plot both the upper triangle and the lower triangle in a full square with every 2-d plot duplicated.
Seems to be in physical units at the moment.
I've just added a lot of new options (c.f. #41) and the docs were already out of date... need to fix that!
This must be a solved problem by now. Ivezic had a suggestion in his textbook, we could think about tracking it down.
The smoothing implemented in corner.py does not handle strongly correlated variables very well.
The smoothing for 2D histograms is done with the scipy gaussian_filter function, which is isotropic. If your variables are highly correlated this is pretty distorting, since it smooths points in the wrong directions. See the Delta-M versus h subplot in the attached image, where the contour is enlarged a lot in one direction.
The easiest way to solve this in general is to either supply a covariance matrix to use an elliptical smoothing kernel, or transform to variables with unit covariance before smoothing. Neither of those is a simple change here, unfortunately.
Hi! Above all I want to thank you for this greate library which I discovered while working with PyMC (https://users.obs.carnegiescience.edu/cburns/ipynbs/PyMC.html)
I was trying to learn more from the examples in test.py but I got this error:
Testing 'hist2d'
.. cutoff
.. cutoff2
.. basic
.. color
.. levels1
.. levels2
.. filled
.. smooth1
.. smooth2
.. philsplot
.. lowN
Traceback (most recent call last):
File "/home/vital/git/pyResources/pyResources/Tutorials/SearchForLineInTextFile.py", line 110, in
test_hist2d()
File "/home/vital/git/pyResources/pyResources/Tutorials/SearchForLineInTextFile.py", line 49, in test_hist2d
_run_hist2d("lowN", N=20)
File "/home/vital/git/pyResources/pyResources/Tutorials/SearchForLineInTextFile.py", line 31, in _run_hist2d
corner.hist2d(x, y, ax=ax, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/corner.py", line 561, in hist2d
cmap=white_cmap, antialiased=False)
File "/usr/local/lib/python2.7/dist-packages/matplotlib/init.py", line 1812, in inner
return func(ax, _args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/matplotlib/axes/_axes.py", line 5652, in contourf
return mcontour.QuadContourSet(self, _args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/matplotlib/contour.py", line 1424, in init
ContourSet.init(self, ax, _args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/matplotlib/contour.py", line 863, in init
self._process_args(_args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/matplotlib/contour.py", line 1445, in _process_args
x, y, z = self._contour_args(args, kwargs)
File "/usr/local/lib/python2.7/dist-packages/matplotlib/contour.py", line 1540, in _contour_args
self._contour_level_args(z, args)
File "/usr/local/lib/python2.7/dist-packages/matplotlib/contour.py", line 1189, in _contour_level_args
raise ValueError("Contour levels must be increasing")
ValueError: Contour levels must be increasing
Thanks for any advice. Also if you have any other examples I should consider first I would love to hear your opinion.
Thanks
I have K sets of points, each of which is a large number N_k of objects in D dimensions. I want to plot these all on the same corner plot in different colors. How do I do that? I am happy to write the patch.
I am using the triangle plots package together with emcee in a very simple minded way. Using emcee I generated a list of about 1,000,000 samples in 8 dimensions (using 1000 walkers and 1050 samples). and then I invoke the corner plot here is a snippet of code:
sampler = emcee.EnsembleSampler(1000, ndims,
ln_post_prob, args=(cs_lib, struct, bnds))
sampler.run_mcmc(starts, 1050)
num_samples = (1000 * (1050 - 50))
samples = sampler.chain[:, 50:, :].reshape((num_samples, ndims))
lbls = [r"$a_{%d}$" % i for i in range(ndims)]
ranges = [(-0.001, (0.001 + 1.1*samples[:,i].max())) for i in range(ndims)]
fig = tplot.corner(samples, labels=lbls, extents=ranges)
fig.savefig(fig_file_name)
During the MCMC sampling memory consumption of the python process is about 132MB. When the system attempts to create a corner plot using this package immediately after the sampling, memory consumption jumps to 840MB.
Repeating the exercise with 8,000,000 samples in 8 dimensions (using 4000 walkers and 2050 samples) gives memory consumption of about 624MB during sampling and then, during corner plot creation, climbs to 4.8GB.
I recognize that the growth in consumption (as a function of number of samples) is less than linear, but 4.8GB to make a corner plot for 8,000,000 samples in 8 dimensions seems excessive to me.
If this is a problem in the underlying matplotlib, I apologize for raising the issue here here, if there is some set of options that will reduce memory consumption that I neglected to use, then I apologize for not seeing them.
I've bugged Dan about this before on twitter. :)
There's considerable confusion caused by names in this module: The module name is triangle
, the pypi name is "triangle_plot," the repo name is "triangle.py," and the main entry point within the package is corner()
. Results of the confusion: #39 and #28.
I suggest renaming all of these to simply corner
, which is miraculously unclaimed on PyPI. You get:
pip install corner
from corner import corner
or import corner; corner.corner(...)
(I'm not so set on the name, just the consistency.)
Two possible downsides:
corner
module or package (despite not being named "corner"). However, I didn't see any obvious candidates for this when searching "corner" on PyPI.Any thoughts on the color of this bikeshed?
Greetings! I run your few line example exactly as it appears on your web sites, and it crashes as follows:
$ python testCorner0.py
Traceback (most recent call last):
File "testCorner0.py", line 6, in
figure = corner.corner(samples)
File "/work/myPython/epd_free-7.3-1-rh5-x86/lib/python2.7/site-packages/corner.py", line 330, in corner
*_hist2d_kwargs)
File "/work/myPython/epd_free-7.3-1-rh5-x86/lib/python2.7/site-packages/corner.py", line 519, in hist2d
ax.contour(X2, Y2, H2.T, V, *_contour_kwargs)
File "/work/myPython/epd_free-7.3-1-rh5-x86/lib/python2.7/site-packages/matplotlib-1.5.0_511.gf420992-py2.7-linux-i686.egg/matplotlib/init.py", line 1846, in inner
return func(ax, _args, *_kwargs)
File "/work/myPython/epd_free-7.3-1-rh5-x86/lib/python2.7/site-packages/matplotlib-1.5.0_511.gf420992-py2.7-linux-i686.egg/matplotlib/axes/_axes.py", line 5641, in contour
return mcontour.QuadContourSet(self, _args, *_kwargs)
File "/work/myPython/epd_free-7.3-1-rh5-x86/lib/python2.7/site-packages/matplotlib-1.5.0_511.gf420992-py2.7-linux-i686.egg/matplotlib/contour.py", line 1448, in init
ContourSet.init(self, ax, _args, *_kwargs)
File "/work/myPython/epd_free-7.3-1-rh5-x86/lib/python2.7/site-packages/matplotlib-1.5.0_511.gf420992-py2.7-linux-i686.egg/matplotlib/contour.py", line 874, in init
self._process_args(_args, *_kwargs)
File "/work/myPython/epd_free-7.3-1-rh5-x86/lib/python2.7/site-packages/matplotlib-1.5.0_511.gf420992-py2.7-linux-i686.egg/matplotlib/contour.py", line 1469, in _process_args
x, y, z = self._contour_args(args, kwargs)
File "/work/myPython/epd_free-7.3-1-rh5-x86/lib/python2.7/site-packages/matplotlib-1.5.0_511.gf420992-py2.7-linux-i686.egg/matplotlib/contour.py", line 1564, in _contour_args
self._contour_level_args(z, args)
File "/work/myPython/epd_free-7.3-1-rh5-x86/lib/python2.7/site-packages/matplotlib-1.5.0_511.gf420992-py2.7-linux-i686.egg/matplotlib/contour.py", line 1200, in _contour_level_args
raise ValueError("Contour levels must be increasing")
ValueError: Contour levels must be increasing
My OS, below. Thanks, and regards to all, David.
$ lsb_release -a
LSB Version: :core-3.1-ia32:core-3.1-noarch:graphics-3.1-ia32:graphics-3.1-noarch
Distributor ID: ScientificSL
Description: Scientific Linux SL release 5.5 (Boron)
Release: 5.5
Codename: Boron
Hi all,
I ran into the following test errors on a fresh install of corner
today:
https://gist.github.com/tomr-stargazer/b2271fda034533d084f6db94f163fdc9
relevant software versions:
nosetests --version
nosetests version 1.3.7
python -c 'import matplotlib; print(matplotlib.version)'
2.0.2
python --version
Python 3.6.0 :: Anaconda custom (x86_64)
max must be larger than min in range parameter
But aren't some of us astronomers??
To allow for much larger data sets, it would be nice if the array_like argument could accept a generator. This would require the "data structure" for the plot to be a "histogram" (a collection of bins, and for each bin a count/frequency) and probably an additional parameter indicating the sample size (which indicates the bin resolution).
I realise playing around with the displaying of the quantiles on the 1D histograms that the uncertainties are not updated accordingly to the extremes of the percentiles values provided (stay as a default 0.16 and 0.84, in the code, line 266 of the corner.py file, Version 2.0.1).
A small hack to recover the quantiles values provided in the quantiles keywords do the trick (provided that the array as an even number).
Hi all,
Besides the last modification on line 213, I think we also have to define the argument of np.zeros function (on line 535) as int.
These are the messages I got:
/numpy/lib/function_base.py:564: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
n = np.zeros(bins, ntype)
/numpy/lib/function_base.py:600: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
n += np.bincount(indices, weights=tmp_w, minlength=bins).astype(ntype)
Thanks in advance.
Best,
Mariana Penna Lima
They should pivot around their end closest to the plot, not around their centers. Know what I mean? I don't have the foo.
OK @dfm, here's a wishlist for those of us using triangle.py to plot samples drawn from PDFs:
This might take you a while, sorry - let us know when you're done! Thanks Dan.
I see a recent fix for python3. Would it be possible to ship a 0.0.7 version on pypi ?
Hi- I noticed that triangle-plot conflicts with another triangle.py which seems to be necessary for the LinearNDInterpolator routine (scipy.interpolate.LinearNDInterpolator())
I'm not exactly sure what to do in this case... I need the LinearNDInterpolator to perform the calculations, but the corner plots would be really nice as well... Do you know of an easy way to alias the package name? (Sorry, my ignorance of the guts of python is showing!)
Add argument for setting contour levels
Hi there,
I'm relatively new to python and am giving your corner script a go! But I can seem to work out how to get the axes of the plots to display a log scale.
Besides logging the actual data (which doesn't do the right thing to the axes - but does give the right sort of shape to the plots), is there a way to do this?
With the recent bug highlighted in #68 the contour levels have been sorted to ensure they are in an increasing order so the call to contourf() does not fail.
I recently came across another related issue. When you are trying to plot a set of samples which have an extremely skewed (for instance, when the sampling is converging on a hard coded upper/lower limit in some model), it is possible for the contour levels to be equal and will subsequently cause contourf() to fail as there is a difference of 0 between two or more elements of the V list.
A simple fix I have used is to test for equal elements and disable the contour plotting if any are present after the V list in corner.hist2d has been sorted.
if len(V) != len(set(V)):
logging.warn("Equal contour levels found. Explicitly disabling.")
plot_contours = False
no_fill_contours = True
Of course the problem could be because of a bad model or constraint leading to the bad sampling, but that's another problem for another day!
When the plot gets huge, the labels get (relatively) small. This should either be fixed, or given an option=True to fix it at user's discretion.
I should have an upper=True
keyword so I can plot the upper corner instead of the lower corner.
I have been trying to use this library to plot my mcmc results. I have 26 free parameters that have been measured by emcee but when I tried to plot them just before the script return fig I got the core dump error message (I tested where the code would crash). How could I solve the problem. Wouldn't it be more convenient and lighter if you use
gs = gridspec.GridSpec(K, K)
instead of
fig, axes = pl.subplots(K, K, figsize=(dim, dim))
to make subplots?
Thanks in advance.
Hello,
I was wondering, is there was a way to plot two separate corner plots on top of each other? Perhaps in different colors?
Thanks!
Hi,
I'm running a script which invokes corner.corner, and everything goes fine in my local machine (python3).
I'm trying to run the same script in a cluster (python2.7) with no graphics device. To save the plots directly in a pdf file, I use the commands
import matplotlib
matplotlib.use('Agg')
at the beginning of my script. After including these lines, I find that corner.corner hangs.
Can someone tell me why, and how I can solve this? Does it depend on the version of python, or on the absence of a graphics device?
Thanks a lot
In ipython notebook 3 (I forget the exact version), calling triangle.corner
without capturing the return value creates two versions of the plot. I have an example of this at http://nbviewer.ipython.org/gist/DougBurke/a13dda72ca41d0a05e7d where the Out [7]
cell shows the plot twice. I show a workaround in cell 10.
Hello,
On a corner plot with many variables, the axis tick labels overlap with the axis labels. I was wondering how to control the tick label's font size, or chose not to display them.
Thank you!
@drphilmarshall suggested that a good way to demonstrate and document the features of this project would be to have an IPython notebook gallery. It's a great idea and if anyone feels like taking it on, that would be sweet! If not, I might do it some day :-)
I'm looking for a way to plot a range of truth values, sort of like a confidence interval. Ideally this would take the form of a semi-opaque wide line, just like the standard truths function, but the width of the line would represent the confidence width.
Is there any way to do that? I have been playing around with the axes object, maybe that's a way forward.
fig = corner.corner(data)
fig.axes[3].axhline(value, lw=2)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.