It just says, "check out GitHub," without mentioning the fact that you have to run ./generate_flatbuffers.py.

Hi all, thanks a lot for this repo!

What is the status of the stability for the binning types? We are interested to use them as the binning definitions in zfit. Or is there a subset that you would consider "stable", especially the simpler ones?

Is it possible to add boost-histogram to the tutorial on how to use Aghast? People are asking me how to convert ROOT histograms to boost-histograms and it would be nice to point them to the tutorial.

Write tests for corner cases and fix the ones that are broken.

I tried pip install ROOT and cannot find ROOT wheel. How can I install Python ROOT?

• "fast": ignore binning metadata and just add counts arrays naively
• "adjust": the current behavior and future default; moves bin positions or creates bins (with zero content) to make binnings compatible, then add
• "strict": raise an exception (ValueError) if binnings disagree (in a way that would make "fast" invalid).

(At the suggestion of @benkrikler.)

We should add the pre-commit check for this repo's Actions just like boost-histogram and hist do. It would be beneficial for a better coding style.

--Update--
solved by #35

Particularly for CategoryBinning axes, it would be nice to only save a dense binning for the tuple of categories corresponding to a valid bin, rather than the entire product of category values per axis.

A commonly requested feature for Boost.Histogram in C++ is to convert from and to ROOT histograms. In Python, we can do that already now with aghast, but not from C++. Calling into a Python library from C++ is both ghastly and awkward, so I wonder whether we could port Aghast to C++ to support this.

I also like the flatbuffer serialization very much, this could be the "native" format for Boost.Histogram (C++) and boost-histogram (Python) to talk to each other. Right now, the C++ code has only a generic serialization protocol in place that works with Boost.Serialization and compatible libraries like Cereal. The Python library uses this infrastructure to break a Boost.Histogram object into Python primitives, which are then serialized in the usual way. This produces a Python-specific format that is clunky to read from another language. Another commonly requested feature is to be able to dump a histogram in C++ to disk and read it again from Python. This is currently not supported, since the two libraries do not have a common serialization format. The flatbuffers could fill that role.

@jpivarski There might be a bug: when I run the code

root_hist = aghast.to_root(ghastly_hist, "root_hist")

A KeyError is thrown. It says no key "sumw2" is found. I search <"sumw2"> in this file and find this is the only one.

P.S., I am using pip install aghast, so the package I'm using might be a little out of date. But I check the latest code and the latest PR #24 , there are no changes with respect to to_root() in this file.

To the minimum byte-size type that covers both.

It says in the docs:

"The error_method does not specify how the histograms or functions were filled, but how the fraction should be interpreted statistically. It may be unspecified, leaving that interpretation unspecified. The normal method (sometimes called “Wald”) is a naive binomial interpretation, in which zero passing or zero failing values are taken to have zero uncertainty. The clopper_pearson method (sometimes called “exact”) is a common choice, though it fails in some statistical criteria. The computation and meaning of the methods are described in the references below."

If Aghast is only about exchanging data, is it necessary to specify the error_method? This is not part of the data, but part of the interpretation. You follow here the assumption that histogram counts should be modeled as binomial proportions, but in HEP that usually does not make sense. We use the Poisson model not only because it is easier, but because it is the better approximation of reality.

Example. You want to measure the differential cross-section of some process p+p -> X as a function of a kinematic variable, let's say eta. You count how often a particle X falls into an eta bin, call that \Delta N. The differential cross-section then is:

d\sigma / d\eta = 1/L \Delta N / \Delta \eta

where L is the luminosity. I use "=" here, although strictly speaking the equality requires the bin width to go to zero.

You care about the uncertainty of d\sigma / d\eta. You don't know the total sum of all \Delta N here, because you have not observed all X, only those in the acceptance of your detector, so you cannot compute the binomial proportional error. The measurement also doesn't depend on the fraction of events in each bin, but really on the number, which you divide by L and not sum over \Delta N. The Poisson is more correct here.

It is not exactly correct either, because the bins are not really independent for a physics reason: you observe K particles per event, and these K particles are not generated independently, they form jets and have some correlations. We usually neglect these correlations. They are not correctly represented by either Poisson or binomial proportion errors.

This would replace the _fromflatbuffers pure-Python infrastructure with an extension module that uses C++ Flatbuffers generated code, bound to Python with pybind11. Objects made in Python and the _toflatbuffers methods would not be affected, but properties providing data from a buffer would not cache in Python, but deliver from the fast C++ every time.

Having a C++ layer would also open the door to fast adding (in "fast," "strict," and maybe even "adjust" modes). Not all functionality would be ported to the C++ layer, only those that would stand the most from this optimization.

The base C++ classes should be separately defined (header-only?), so that they can be reused in other C++ projects.

I am having issues converting a TH1D into an aghast histogram.
Since I am working with rootpy histograms, the blow example uses 2D histograms and ProjectionX() to return the TH1D. The example includes the patch in #22 (otherwise it will stop there)

Code example

import aghast
import awkward
from rootpy.plotting import Hist2D
import numpy as np
import ROOT


def test_fill():
    pileup_bins = [0, 10, 15, 20, 30, 999]
    jet_pt_bins = [35, 90, 120]
      hist = Hist2D(pileup_bins, jet_pt_bins, name='test')

    ets = awkward.fromiter([
        np.random.poisson(30, 5),
        np.random.poisson(30, 2),
        np.random.poisson(30, 3),
    ])
    repeat = ets.stops - ets.starts

    weights = np.ones(len(ets))
    weights = np.repeat(weights, repeat, axis=0)
    pileup = np.random.poisson(50, len(ets))
    pileup = np.repeat(pileup, repeat, axis=0)
   
    hist.fill_array(np.vstack((pileup, ets.content)).T, weights)
    proX = hist.ProjectionX()
    print(type(proX)) # <class 'ROOT.TH1D'>
    assert isinstance(proX, (ROOT.TH1D, ROOT.TH2D, ROOT.TH3D))
    aghast.from_root(proX) # <-- fails with AssertionError: <class 'ROOT.TArrayD'>

Full trace

/software/miniconda/envs/hep_py3/lib/python3.6/site-packages/aghast/__init__.py:31: in from_root
    return aghast._connect._root.from_root(obj, collection=collection)
/software/miniconda/envs/hep_py3/lib/python3.6/site-packages/aghast/_connect/_root.py:286: in from_root
    sumw2array = getbincontents(sumw2obj)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

obj = <ROOT.TArrayD object at 0x55e5e196d5d0>

    def getbincontents(obj):
        if isinstance(obj, (ROOT.TH1C, ROOT.TH2C, ROOT.TH3C)):
            out = numpy.empty(obj.GetNcells(), dtype=numpy.int8)
            arraytype = "char"
        elif isinstance(obj, (ROOT.TH1S, ROOT.TH2S, ROOT.TH3S)):
            out = numpy.empty(obj.GetNcells(), dtype=numpy.int16)
            arraytype = "short"
        elif isinstance(obj, (ROOT.TH1I, ROOT.TH2I, ROOT.TH3I)):
            out = numpy.empty(obj.GetNcells(), dtype=numpy.int32)
            arraytype = "int"
        elif isinstance(obj, (ROOT.TH1F, ROOT.TH2F, ROOT.TH3F)):
            out = numpy.empty(obj.GetNcells(), dtype=numpy.float32)
            arraytype = "float"
        elif isinstance(obj, (ROOT.TH1D, ROOT.TH2D, ROOT.TH3D)):
            out = numpy.empty(obj.GetNcells(), dtype=numpy.float64)
            arraytype = "double"
        else:
>           raise AssertionError(type(obj))
E        

Unless I'm misunderstanding how the slicing should work, it seems like slicing on a categorical axis when there is also another axis, does not have the desired effect:

h = aghast.Histogram([aghast.Axis(aghast.CategoryBinning(["cat1", "cat2"])),
                      aghast.Axis(aghast.RegularBinning(10, aghast.RealInterval(-5, 5)))],
                    aghast.UnweightedCounts(
                        aghast.InterpretedInlineBuffer.fromarray(
                            numpy.array([[  9,  25,  29,  35,  54,  67,  60,  84,  80,  94],
                                         [ 99, 119, 109, 109,  95, 104, 102, 106, 112, 122]]))))

If I wanted to select only the counts for "cat2" I assumed this would work:

h_cat2 = h.loc['cat2']

However, that just produces a histogram with the counts for both categories:

h_cat2.counts.array
array([[ 99, 119, 109, 109,  95, 104, 102, 106, 112, 122],
       [  9,  25,  29,  35,  54,  67,  60,  84,  80,  94]])

... but not as prolific.

Does Aghast have deserialization support? I only notice that we can use tobuffer() to translate it into a byte array, but is there a frombuffer() method?

I've been working on adapting our pandas-based histogram filling code to produce aghasts, and was trying to understand IntegerBinning and CategoryBinning better.

In our current version that produces pandas dataframes, if a user asks to bin on a variable but doesn't provide a binning scheme, that's interpreted as implying the variable to already be discrete (ie. integral, categorical, or a finite number of floats-- the latter is a little dangerous, but so far not caught anyone out). This works nicely, in that we can distribute the filling jobs with such configurations, produce sparse binned dataframes, where bins are only present if at least one instance of its value occurred, and then merge these together.

Switching to aghast I can partially solve this by deducing all possible categories or the min / max value for of an integer variable for each chunk and only then create the Ghast, but then I can't be sure that these will be the same between distributed chunks of data.

I've had a look in the aghast code and the specification, and it looks to me like IntegerBinning isn't a sparse implementation. I've also looked at the merging of histograms (the _add method in particular) but struggled to understand the merging process for such a situation. What would happen ff two Histograms are combined, both with an IntegerBinning axis but with different min/max values? Will the result be the union of these axes or the intersection? And similarly for the set of values in two CategoricalBinning axes, would they be unioned or intersected if we add such histograms?

(I wasn't sure whether this should be a question on gitter, email, or an issue, but since it's potentially a feature request, I thought this might be the best place; please say if you prefer I use somewhere for such questions in the future! 😄 )

Could this follow standard Python layout? The following line should install the package:

pip install git+https://github.com/scikit-hep/aghast.git

But it doesn't since this does not follow standard layout. We have to use:

pip install git+https://github.com/scikit-hep/aghast#subdirectory=python

'test_root.py' only checks 1-d ROOT histogram classes. We should add [TH2*] and [TH3*] options. Besides, we can create some demo histograms to test, just like what boost_histogram has done.

I'll do it if time permits.

should be able to convert to/from pyhf representation of histograms.

@matthewfeickert @lukasheinrich @kratsg

Would be nice to be able to use aghast to convert to/from HEPData format

Considering that aghast is under heavy development, I think it would be nice to add something like CONTRIBUTING.md to involve others in.

Maybe split manual installation from README to CONTRIBUTING. I'll work on it and pull you a request after finishing. 😊