Look at the qastled generated for this query:

def apply_event_cuts (source: ObjectStream) -> ObjectStream:
    '''Event level cuts for the analysis. Keep from sending data that we aren't going to need at all in the end.
    '''
    return (source
        .Where(lambda e: e.trigE or e.trigM))

def good_leptons(source: ObjectStream) -> ObjectStream:
    '''Select out all good leptons from each event. Return their pt, eta, phi, and E, and other
    things needed downstream.

    Because uproot doesn't tie toegher the objects, we can't do any cuts at this point.
    '''
    return source.Select(lambda e:
        {
            'lep_pt': e.lep_pt,
            'lep_eta': e.lep_eta,
            'lep_phi': e.lep_phi,
            'lep_energy': e.lep_E,
            'lep_charge': e.lep_charge,
            'lep_ptcone30': e.lep_ptcone30,
            'lep_etcone20': e.lep_etcone20,
            'lep_typeid': e.lep_type,
            'lep_trackd0pvunbiased': e.lep_trackd0pvunbiased,
            'lep_tracksigd0pvunbiased': e.lep_tracksigd0pvunbiased,
            'lep_z0': e.lep_z0,
            'mcWeight': e.mcWeight,
            # 'scaleFactor': e.scaleFactor_ELE*e.scaleFactor_MUON*e.scaleFactor_LepTRIGGER*e.scaleFactor_PILEUP,
        }) \
        .AsParquetFiles('junk.parquet')

ds = ServiceXSourceUpROOT('cernopendata://dummy',  "mimi", backend='open_uproot')
ds.return_qastle = True
leptons = good_leptons(apply_event_cuts(ds))

See this notebook

qastle:

(Select (Where (call EventDataset 'mimi') (lambda (list e) (or (attr e 'trigE') (attr e 'trigM')))) (lambda (list e) (dict (list 'lep_pt' 'lep_eta' 'lep_phi' 'lep_energy' 'lep_charge' 'lep_ptcone30' 'lep_etcone20' 'lep_typeid' 'lep_trackd0pvunbiased' 'lep_tracksigd0pvunbiased' 'lep_z0') (list (attr e 'lep_pt') (attr e 'lep_eta') (attr e 'lep_phi') (attr e 'lep_E') (attr e 'lep_charge') (attr e 'lep_ptcone30') (attr e 'lep_etcone20') (attr e 'lep_type') (attr e 'lep_trackd0pvunbiased') (attr e 'lep_tracksigd0pvunbiased') (attr e 'lep_z0')))))

Note that the EventDataSet has only a single argument - and it is the tree name - it should be the file plus tree name. What happened?

Hi @gordonwatts, could you have a look the following error? Import the package just after installation doesn't work.

...
Successfully installed func-adl-servicex-1.1.2 idna-2.10 qastle-0.11.0 servicex-2.4
(base) KE-MBP:servicex-access-examples kchoi$ python
Python 3.7.4 (default, Aug 13 2019, 15:17:50)
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import func_adl_servicex
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/kchoi/anaconda3/lib/python3.7/site-packages/func_adl_servicex/__init__.py", line 3, in <module>
    from .ServiceX import ServiceXSourceUpROOT, ServiceXSourceXAOD, ServiceXSourceCMSRun1AOD, FuncADLServerException  # NOQA
  File "/Users/kchoi/anaconda3/lib/python3.7/site-packages/func_adl_servicex/ServiceX.py", line 10, in <module>
    from servicex import ServiceXDataset
  File "/Users/kchoi/anaconda3/lib/python3.7/site-packages/servicex/__init__.py", line 1, in <module>
    from .servicex import ServiceXDataset, StreamInfoUrl, StreamInfoPath  # NOQA
  File "/Users/kchoi/anaconda3/lib/python3.7/site-packages/servicex/servicex.py", line 15, in <module>
    from servicex.servicex_config import ServiceXConfigAdaptor
  File "/Users/kchoi/anaconda3/lib/python3.7/site-packages/servicex/servicex_config.py", line 7, in <module>
    from .utils import ServiceXException
  File "/Users/kchoi/anaconda3/lib/python3.7/site-packages/servicex/utils.py", line 467, in <module>
    from backoff._common import (  # noqa: E402
ImportError: cannot import name '_prepare_logger' from 'backoff._common' (/Users/kchoi/anaconda3/lib/python3.7/site-packages/backoff/_common.py)
>>>

As a user/developer i would like to easily see the qastle that is sent for various func_adl queries to the backend.

Approach

Decide on one of two approaches:

1. Add a bool parameter that will have ServiceXDatasetSourcexAOD and ServiceXDatasetSourceUpROOT to return the qastle rather than the data
2. Create a new data source that returns the qastle

Create doctest to show off some typical queries to both the xAOD and uproot backend.

The ASRootTree return for the uproot backend currently throws NotImplementedError.

Once issues #3, #4, #5 are done, make sure that our code coverage numbers are reasonable.

The AsPandas return for the uproot backend currently throws NotImplementedError.

The current version has AsParquet, etc. These are things that might cause problems now - so make sure everything in the documentation is working properly.

Awkward has weird symantics. If we invent a column name as part of the query, then we should not force the user to know about that column name. The same does not hold for just about everything else we return.

If you do this query first:

from servicex import ignore_cache
with ignore_cache():
    electrons_async = [
        (calib_tools.query_update(ds, electron_working_point=wp)
            .SelectMany(lambda e: [ele.eta() for ele in e.Electrons()])
            .value_async())
        for wp in ['TightLHElectron', 'MediumLHElectron', 'LooseLHElectron']
    ]
    electrons = await asyncio.gather(*electrons_async)

you will get pack paths to a local file, as expected. If you then do:

from servicex import ignore_cache
with ignore_cache():
    electrons_async = [
        (calib_tools.query_update(ds, electron_working_point=wp)
            .SelectMany(lambda e: [ele.eta() for ele in e.Electrons()])
            .AsAwkwardArray()
            .value_async())
        for wp in ['TightLHElectron', 'MediumLHElectron', 'LooseLHElectron']
    ]
    electrons = await asyncio.gather(*electrons_async)

you'll also get back windows path - which is not what you would expect. The cache see the windows path and doesn't see the AsAwkwardArray.

The AsAwkwardArray return for the uproot backend currently throws NotImplementedError.

To use func_adl on servicex, the only func_adl package you only need to install this package.

I think I undertand what this means, but...

The uproot transformer doesn't understand the RequestParquet method:

• Do not add it in when we are emitting parquet files, particularly when we are trying to steer which way files go internally
• Do let through the user trying to do this - they will get get a failure message from the ServcieX code generator complaining.

During the last AGC challenge, the idea of a dataset group came up.

• “ServiceXDataSetGroup” like structure perhaps that wraps multiple datasets

We should explore this, taking into account prior work done by @kyungeonchoi on his project

You'd like to specify http://xxx.yy.net/filename as a dataset, rather than having to put it in a list, [http://xxx.yy.net/filename]

We needed these for Awkward, but it should be there for everything:

• with a list of column names
• With a single name
• With nothing (and dict in the last select statement).

As an example:

def test_sx_xaod_awkward_no_columns(async_mock):
    'Test a request for awkward arrays from an xAOD backend'
    sx = async_mock(spec=ServiceXDataset)
    sx.first_supported_datatype.return_value = 'root'
    ds = ServiceXSourceXAOD(sx)
    q = ds.Select("lambda e: e.MET").AsAwkwardArray()

    q.value()

Some standard queries for Uproot do not work:

import pandas as pd
from func_adl_servicex import ServiceXSourceUpROOT

dataset_name = "data15_13TeV:data15_13TeV.00282784.physics_Main.deriv.DAOD_PHYSLITE.r9264_p3083_p4165_tid21568807_00"
src = ServiceXSourceUpROOT(dataset_name, "CollectionTree")
data = src.Select("lambda e: {'JetPT': e['AnalysisJetsAuxDyn.pt']}") \
    .AsParquetFiles('junk.parquet') \
    .value()
df = pd.read_parquet(data[0])
print(df)

The reason is the Uproot backend can't deal with AsParquetFiles. This should be swallowed. And one should sort out what
to do with this when column names are added in.

Make sure this works well for 3.8

Doesn't seem to be working the way it should in the README. Understand what is going on.

Change to a SX dataset that can run locally. In this case, we do this in a way that makes it possible to support everything else nice about SX (caching, streaming, etc.).

Change default image location to be gitlab-registry.cern.ch/atlas/athena/analysisbase

It would be nice to see a more carefully thought out and straight forward way to ask for data to come back from ServiceX. In short, normalize the access patterns for servicex. The current interface has grown organically, and there are now so many operations and it is hard to surface them from one place to the other. Time to take a step back, perhaps.

What we have now

        sx = ServiceXDataset([uproot_single_file],
                             backend_name=endpoint_uproot,
                             status_callback_factory=None)
        src = ServiceXSourceUpROOT(sx, 'mini')
        r = (src.Select(lambda e: {'lep_pt': e['lep_pt']})
                .AsAwkwardArray()
                .value())

And AsAwkwardArray can be replaced by a bunch of different things:

• AsPandasDF, as_pandas - a panda dataframe (this does not support nested objects!)
• AsROOTTTree, as_ROOT_tree - a list of file(s) that contains a root TTree object
• AsParquetFiles, as_parquet - a list of parquet file(s) built from awkward's to_parquet method
• AsAwkwardArray, as_awkward - returns an awkward array of all the data

These methods do not return the actual data - just the request to generate the data. The value() call at the end actually triggers the infrastructure to generate the data. There is another version of the method called value_async() that does the same thing, but allows you to easily queue up many requests at once.

There are at least two axes here:

• What data format should come back from the ServiceX query
• Should programming interface be sync or asynchronous?

There is yet another axis for the root and parquet queries - do you want the files downloaded locally into a cache or just a uri to access them over the web? This is only accessible via direct calls to the servicex library (e.g. see get_root_files_async, get_root_files_stream, and get_data_rootfiles_uri_stream and get_data_rootfiles_uri_async).

What do users of func_adl want?

Let's look at each one and reason about why different choices are made.

• Data Format
  • What the user is familiar with ("what is a parquet file?")
  • How the data will be consumed downstream of servicex. Input to awkward distributed processing?
• URI's to files (for root and parquet) or local copied files?
  • At an analysis facility probably want the local files downloaded. On a local laptop developing downstream code, want the local file option.
  • Downstream code will be quite opinionated about what it can and can't use.
• Async or Synch access
  • One or many datasets? One probably wants to parallelize access to many datasets.
  • In a notebook almost certainly want synch access to make a "demo" work well.
• Streaming URI's
  • Only interesting if downstream access can process a stream of URI's.
  • Uses well understood but not widely known streaming async infrastructure in python.

Starting from Scratch

Some differences observed during this work that should be unified (this needs to be turned into some issues in the xaod and uproot backends):

• Uproot can deal with linq type ast's (like (Select xxx) and xAOD needs (call Select xxx. Further, the xAOD backend crashes in a rather subtle way if it isn't given the right version. For xAOD background you can't call qastle.insert_linq_nodes first.
• the uproot backend can deal with a dictionary as a final input to the ntupler, taking column names from the dictionaries names. The xAOD cannot.
• Neither backend can deal with dictionaries in the middle of the code (that get made and then destroyed)
• The xAOD backend needs an AsROOTTtree as a final call, the uproot backend does not. Should decide if different output types should be produced by the transformers and if not, then both should look like the uproot backend.
• Parquet files are made by the uproot backend and not root files, and vice versa. Should this be normalized somewhere in servicex's infrastructure?
• Should we be able to tell what endpoint is serving what type of backend by querying it?

This needs to be combined with any input from the testing that is done in the servicex backend tests repo.

This assert prevents tuples as opposed to lists. In the end, it will all be reduced to the same thing - so we should be able to allow it. See the assert statement.

I am trying to do a query passing through all columns. This works fine with func_adl_uproot:

from func_adl_uproot import UprootDataset
ds = UprootDataset("nanoaod15.root", "Events")
ds.value()

It does not work with func_adl_servicex:

import servicex
from func_adl_servicex import ServiceXSourceUpROOT

dataset_name = ["https://xrootd-local.unl.edu:1094//store/user/AGC/nanoAOD/nanoaod15.root"]
sx_dataset = servicex.ServiceXDataset(dataset_name, "uproot")
ds = ServiceXSourceUpROOT(sx_dataset, "Events")
ds.value()

This crashes with

Transform be73e06d-0c92-46e5-8b43-13ba009717d4 had 1 errors:
  Error transforming file: h
  -> error.

(see ssl-hep/ServiceX#408). The query succeeds when using ds.Select(lambda e: e.Jet_pt).value() instead.

The file being used is equivalent in both cases. I am using the following versions of libraries:

func-adl             3.0
func-adl-servicex    2.0
func-adl-uproot      1.9.0
servicex             2.5.4

https://github.com/iris-hep/func_adl_servicex/blob/a08e27ec339e78e390a030c6bb426ce04fd0b39f/func_adl_servicex/ServiceX.py#L73-L73

A list of locally copied root files should be a possible return here