Our goal is to implement the k4LCIOreader as a standalone application to read and convert LCIO files. We have run into a Segmentation Fault in edm4hep::ReconstructedParticleCollectionData::setReference, when trying this on miniDST files. Below is a list of commands to reproduce the error.

First you clone the github repository with the minimal reproducer and afterwards source the setup file from the nightlies.hsf to get the environment set up.

git clone --branch lcioedmconv https://gitlab.desy.de/ftx-sft-key4hep/edmconverter.git

source /cvmfs/sw-nightlies.hsf.org/spackages5/key4hep-stack/master-2022-08-03/x86_64-centos7-gcc11.2.0-opt/gmz2k/setup.sh

After that is done you build the reproducer.

cd edmconverter

mkdir build

cd build

cmake ..

make

download the miniDST file

cd ..

wget https://desycloud.desy.de/index.php/s/5LmrjGWqziQfMe7/download/rv01-16-p10_250.sv01-14-01-p00.mILD_o1_v05.E250-TDR_ws.I106479.Pe2e2h.eL.pR-00001-ILDminiDST.slcio

The programm is run with gdb to get the seg fault. Without gdb the programm just freezes in our case.

cd build

gdb -ex run --args ./minimalrep "../rv01-16-p10_250.sv01-14-01-p00.mILD_o1_v05.E250-TDR_ws.I106479.Pe2e2h.eL.pR-00001-ILDminiDST.slcio"

We are unsure whether this issue arises because the miniDST format has some Collections dropped. Or also because it has subset Collections. The Collection used in this case is not a subset Collection, but in a test case using a subset Collection the subset flag did not propagate through the program to the point where the references where set and the same seg fault came up.

The change in the IReader interface (https://github.com/AIDASoft/podio/pull/238/files#diff-59c18d39992e239d6dd3790be8d8140ec9d1a8984db415ddcd1ad3599716e996R49) needs to be propagated to fix a build error:

https://gitlab.cern.ch/key4hep/k4-deploy/-/jobs/19375843#L4017

I created a test in k4MarlinWrapper to test conversion of collections from EDM4hep to LCIO to EDM4hep.
When converting Tracks, these can have other Tracks linked to it. I test that these linked Tracks remain in the conversion between EDM4hep to LCIO.
But when converting from LCIO to EDM4hep these linked Tracks are missing in the resulting EDM4hep collection. This is probably true for other collections that have linked collections.

The conversion is done using k4LCIOConverter like so:

podio::CollectionIDTable* id_table = new podio::CollectionIDTable();
k4LCIOConverter* lcio_converter = new k4LCIOConverter(id_table);
lcio_converter->set(the_event);

T* mycoll = dynamic_cast<T*>(lcio_converter->getCollection(lcio_name));

The LCIO covariance matrix is lower triangle, while the one in EDM4hep is the upper triangle.

http://lcio.desy.de/v02-09/doc/doxygen_api/html/classEVENT_1_1TrackState.html#a667b8774ce1b0f42dbae11a59dc7e09b

https://github.com/key4hep/EDM4hep/blob/e17d47ab9f17f3a56fb90550d307275cacd93153/edm4hep.yaml#L63

Now the converters in both directions can be found in https://github.com/key4hep/k4EDM4hep2LcioConv and should be maintained there

When using edm4hep input the MCParticles collection already exists in edm4hep. Therefore, in the conversion from lcio to edm4hep getCollection() is not called for this collection. When now converting the RecoMCTruthLink collection, the lcio MCParticles collection gets converted temporarily to an edm4hep collection. The entries from this collection are then used as sim particles in the edm4hep association. In the end, when writing the edm4hep output file either PodioOutput crashes complaining about

[ctest] PodioOutput         FATAL  Standard std::exception is caught 
[ctest] PodioOutput         ERROR Trying to persistify untracked object
[ctest] EventLoopMgr        FATAL .executeEvent(): Standard std::exception thrown by PodioOutput
[ctest] EventLoopMgr        ERROR Trying to persistify untracked object

or silently writes out garbage collectionID and index values for this association. (I have not understood yet which configuration changes between those two)

This does not happen for all particles as there is still another "bug" hiding this. As getCorresponding simply returns the particle from the first collection found in the (alphabetically ordered) m_type2cols map, this problem does not occur for charged particles making it into the EfficientMCParticles or InefficientMCParticles collections created by the CLIC reconstruction. The latter can probably be fixed by limiting m_type2cols to non-subset collections.

I will not have time to try to build a fix for this in the next 2-3 weeks, so if anyone wants to have a go at this...

@zoujh as a member of the key4hep project you should be able to create new repositories there. So this should mean you could move this repository there already.

K4FWCore already offers some I/O functionality that could be useful here. Unless you think a standalone program has some advantages, I'd propose to rewrite the Reader/Converter as an Algorithm. I'll prepare a PR.