Fileset closing is not waiting for the data in the run_stream_fileset to be complete about t0 HOT 6 CLOSED

Behavior is a bug, but I don't think the problem is necessarily in Tier0 code, more in how Tier0 code and WMCore code interacts. @amaltaro , what triggers input fileset for a task closing? The Tier0 case is special in that we inject files over an extended period of time and we need to control when the fileset is being closed.

Alternatively, if this isn't possible, I can also rework the Tier0 logic to completely ignore the closed flag for the fileset and just inject new data anyways and trigger closeout some other way.

from t0.

Dirk, this is the DAO querying for filesets that can be closed:
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/WMBS/MySQL/Fileset/ListClosable.py

trying to make sense of this query in plain english:

• workflow entry needs to be injected or have that alt_fs_close=1 (L22)
• have 0 running jobs, i.e., no jobs in a state != cleanout (L35)
• have 0 files acquired - wmbs_sub_files_acquired
• have 0 files available - wmbs_sub_files_available
• and, of course, be open :)

Did I miss anything?
BTW, since the T0 agent doesn't run WorkQueue/WMBSHelper code, the only place responsible for setting a fileset to open or not is in the JobArchiverPoller component.

from t0.

The original design was that the Tier0 controlled when a workflow was set to injected, we only did it when all the data from P5 was transferred and the bookkeeping consistent,

Guess I have to check that code. Could still be operational problems, i.e. P5 bookkeeping and transfers were changed after the fact.

from t0.

Those files were transferred some time later than the rest of streamer files from those two runs.
Therefore, it could still be some operational problems at P5.

from t0.

Yes, but late transfer in itself wouldn't have the Tier0 mark the workflow as injected. As far as I remember, it does check for bookkeeping consistency as well, i.e. it should know if there is still data to be transferred.

from t0.

As we haven's seen this issue in years, I'll close this issue

from t0.