Reprocessing on midway cluster
| Package | CI |
|---|---|
 |
|
 |
 |
 |
|
 |
Please visit the documentation for installation instructions and examples.
Can be found either on github or the online documentation.
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
![sourcery-ai[bot] avatar](https://avatars.githubusercontent.com/in/48477?v=4 "sourcery-ai[bot]")
Should be easy to implement. Just pass to select_runs
reprox-find-data --context xenonnt_online --config-kwargs '{"minimum_run_number":51379,"maximum_run_number":52358,"_rucio_path":"/project/lgrandi/rucio","output_folder":"/project2/lgrandi/xenonnt/processed"}' --targets cut_daq_vet
This will lead to TypeError: get_context() got an unexpected keyword argument '_rucio_path'. To make it work, you need to do this instead:
reprox-find-data --context xenonnt_online --context-kwargs '{"minimum_run_number":51379,"maximum_run_number":52358,"_rucio_path":"/project/lgrandi/rucio","output_folder":"/project2/lgrandi/xenonnt/processed"}' --targets cut_daq_vet
However, this will fail:
reprox-start-jobs --context xenonnt_online --context-kwargs '{"minimum_run_number":51379,"maximum_run_number":52358,"_rucio_path":"/project/lgrandi/rucio","output_folder":"/project2/lgrandi/xenonnt/processed"}' --targets cut_daq_veto
To make it work, you need to do
reprox-start-jobs --context xenonnt_online --config-kwargs '{"minimum_run_number":51379,"maximum_run_number":52358,"_rucio_path":"/project/lgrandi/rucio","output_folder":"/project2/lgrandi/xenonnt/processed"}' --targets cut_daq_veto
>>> from reprox import find_data, submit_jobs, validate_run
RuntimeError: module compiled against API version 0x10 but this version of numpy is 0xf . Check the section C-API incompatibility at the Troubleshooting ImportError section at https://numpy.org/devdocs/user/troubleshooting-importerror.html#c-api-incompatibility for indications on how to solve this problem .
Using /home/yuanlq/software/reprox/reprox/reprocessing.ini-config. Overwrite by setting "REPROX_CONFIG" as an environment variable
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: cannot import name 'find_data' from 'reprox' (/home/yuanlq/software/reprox/reprox/__init__.py)Should be able to specify min and max so we don't waste time scanning the whole DB.
Currently it is impossible to change strax directories. Should be flexible with it.
The following runs seem to be problematic in cuts computation. Example error looks like below. [45501, 44405, 46721, 47083, 47287, 47296, 47298, 46731, 45437, 45451, 045462, 045501]
ValueError: Cannot merge chunks with different number of items: [[046731.peak_positions_cnn: 1659907969sec 424508440 ns - 1659907990sec 899344920 ns, 22238 items, 0.0 MB/s], [046731.peak_positions_mlp: 1659907969sec 424508440 ns - 1659907990sec 899344920 ns, 22234 items, 0.0 MB/s], [046731.peak_positions_gcn: 1659907969sec 424508440 ns - 1659907990sec 899344920 ns, 22238 items, 0.0 MB/s]]
More details you can find logs at /project/lgrandi/xenonnt/data_management_reprocessing/job_logs. Already used --notlazy in straxer, and I am requiring single core at this point.
Maybe we shouldn't retry doing jobs that failed earlier
Will do after
straxenIf this is the second time you run reprox in a singularity container in midway3: this will fail.
(XENONnT_development) Singularity xenonnt-development.simg:~> ~/software/reprox/bin/reprox-start-jobs --context xenonnt_online --config-kwargs '{"_rucio_local_path":"/project/lgrandi/rucio","include_rucio_local":"true","output_folder":"/project2/lgrandi/xenonnt/processed"}' --targets cuts_basic --force-non-admin
Traceback (most recent call last):
File "/home/yuanlq/software/reprox/bin/reprox-start-jobs", line 2, in <module>
from reprox import core, submit_jobs
File "/home/yuanlq/software/reprox/reprox/__init__.py", line 3, in <module>
from . import core
File "/home/yuanlq/software/reprox/reprox/core.py", line 12, in <module>
from strax import to_str_tuple
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.8/site-packages/strax/__init__.py", line 13, in <module>
from .storage.files import *
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.8/site-packages/strax/storage/files.py", line 5, in <module>
from bson import json_util
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.8/site-packages/bson/json_util.py", line 130, in <module>
from pymongo.errors import ConfigurationError
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.8/site-packages/pymongo/__init__.py", line 117, in <module>
from pymongo.mongo_client import MongoClient
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.8/site-packages/pymongo/mongo_client.py", line 76, in <module>
from pymongo.topology import Topology, _ErrorContext
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.8/site-packages/pymongo/topology.py", line 45, in <module>
from pymongo.server import Server
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 839, in exec_module
File "<frozen importlib._bootstrap_external>", line 971, in get_code
File "<frozen importlib._bootstrap_external>", line 640, in _compile_bytecode
EOFError: marshal data too short
So that the reprocessing is automatically run... This doesn't have to be in this repo.
Exclude it from finding and submission.
So that we don't scan over unnecessary run range, but instead we will just check the runlets submitted in OSG.
/project/lgrandi/xenonnt/data_management_reprocessing/job_logs/049655.txt as an example here:
strax.mailbox.MailboxKilled: (<class 'TypeError'>, TypeError("can't unbox array from PyObject into native value. The object maybe of a different type"), <traceback object at 0x7fef0a689640>)
strax.mailbox.MailboxKilled: (<class 'TypeError'>, TypeError("can't unbox array from PyObject into native value. The object maybe of a different type"), <traceback object at 0x7fef0a689640>)
strax.mailbox.MailboxKilled: (<class 'TypeError'>, TypeError("can't unbox array from PyObject into native value. The object maybe of a different type"), <traceback object at 0x7fef0a689640>)
raise MailboxKilled(self.killed_because)
strax.mailbox.MailboxKilled: (<class 'TypeError'>, TypeError("can't unbox array from PyObject into native value. The object maybe of a different type"), <traceback object at 0x7fef0a689640>)
self.close()
strax.mailbox.MailboxKilled: (<class 'TypeError'>, TypeError("can't unbox array from PyObject into native value. The object maybe of a different type"), <traceback object at 0x7fef0a689640>)
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/mailbox.py", line 361, in close
x = next(iterable)
self.send(StopIteration)
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/plugins/plugin.py", line 456, in iter
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/mailbox.py", line 313, in send
raise MailboxKilled(self.killed_because)
strax.mailbox.MailboxKilled: (<class 'TypeError'>, TypeError("can't unbox array from PyObject into native value. The object maybe of a different type"), <traceback object at 0x7fef0a689640>)
self.input_buffer[d].split(
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/chunk.py", line 170, in split
data1, data2, t = split_array(
TypeError: can't unbox array from PyObject into native value. The object maybe of a different type
You specified _auto_append_rucio_local=True and you are not on dali compute nodes,so we will add the following rucio local path: /project/lgrandi/rucio/
Traceback (most recent call last):
File "/opt/XENONnT/anaconda/envs/XENONnT_development/bin/straxer", line 352, in <module>
sys.exit(main(args))
File "/opt/XENONnT/anaconda/envs/XENONnT_development/bin/straxer", line 299, in main
run_make(target)
File "/opt/XENONnT/anaconda/envs/XENONnT_development/bin/straxer", line 268, in run_make
"050111.txt" 400L, 26916C 366,5 94%
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/mailbox.py", line 281, in _send_from
raise MailboxKilled(self.killed_because)
raise MailboxKilled(self.killed_because)
self.close()
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/mailbox.py", line 361, in close
x = next(iterable)
self.send(StopIteration)
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/plugins/plugin.py", line 456, in iter
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/mailbox.py", line 313, in send
raise MailboxKilled(self.killed_because)
strax.mailbox.MailboxKilled: (<class 'TypeError'>, TypeError("can't unbox array from PyObject into native value. The object maybe of a different type"), <traceback object at 0x7fef0a689640>)
self.input_buffer[d].split(
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/chunk.py", line 170, in split
data1, data2, t = split_array(
TypeError: can't unbox array from PyObject into native value. The object maybe of a different type
You specified _auto_append_rucio_local=True and you are not on dali compute nodes,so we will add the following rucio local path: /project/lgrandi/rucio/
Traceback (most recent call last):
File "/opt/XENONnT/anaconda/envs/XENONnT_development/bin/straxer", line 352, in <module>
sys.exit(main(args))
File "/opt/XENONnT/anaconda/envs/XENONnT_development/bin/straxer", line 299, in main
run_make(target)
File "/opt/XENONnT/anaconda/envs/XENONnT_development/bin/straxer", line 268, in run_make
for i, d in enumerate(get_results()):
File "/opt/XENONnT/anaconda/envs/XENONnT_development/bin/straxer", line 265, in get_results
yield from st.get_iter(**kwargs)
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/context.py", line 1324, in get_iter
generator.throw(e)
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/context.py", line 1296, in get_iter
for n_chunks, result in enumerate(strax.continuity_check(generator), 1):
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/chunk.py", line 303, in continuity_check
for chunk in chunk_iter:
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/processor.py", line 302, in iter
raise exc.with_traceback(traceback)
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/mailbox.py", line 281, in _send_from
x = next(iterable)
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/plugins/plugin.py", line 456, in iter
self.input_buffer[d].split(
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/chunk.py", line 170, in split
data1, data2, t = split_array(
TypeError: can't unbox array from PyObject into native value. The object maybe of a different type
For example, if only peaklets available, we should not process this run with reprox
Need to measure: how much ram is needed for each data type and source.
After new environment tag released, where straxen context has the correct storage directories by default.
Singularity xenonnt-development.simg:~> reprox-find-data --context xenonnt_online --config-kwargs {"minimum_run_number":51381} --target cuts_basic --cmt-version False
RuntimeError: module compiled against API version 0x10 but this version of numpy is 0xf . Check the section C-API incompatibility at the Troubleshooting ImportError section at https://numpy.org/devdocs/user/troubleshooting-importerror.html#c-api-incompatibility for indications on how to solve this problem .
Using /home/yuanlq/software/reprox/reprox/reprocessing.ini-config. Overwrite by setting "REPROX_CONFIG" as an environment variable
usage: reprox-find-data [-h] [--package {straxen,cutax}] [--context CONTEXT] [--context-kwargs CONTEXT_KWARGS] [--config-kwargs CONTEXT_CONFIG_KWARGS]
[--targets [TARGETS [TARGETS ...]]] [--ignore_runs [IGNORE_RUNS [IGNORE_RUNS ...]]] [--force-non-admin]
[--detectors [DETECTORS [DETECTORS ...]]] [--cmt-version CMT_VERSION]
reprox-find-data: error: argument --config-kwargs/--config_kwargs: invalid loads value: '{minimum_run_number:51381}'
Singularity xenonnt-development.simg:~> reprox-find-data --context xenonnt_online --config-kwargs {"s1_min_coincidence":2} --target cuts_basic --cmt-version False
RuntimeError: module compiled against API version 0x10 but this version of numpy is 0xf . Check the section C-API incompatibility at the Troubleshooting ImportError section at https://numpy.org/devdocs/user/troubleshooting-importerror.html#c-api-incompatibility for indications on how to solve this problem .
Using /home/yuanlq/software/reprox/reprox/reprocessing.ini-config. Overwrite by setting "REPROX_CONFIG" as an environment variable
usage: reprox-find-data [-h] [--package {straxen,cutax}] [--context CONTEXT] [--context-kwargs CONTEXT_KWARGS] [--config-kwargs CONTEXT_CONFIG_KWARGS]
[--targets [TARGETS [TARGETS ...]]] [--ignore_runs [IGNORE_RUNS [IGNORE_RUNS ...]]] [--force-non-admin]
[--detectors [DETECTORS [DETECTORS ...]]] [--cmt-version CMT_VERSION]
reprox-find-data: error: argument --config-kwargs/--config_kwargs: invalid loads value: '{s1_min_coincidence:2}'
We should be able to run in a midway3-only mode, since DaLI is ramping down.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.