xenonnt / saltax Goto Github PK
View Code? Open in Web Editor NEWSalting MC into straxen
License: MIT License
Salting MC into straxen
License: MIT License
Now it has this problem very frequently, for unclear reason:
Loading match_acceptance_extended: | | 1.01 % [00:03<05:00], #7 (0.17 s). 5.9 MB/s
Traceback (most recent call last):
File "/home/ghusheng/saltax/jobs/job.py", line 176, in <module>
st.make(strrunid, 'match_acceptance_extended', progress_bar=True)
File "/home/ghusheng/.local/lib/python3.9/site-packages/strax/context.py", line 1576, in make
for _ in self.get_iter(run_ids[0], targets, save=save, max_workers=max_workers, **kwargs):
File "/home/ghusheng/.local/lib/python3.9/site-packages/strax/context.py", line 1472, in get_iter
generator.throw(e)
File "/home/ghusheng/.local/lib/python3.9/site-packages/strax/context.py", line 1439, in get_iter
for n_chunks, result in enumerate(strax.continuity_check(generator), 1):
File "/home/ghusheng/.local/lib/python3.9/site-packages/strax/chunk.py", line 298, in continuity_check
for chunk in chunk_iter:
File "/home/ghusheng/.local/lib/python3.9/site-packages/strax/processor.py", line 312, in iter
raise exc.with_traceback(traceback)
File "/home/ghusheng/.local/lib/python3.9/site-packages/strax/mailbox.py", line 274, in _send_from
x = next(iterable)
File "/home/ghusheng/.local/lib/python3.9/site-packages/strax/plugins/plugin.py", line 505, in iter
yield from self._iter_compute(chunk_i=chunk_i, **inputs_merged)
File "/home/ghusheng/.local/lib/python3.9/site-packages/strax/plugins/plugin.py", line 532, in _iter_compute
yield self.do_compute(chunk_i=chunk_i, **inputs_merged)
File "/home/ghusheng/.local/lib/python3.9/site-packages/strax/plugins/plugin.py", line 619, in do_compute
result = self.compute(**kwargs)
File "/home/ghusheng/pema/pema/match_plugins.py", line 106, in compute
sel_peaks = peaks[get_idx(sel_from_peaks, peaks['id'], INT_NAN)]
File "/home/ghusheng/pema/pema/match_plugins.py", line 336, in get_idx
raise ValueError()
ValueError
Python script failed with exit code 1
Note that, we will no longer need pema
features once we move to fuse
, as in #37. And also currently the pema
features are not used anyway.
Describe the feature
Unless rate, generator etc changed. We can just add seeds into wfsim instructions?
Expected behavior
Make an imaginary example of how the feature would work.
YOUR (PSEUDO) CODE GOES HERE
Proposed implementation
Any rough idea how would you implement this feature?
Potential risk
Will there be any risk after adding this feature?
All italic comments can be removed from this template.
Describe the feature
Want to have dependa-bot as we have for pema
and reprox
. The point is that in contexts.py
we are hard-copying what we have in cutax
for arguments. It is better if we can keep track of things more carefully, even though it is not automatic. A PR by it can be treated as an alarm for human.
Expected behavior
Proposed implementation
Potential risk
Hopefully happen after #37 is resolved.
Describe the bug
Negative length container in records building shows up. Reasons uncleared.
To Reproduce
Try salting 023747
.
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Traceback (most recent call last):
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/processing/general.py", line 394, in touching_windows
_check_objects_non_negative_length(objects)
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/processing/general.py", line 452, in _check_objects_non_negative_length
assert mask
AssertionError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/yuanlq/software/saltax/jobs/job.py", line 39, in <module>
st.make(strrunid, 'event_basics')
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/context.py", line 1426, in make
for _ in self.get_iter(run_ids[0], targets,
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/context.py", line 1336, in get_iter
generator.throw(e)
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/context.py", line 1307, in get_iter
for n_chunks, result in enumerate(strax.continuity_check(generator), 1):
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/chunk.py", line 304, in continuity_check
for chunk in chunk_iter:
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/processor.py", line 302, in iter
raise exc.with_traceback(traceback)
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/mailbox.py", line 281, in _send_from
x = next(iterable)
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/plugins/plugin.py", line 499, in iter
yield self.do_compute(chunk_i=chunk_i, **inputs_merged)
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/plugins/plugin.py", line 608, in do_compute
result = self.compute(**kwargs)
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/straxen/plugins/peaks/peaks.py", line 44, in compute
peaks = strax.replace_merged(peaklets[~is_s1], merged_s2s)
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/processing/peak_merging.py", line 110, in replace_merged
skip_windows = strax.touching_windows(orig, merge)
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/processing/general.py", line 396, in touching_windows
raise ValueError(f'{names} should have non-negative length!')
ValueError: containers should have non-negative length!
Versions
Please add the output of:
straxen.print_versions()
module version path git
python 3.9.18 /opt/XENONnT/anaconda/envs/XENONnT_development/bin/python None
strax 1.5.4 /opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax None
straxen 2.1.5 /opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/straxen None
cutax 1.15.5 /dali/lgrandi/xenonnt/software/cutax/v1.15.5/cutax None
All italic comments can be removed from this template.
Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Two failures in SR0 Rn220 runs.
st_salt = saltax.contexts.sxenonnt(runid=21952,
saltax_mode='salt',
output_folder='/scratch/midway2/yuanlq/salt/sr0_rn220',
faxconf_version="sr0_v4",
generator_name='flat',
recoil=7,
mode='all')
st_salt.make('021952', 'raw_records_simu')
...
ValueError: Attempt to create chunk [021952.raw_records_simu: 1623158479sec 407739940 ns - 1623158485sec 245093400 ns, 148275 items, 6.2 MB/s] whose data ends late at 1623158485245312770
st_salt = saltax.contexts.sxenonnt(runid=21967,
saltax_mode='salt',
output_folder='/scratch/midway2/yuanlq/salt/sr0_rn220',
faxconf_version="sr0_v4",
generator_name='flat',
recoil=7,
mode='all')
st_salt.make('021967', 'raw_records_simu')
...
ValueError: Attempt to create chunk [021967.raw_records_simu: 1623165122sec 491949950 ns - 1623165128sec 329327640 ns, 154089 items, 6.4 MB/s] whose data ends late at 1623165128330200090
st_salt = saltax.contexts.sxenonnt(runid=23747,
saltax_mode='salt',
output_folder='/scratch/midway2/yuanlq/salt/sr0_rn220',
faxconf_version="sr0_v4",
generator_name='flat',
recoil=7,
mode='all')
st_salt.make('021967', 'raw_records_simu')
...
ValueError: Attempt to create chunk [023747.raw_records_simu: 1624423295sec 764783640 ns - 1624423301sec 133492760 ns, 143198 items, 6.5 MB/s] whose data ends late at 1624423301133672210
st_salt = saltax.contexts.sxenonnt(runid=22175,
saltax_mode='salt',
output_folder='/scratch/midway2/yuanlq/salt/sr0_rn220',
faxconf_version="sr0_v4",
generator_name='flat',
recoil=7,
mode='all')
st_salt.make('022175', 'raw_records_simu')
...
ValueError: Attempt to create chunk [022175.raw_records_simu: 1623283044sec 798690840 ns - 1623283050sec 167399960 ns, 133904 items, 6.1 MB/s] whose data ends late at 1623283050167815950
However, there is successful example in SR1 BKG run:
st = saltax.contexts.sxenonnt(output_folder='/scratch/midway2/yuanlq/salt/tests',
runid=40671, saltax_mode='salt')
st.make('040671', 'raw_records_simu')
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Versions
Please add the output of:
straxen.print_versions()
module version path git
0 python 3.9.18 /project/lgrandi/xenonnt/env_tests/development... None
1 strax 1.5.4 /project/lgrandi/xenonnt/env_tests/development... None
2 straxen 2.1.5 /project/lgrandi/xenonnt/env_tests/development... None
3 cutax 1.15.5 /dali/lgrandi/xenonnt/software/cutax/v1.15.5/c... None
All italic comments can be removed from this template.
Describe the bug
More details here. Basically it means all saltax simulated S2s have to have such problem
Describe the bug
ValueError: SPulseProcessing got inconsistent time ranges of inputs: {'raw_records': (1624759990000000000, 1624759995368708120), 'raw_records_simu': (1624759991725899872, 1624759995368708120)}
To Reproduce
Insert the MWE of how to reproduce the error
st = saltax.contexts.sxenonnt(output_folder='/home/yuanlq/scratch-midway2/salt/tests',
runid=24109, saltax_mode='salt')
st.storage.append(strax.DataDirectory('/project2/lgrandi/tutorial_data', readonly=True))
records = st.get_array('024109', 'records')
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Versions
Please add the output of:
straxen.print_versions()
All italic comments can be removed from this template.
Describe the feature
The plugins are mostly modified based on the ones from straxen
. If straxen
is updated for these, there is danger that the results are not consistent anymore. To avoid this, we can try to replace where it is meant to be replaced only.
Expected behavior
Make an imaginary example of how the feature would work.
YOUR (PSEUDO) CODE GOES HERE
Proposed implementation
Any rough idea how would you implement this feature?
Potential risk
Will there be any risk after adding this feature?
All italic comments can be removed from this template.
Describe the bug
Seems the peak timing of S2s have been shifted about O(1ms).
To Reproduce
Insert the MWE of how to reproduce the error
st_simu = saltax.contexts.sxenonnt(runid=33048,
saltax_mode='simu',
output_folder='/project/lgrandi/yuanlq/salt',
faxconf_version="sr0_v4",
generator_name='flat',
recoil=7,
mode='all')
events_simu = st_simu.get_array('033048', 'event_info')
truth = st_simu.get_array('033048', 'truth')
match = st_simu.get_array('033048', "match_acceptance_extended")
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Versions
Please add the output of:
straxen.print_versions()
module version path git
0 python 3.9.18 /project/lgrandi/xenonnt/env_tests/development... None
1 strax 1.5.4 /project/lgrandi/xenonnt/env_tests/development... None
2 straxen 2.1.5 /project/lgrandi/xenonnt/env_tests/development... None
3 cutax 1.15.5 /dali/lgrandi/xenonnt/software/cutax/v1.15.5/c... None
All italic comments can be removed from this template.
Describe the bug
Some match_acceptance_extended
computation is impossible.
Segmentation fault (core dumped)
To Reproduce
Insert the MWE of how to reproduce the error
st_simu = saltax.contexts.sxenonnt(runid=21952,
saltax_mode='simu',
output_folder='/project/lgrandi/yuanlq/salt',
faxconf_version="sr0_v4",
generator_name='flat',
recoil=8,
mode='all')
st_simu.make('022685', "match_acceptance_extended")
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Versions
Please add the output of:
straxen.print_versions()
All italic comments can be removed from this template.
Describe the bug
Otherwise the test workflow keep failing.
To Reproduce
Insert the MWE of how to reproduce the error
YOUR CODE GOES HERE
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Versions
Please add the output of:
straxen.print_versions()
All italic comments can be removed from this template.
Describe the bug
ValueError: Hit outside of full containment! This seem to show up in merged_s2s
.
To Reproduce
Salt 022175
.
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Traceback (most recent call last):
File "/home/yuanlq/software/saltax/jobs/job.py", line 38, in <module>
st.make(strrunid, 'merged_s2s')
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/context.py", line 1426, in make
for _ in self.get_iter(run_ids[0], targets,
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/context.py", line 1336, in get_iter
generator.throw(e)
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/context.py", line 1307, in get_iter
for n_chunks, result in enumerate(strax.continuity_check(generator), 1):
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/chunk.py", line 304, in continuity_check
for chunk in chunk_iter:
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/processor.py", line 302, in iter
raise exc.with_traceback(traceback)
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/mailbox.py", line 281, in _send_from
x = next(iterable)
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/plugins/overlap_window_plugin.py", line 32, in iter
yield from super().iter(iters, executor=executor)
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/plugins/plugin.py", line 499, in iter
yield self.do_compute(chunk_i=chunk_i, **inputs_merged)
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/plugins/overlap_window_plugin.py", line 49, in do_compute
result = super().do_compute(chunk_i=chunk_i, **kwargs)
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/plugins/plugin.py", line 608, in do_compute
result = self.compute(**kwargs)
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/straxen/plugins/merged_s2s/merged_s2s.py", line 132, in compute
strax.add_lone_hits(merged_s2s, lh, self.to_pe, n_top_channels=n_top_pmts_if_digitize_top)
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/processing/peak_merging.py", line 170, in add_lone_hits
_add_lone_hits(peaks, lone_hits, to_pe, n_top_channels=0)
File "/opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax/processing/peak_merging.py", line 189, in _add_lone_hits
raise ValueError('Hit outside of full containment!')
ValueError: Hit outside of full containment!
Python script failed with exit code 1
Versions
Please add the output of:
straxen.print_versions()
Host midway3-0064.rcc.local
module version path git
python 3.9.18 /opt/XENONnT/anaconda/envs/XENONnT_development/bin/python None
strax 1.5.4 /opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/strax None
straxen 2.1.5 /opt/XENONnT/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/straxen None
cutax 1.15.5 /dali/lgrandi/xenonnt/software/cutax/v1.15.5/cutax None
All italic comments can be removed from this template.
Can we run saltax in outsource?
Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Insert the MWE of how to reproduce the error
YOUR CODE GOES HERE
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Versions
Please add the output of:
straxen.print_versions()
All italic comments can be removed from this template.
Shows up in 100Hz AmBe sprinkling
ValueError: Attempt to create chunk [051913.raw_records_simu: 1681823850sec 329999748 ns - 1681823856sec 167399960 ns, 717542 items, 30.0 MB/s] whose data starts early at 1681823850329999570
1681823850329999570alueError: Attempt to create chunk [051913.raw_records_simu: 1681823850sec 329999748 ns - 1681823856sec 167399960 ns, 717542 items, 30.0 MB/s] whose data starts early at 1681823850329999570
So that it is possible to study other efficiency loss and bias in general calibration without true timing known
Describe the feature
Run auto test after every commit.
Expected behavior
As is said above.
Proposed implementation
Something like the one in cutax?
Potential risk
Will there be any risk after adding this feature?
All italic comments can be removed from this template.
Describe the feature
Currently you have to specify a single runid
in context sxenonnt
, which is quite bad for strax logic. In principle, this information can be got from data making process.
Expected behavior
Make an imaginary example of how the feature would work.
YOUR (PSEUDO) CODE GOES HERE
Proposed implementation
Any rough idea how would you implement this feature?
Potential risk
Will there be any risk after adding this feature?
All italic comments can be removed from this template.
Currently the translator is in microphysics_summary
plugin, which will be triggered every time in each chunk. This is stupid and may take extra 5s to compute for each chunk. We want to fix it at some point.
Describe the feature
Want to generate instruction file withe the following input
Expected behavior
Make an imaginary example of how the feature would work.
file = saltax.generate_instructions()
Proposed implementation
Any rough idea how would you implement this feature?
Potential risk
Will there be any risk after adding this feature?
All italic comments can be removed from this template.
Basically the same as utils.show_eff1d
, but now want show_eff2d
Describe the feature
A clear and concise description of what the feature is.
Something like black should work.
Expected behavior
Make an imaginary example of how the feature would work.
Hopefully auto PR?
Proposed implementation
Any rough idea how would you implement this feature?
Potential risk
Will there be any risk after adding this feature?
All italic comments can be removed from this template.
Describe the bug
The current implementation of peak_saturation_correction will break if we input salts with saturation. The saturation correction will be unphysical because it will correct using only real TPC channels without saturation. However, the salt's contribution in these channels used to correct should also be considered.
To Reproduce
Insert the MWE of how to reproduce the error
Salt big enough S2s should show the effect. Note that this will NOT trigger a bug anywhere. It is just physically wrong.
Expected behavior
A clear and concise description of what you expected to happen.
The correction reference channels should also consider salt's contribution
Screenshots
If applicable, add screenshots to help explain your problem.
Versions
Please add the output of:
straxen.print_versions()
All italic comments can be removed from this template.
Currently it is too messy and redundant. We want to wrap up functions and also put to-process datatypes into config
Describe the feature
Want a waveform viewer, which is hopefully bokeh
powered.
Expected behavior
Make an imaginary example of how the feature would work.
saltax.display(event_salted)
This return 3 rows of plots, with synchronized dynamic time axis:
straxen.event_display
Proposed implementation
Any rough idea how would you implement this feature?
Potential risk
Will there be any risk after adding this feature?
All italic comments can be removed from this template.
Describe the feature
Currently the instruction is pre-selected before chunking happens. However, it may be dangerous how we are doing it now. By the currently implementation, it could happen that the end of a pulse simulated goes beyond the chunk end time. We need additional safeguard. Maybe 1 FDT?
Expected behavior
Make an imaginary example of how the feature would work.
YOUR (PSEUDO) CODE GOES HERE
Proposed implementation
instructions = instructions[(instructions['time'] >= start) &
(instructions['time'] < end - SAFEGUARD)]
Potential risk
Will a chunk be ever shorter than one FDT?
All italic comments can be removed from this template.
Describe the feature
Make configs in plugins so that people can get the following in same context with different config passed when st.get_array
:
Expected behavior
data_only = st.get_array('066666', 'event_info', config=dict(saltax_mode='data-only',))
simu_only = st.get_array('066666', 'event_info', config=dict(saltax_mode='simu-only',))
salted = st.get_array('066666', 'event_info', config=dict(saltax_mode='salted',))
Proposed implementation
Make saltax_mode
as a config in saltax.Records
. Data structure like this
Potential risk
data-only
, we are literally just recomputing the already processed data, but hashes are completely different starting from records
. Maybe it is too wasteful.Describe the feature
WFSim is getting abandoned. Need to switch to fuse for better maintenance.
Expected behavior
Make an imaginary example of how the feature would work.
YOUR (PSEUDO) CODE GOES HERE
Proposed implementation
Any rough idea how would you implement this feature?
Potential risk
Will there be any risk after adding this feature?
All italic comments can be removed from this template.
Describe the feature
Have pema embedded in saltax
, and extend the labels based on waveform watching results.
Expected behavior
Make an imaginary example of how the feature would work.
YOUR (PSEUDO) CODE GOES HERE
Proposed implementation
Any rough idea how would you implement this feature?
Potential risk
Will there be any risk after adding this feature?
All italic comments can be removed from this template.
So we can embrace the good convention in fuse
Describe the feature
The saltax
context will register plugins except for records
.
Expected behavior
Make an imaginary example of how the feature would work.
st._plugin_class_registry
>>> {'aqmon_hits': straxen.plugins.aqmon_hits.aqmon_hits.AqmonHits,
...
'records': saltax.plugins.s_records.PulseProcessing,
...
'peak_classification_bayes': straxen.plugins.peaks.peak_classification_bayes.BayesPeakClassification,
Proposed implementation
Any rough idea how would you implement this feature?
Not yet...
Potential risk
Will there be any risk after adding this feature?
Not yet...
For stuff like config and unchanged static functions, we want to just inherit from straxen
for better version control consistent with straxen
.
Describe the feature
We want seriously implement auto test pipelines for this package, so coverall
helps us find where the tests are missing.
Expected behavior
Proposed implementation
Potential risk
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.