wipacrepo / hitspool Goto Github PK

[master 9e23eea] 1 changed, 1 inserted, 1 deleted

Hitspool realizes the loopback window using a fixed set of rolling files. The write point is determined by hit time in daq ticks as so:

final int fileNo =
(uint32) ((t / fileInterval) % maxNumberOfFiles);

Unfortunately this does not account for the year rollover when daq ticks reset to zero which cases the write point to jump to an arbitrary point in the loopback. In 2023 with fileInterval=15s and maxNumberOfFiles=72000 this resulted in loopback decreasing from 12.5 days to just 36 hours at the start of the year.

[master 734c754] 23 changed, 23 inserted, 23 deleted

[master bd556d7] 2 changed, 2 inserted, 2 deleted

[master fa7af60] 5 changed, 5 inserted, 5 deleted

If a command is run in hsadmin.py and produces output on stderr, the attempt to print will crash:

File "./hsadmin.py", line 130, in __run_command
    print("%s ERROR: %s" % (line, ), file=sys.stderr)
TypeError: not enough arguments for format string

[master 30b25eb] 9 changed, 9 inserted, 9 deleted

[master 0d4db1f] 1 changed, 1 inserted, 1 deleted

[master 0362cd7] 1 changed, 1 inserted, 1 deleted

[master f756360] 1 changed, 1 inserted, 1 deleted

[master 2886c77] 8 changed, 8 inserted, 8 deleted

[master c95439b] 4 changed, 4 inserted, 4 deleted

[master 61cf9ef] 2 changed, 2 inserted, 2 deleted

[master 2f13f22] 20 changed, 20 inserted, 20 deleted

The operator wiki needs to be updated for hsadmin.py instead of fab. The HINTS.md file here also should be updated with a git-based release procedure.

[master 50c6007] 1 changed, 1 inserted, 1 deleted

[master 3f21d78] 5 changed, 5 inserted, 5 deleted

[master 8039862] 2 changed, 2 inserted, 2 deleted

[master 6e0e3f1] 21 changed, 21 inserted, 21 deleted

[master 5544745] 1 changed, 1 inserted, 1 deleted

[master 5f6caa6] 4 changed, 4 inserted, 4 deleted

Currently all hitspool systems are targeted by an action in hsadmin.py. For re-kickstarting a machine such as a DOMHub, it's convenient to be able to specify just one target host.

[master 7bf2829] 3 changed, 3 inserted, 3 deleted

[master 0a77b49] 1 changed, 1 inserted, 1 deleted

[master 630ef8d] 7 changed, 7 inserted, 7 deleted

[master 4be8975] 2 changed, 2 inserted, 2 deleted

[master c95d129] 4 changed, 4 inserted, 4 deleted

[master 96c1d93] 26 changed, 26 inserted, 26 deleted

[master dd94d29] 11 changed, 11 inserted, 11 deleted

[master c361193] 8 changed, 8 inserted, 8 deleted

[master 70be531] 9 changed, 9 inserted, 9 deleted

[master e2a4117] 1 changed, 1 inserted, 1 deleted

[master d0aa120] 1 changed, 1 inserted, 1 deleted

[master 1d89387] 2 changed, 2 inserted, 2 deleted

[master 5eccdd1] 1 changed, 1 inserted, 1 deleted

DMICE has shown moments of sub 1hz data production. A hitspool request within this period resulted in an incomplete hitspool status rather than a success. The worker should not fail the request, perhaps issuing a nil data file when it knows that there is no data.

Details:

:: failed hitspool request ::

[
{
"_id": "63ce32e346b0af",
"service": "hitspool",
"varname": "hsrequest_info",
"value": {
"request_id": "63ce32e3000305",
"username": "unknown",
"prefix": "HESE",
"start_time": "2023-01-23 07:09:58.492989",
"stop_time": "2023-01-23 07:09:59.492989",
"destination_dir": "/mnt/data/hese_hs/137575_51122186_2023-01-23T07:09:58.992989",
"update_time": "2023-01-23 07:10:27.382328",
"status": "QUEUED"
},
"prio": 1,
"time": "2023-01-23 07:10:27.382328",
"insert_time": "2023-01-23 07:10:45.639269"
},
{
"_id": "63ce32e346b0b1",
"service": "hitspool",
"varname": "hsrequest_info",
"value": {
"request_id": "63ce32e3000305",
"username": "unknown",
"prefix": "HESE",
"start_time": "2023-01-23 07:09:58.492989",
"stop_time": "2023-01-23 07:09:59.492989",
"destination_dir": "/mnt/data/hese_hs/137575_51122186_2023-01-23T07:09:58.992989",
"update_time": "2023-01-23 07:10:27.390132",
"status": "IN PROGRESS"
},
"prio": 1,
"time": "2023-01-23 07:10:27.390132",
"insert_time": "2023-01-23 07:10:45.644914"
},
{
"_id": "63ce332646b0d1",
"service": "hitspool",
"varname": "hsrequest_info",
"value": {
"request_id": "63ce32e3000305",
"username": "unknown",
"prefix": "HESE",
"start_time": "2023-01-23 07:09:58.492989",
"stop_time": "2023-01-23 07:09:59.492989",
"destination_dir": "/mnt/data/hese_hs/137575_51122186_2023-01-23T07:09:58.992989",
"update_time": "2023-01-23 07:11:33.992774",
"status": "PARTIAL",
"success": "1-86,201-211",
"failed": "300"
},
"prio": 1,
"time": "2023-01-23 07:11:33.992774",
"insert_time": "2023-01-23 07:12:27.818381"
}
]

:: dmice hitspool worker ::

2023-01-23 07:10:27 INFO HsWorker queued request:
WorkerRequest(msgtype='REQUEST', version=2, request_id='63ce32e3000305', username='unknown', start_ticks=19265984929890000, stop_ticks=19265994929890000, copy_dir=None, destination_dir='/mnt/data/hese_hs/137575_51122186_2023-01-23T07:09:58.992989', prefix='HESE', extract=True, hubs=None, host='2ndbuild')
from Publisher
2023-01-23 07:10:27 INFO START = 19265984929890000 (2023-01-23 07:09:58.492989)
2023-01-23 07:10:27 INFO STOP = 19265994929890000 (2023-01-23 07:09:59.492989)
2023-01-23 07:10:27 INFO HsInterface running on: SPS
2023-01-23 07:10:27 ERROR No hits found for [2023-01-23 07:09:58.492989-2023-01-23 07:09:59.492989] in [('/mnt/data/pdaqlocal/hitspool', 'HitSpool-56439.dat')]
2023-01-23 07:10:27 INFO KEEP tmp dir with data in: /mnt/data/pdaqlocal/tmp/tmp72hu3gjp20230123_070958
2023-01-23 07:10:27 ERROR Request failed

:: From dmice HitSpool-56439.dat ::

0010: 00 44 72 4E CA 7F 90 18 timestamp : 1926598 1146697752 (2023-01-23 07:09:58.1146697752)
0010: 00 44 72 4E CA 7F 90 6C timestamp : 1926598 1146697836 (2023-01-23 07:09:58.1146697836)
0010: 00 44 72 4F 0C 07 0C C1 timestamp : 1926598 2246096065 (2023-01-23 07:09:58.2246096065)
0010: 00 44 72 4F 0C 07 0D 33 timestamp : 1926598 2246096179 (2023-01-23 07:09:58.2246096179)
0010: 00 44 72 4F 0C 08 15 D9 timestamp : 1926598 2246163929 (2023-01-23 07:09:58.2246163929)
0010: 00 44 72 4F 2C C4 15 95 timestamp : 1926598 2795355541 (2023-01-23 07:09:58.2795355541)
0010: 00 44 72 4F 3F 4D 99 8F timestamp : 1926598 3106357647 (2023-01-23 07:09:58.3106357647)
0010: 00 44 72 4F 7D 5F 5E B7 timestamp : 1926598 4147709623 (2023-01-23 07:09:58.4147709623)
0010: 00 44 72 4F A5 58 FD F5 timestamp : 1926598 4818380277 (2023-01-23 07:09:58.4818380277)
0010: 00 44 72 4F A5 58 FE 97 timestamp : 1926598 4818380439 (2023-01-23 07:09:58.4818380439)
1926598 4929890000
1926599 4929890000
0010: 00 44 72 52 0C 1F D4 CF timestamp : 1926599 5132622031 (2023-01-23 07:09:59.5132622031)
0010: 00 44 72 52 F3 44 AC 15 timestamp : 1926599 9010573333 (2023-01-23 07:09:59.9010573333)
0010: 00 44 72 53 27 4E 19 50 timestamp : 1926599 9883606352 (2023-01-23 07:09:59.9883606352)

[master 3efe781] 3 changed, 3 inserted, 3 deleted

[master 3f1a7d1] 1 changed, 1 inserted, 1 deleted