snews2 / hop-snalert-app Goto Github PK
View Code? Open in Web Editor NEWClient application for the SuperNova Early Warning System 2.0 network
License: BSD 3-Clause "New" or "Revised" License
Client application for the SuperNova Early Warning System 2.0 network
License: BSD 3-Clause "New" or "Revised" License
As a user of the snews
app, I would like to use snews latency
to filter the latency messages by other fields, e.g. detector name, in addition to message type (alert
and observation
).
See related PR for snews latency
#15.
It appears we have one notebook in the project to test connections and the coincidence logic of the package. We should have some automated unit and integration tests in addition to the notebook.
We discussed this at today’s check in meeting and I wanted to submit it here so we don’t lose track:
The checklist items on every new PR (example) should be automated as much as possible—both to save the core developers some time but also to give contributors immediate feedback and allow them to fix any remaining issues while they’re still in the headspace for doing it.
I can see that .github/workflows/build.yml
exists and already runs make lint
and make test
on every push; so a good first step would be to change that to run on: [push, pull_request]
. Maybe it would be possible to add other checklist items there, too?
Here are the dataformats I suggested during one of the Virtual Offices:
proposed formats:
AlertObservation:
detector_name: str that must be in a list of detector names
sent_time: str in the format of "YYYY-MM-DD hh:mm:ss ns" (precision to the ns not needed?)
event_time: str in the format of "YYYY-MM-DD hh:mm:ss ns"
detector_status: str that must be in a list of possible status (to be determined with the experiments)
FAR? FAR-tier?
should the FAR-tier be included in the content of the message,
or should two models (with different ids) be made for SNEWS-LOW and SNEWS
SignificanceObservation:
content of AlertObservation
p-value: float
TimingObservation:
content of SignificanceObservation
time-series: two possibilities, reconstruction of single events or distribution
two models to include those two possibilities?
reconstruction of single events:
{channel1: {id1: {time: time1, energy: energy1}, id2: {time: time2, energy2}}, channel2: …}
distribution of binned events:
{list of times, list of number of events}
There are also some other questions I will put as other issues, and reference them here.
Just make sure all classes are documented. We can use sphinx-style docstrings or the simpler numpy style (which I personally prefer).
To transmit time series that can be a big amount of data, json might not be the best format.
Another proposition could be FITS, as it is compressed, can contain metadata in the header and is widely used in astronomy.
Related to #24
PS: Sorry for all the edits I got mixed between the issues I wanted to open
Users have reported errors when running the SCiMMA Server container during local development:
# source local-config.env; echo $ALERT_TOPIC; source run-containers.sh; hop subscribe -s EARLIEST --no-auth $ALERT_TOPIC
kafka://localhost:9092/snews-alert
766a41f9a039f6fdbc6e0ffd903d0b09b04d6befb3d4ae928c84cd1e9623801d
d24e63e3ebb0983f412688dd4fa7f05d4881af2cba34dd799bf74b215957fddb
%6|1612548695.721|FAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 5ms in state APIVERSION_QUERY)
internal kafka error: KafkaError{code=_TRANSPORT,val=-195,str="localhost:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 5ms in state APIVERSION_QUERY)"}
Traceback (most recent call last):
File "/Users//snews/snews-venv/bin/hop", line 8, in <module>
sys.exit(main())
File "/Users//snews/snews-venv/lib/python3.8/site-packages/hop/__main__.py", line 54, in main
args.func(args)
File "/Users//snews/snews-venv/lib/python3.8/site-packages/hop/publish.py", line 42, in _main
s.write(loader.load(message))
File "/Users//snews/snews-venv/lib/python3.8/site-packages/hop/io.py", line 343, in __exit__
return self._producer.__exit__(*exc)
File "/Users//snews/snews-venv/lib/python3.8/site-packages/adc/producer.py", line 59, in __exit__
n_unsent = self.close()
File "/Users//snews/snews-venv/lib/python3.8/site-packages/adc/producer.py", line 49, in close
return self.flush()
File "/Users//snews/snews-venv/lib/python3.8/site-packages/adc/producer.py", line 40, in flush
n = self._producer.flush(timeout.total_seconds())
File "/Users//snews/snews-venv/lib/python3.8/site-packages/adc/errors.py", line 22, in log_client_errors
raise(KafkaException.from_kafka_error(kafka_error))
adc.errors.KafkaException: Error communicating with Kafka: code=_TRANSPORT localhost:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 5ms in state APIVERSION_QUERY)
%4|1612548695.749|TERMINATE|rdkafka#producer-1| [thrd:app]: Producer terminating with 1 message (108 bytes) still in queue or transit: use flush() to wait for outstanding message delivery
%6|1612548696.185|FAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 3ms in state APIVERSION_QUERY)
internal kafka error: KafkaError{code=_TRANSPORT,val=-195,str="localhost:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 3ms in state APIVERSION_QUERY)"}
Traceback (most recent call last):
File "/Users//snews/snews-venv/bin/hop", line 8, in <module>
sys.exit(main())
File "/Users//snews/snews-venv/lib/python3.8/site-packages/hop/__main__.py", line 54, in main
args.func(args)
File "/Users//snews/snews-venv/lib/python3.8/site-packages/hop/publish.py", line 42, in _main
s.write(loader.load(message))
File "/Users//snews/snews-venv/lib/python3.8/site-packages/hop/io.py", line 343, in __exit__
return self._producer.__exit__(*exc)
File "/Users//snews/snews-venv/lib/python3.8/site-packages/adc/producer.py", line 59, in __exit__
n_unsent = self.close()
File "/Users//snews/snews-venv/lib/python3.8/site-packages/adc/producer.py", line 49, in close
return self.flush()
File "/Users//snews/snews-venv/lib/python3.8/site-packages/adc/producer.py", line 40, in flush
n = self._producer.flush(timeout.total_seconds())
File "/Users//snews/snews-venv/lib/python3.8/site-packages/adc/errors.py", line 22, in log_client_errors
raise(KafkaException.from_kafka_error(kafka_error))
adc.errors.KafkaException: Error communicating with Kafka: code=_TRANSPORT localhost:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 3ms in state APIVERSION_QUERY)
%4|1612548696.195|TERMINATE|rdkafka#producer-1| [thrd:app]: Producer terminating with 1 message (102 bytes) still in queue or transit: use flush() to wait for outstanding message delivery
Traceback (most recent call last):
File "/Users//snews/snews-venv/lib/python3.8/site-packages/adc/consumer.py", line 36, in subscribe
topic_meta = self.describe_topic(topic, timeout)
File "/Users//snews/snews-venv/lib/python3.8/site-packages/adc/consumer.py", line 61, in describe_topic
return cluster_meta.topics[topic]
KeyError: 'snews-alert'During handling of the above exception, another exception occurred:Traceback (most recent call last):
File "/Users//snews/snews-venv/bin/hop", line 8, in <module>
sys.exit(main())
File "/Users//snews/snews-venv/lib/python3.8/site-packages/hop/__main__.py", line 54, in main
args.func(args)
File "/Users//snews/snews-venv/lib/python3.8/site-packages/hop/subscribe.py", line 55, in _main
with stream.open(args.url, "r") as s:
File "/Users//snews/snews-venv/lib/python3.8/site-packages/hop/io.py", line 103, in open
return _Consumer(
File "/Users//snews/snews-venv/lib/python3.8/site-packages/hop/io.py", line 246, in __init__
self._consumer.subscribe(t)
File "/Users//snews/snews-venv/lib/python3.8/site-packages/adc/consumer.py", line 38, in subscribe
raise ValueError(f"topic {topic} does not exist on the broker, so can't subscribe")
ValueError: topic snews-alert does not exist on the broker, so can't subscribe
This is due to the SCiMMA Server container not fully loading before the start-up messages are published in run-containers.sh
, causing ValueError: topic snews-alert does not exist on the broker, so can't subscribe
issues when trying to access the topic.
Subsequent runs of run-containers.sh
usually do not have this issue, since the container usually is fully loaded by then and can hence receive messages and create topics.
A workaround would be to increase the sleep
time in run-containers.sh
so that the container loads before receiving the start-up messages.
A more robust solution would be to modify scimma-server-container
to hold incoming message requests until the container has loaded.
An alternative could be to modify hop-client
to create non-existing topics when subscribing, but this would be a substantial design change with potentially undesired effects.
@myNameIsPatrick suggested to use jsonschma
, but I wonder if it might be a bit too powerful and complex for what we want to do (verification that a string belongs to a list, verification that the string is a date…)
See #24
There have been a few snews app crashes over the last two days. See the most recent stack trace for details.
...
2021-05-25 09:55:18,852 | model : INFO : starting decider
2021-05-25 09:55:18,853 | model : INFO : processing messages from kafka://kafka.scimma.org/snews.experiments-test
2021-05-25 09:56:30,862 | model : DEBUG : read message from partition 7
2021-05-25 09:56:30,863 | model : DEBUG : processing SNEWSObservation
2021-05-25 09:56:30,863 | model : INFO : shutting down
2021-05-25 09:56:30,866 | model : DEBUG : shutting down producer
2021-05-25 09:56:30,867 | model : DEBUG : flushed all messages
Traceback (most recent call last):
File "/depot/darkmatter/apps/snews/sandbox/snews-venv/bin/snews", line 8, in
sys.exit(main())
File "/depot/darkmatter/apps/snews/sandbox/snews-venv/lib/python3.7/site-packages/snews/main.py", line 53, in main
args.func(args)
File "/depot/darkmatter/apps/snews/sandbox/snews-venv/lib/python3.7/site-packages/snews/model.py", line 169, in main
model.run()
File "/depot/darkmatter/apps/snews/sandbox/snews-venv/lib/python3.7/site-packages/snews/model.py", line 115, in run
self.processMessage(msg)
File "/depot/darkmatter/apps/snews/sandbox/snews-venv/lib/python3.7/site-packages/snews/model.py", line 134, in processMessage
self.mapping[message_type](message)
File "/depot/darkmatter/apps/snews/sandbox/snews-venv/lib/python3.7/site-packages/snews/model.py", line 137, in processObservationMessage
self.addObservationMsg(message)
File "/depot/darkmatter/apps/snews/sandbox/snews-venv/lib/python3.7/site-packages/snews/model.py", line 128, in addObservationMsg
self.myDecider.addMessage(message)
File "/depot/darkmatter/apps/snews/sandbox/snews-venv/lib/python3.7/site-packages/snews/decider.py", line 54, in addMessage
self.db.insert(message.sent_time, message.neutrino_time, message.asdict())
File "/depot/darkmatter/apps/snews/sandbox/snews-venv/lib/python3.7/site-packages/snews/db_storage.py", line 42, in insert
time2 = datetime.datetime.strptime(sent_time, self.datetime_format)
TypeError: strptime() argument 1 must be str, not int
No passwords
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.