snews2 / hop-snalert-app Goto Github PK

proposed formats:

AlertObservation:
detector_name: str that must be in a list of detector names
sent_time: str in the format of "YYYY-MM-DD hh:mm:ss ns" (precision to the ns not needed?)
event_time: str in the format of "YYYY-MM-DD hh:mm:ss ns"
detector_status: str that must be in a list of possible status (to be determined with the experiments)
FAR? FAR-tier?

should the FAR-tier be included in the content of the message, 
or should two models (with different ids) be made for SNEWS-LOW and SNEWS


SignificanceObservation:
content of AlertObservation
p-value: float


TimingObservation:
content of SignificanceObservation
time-series: two possibilities, reconstruction of single events or distribution

two models to include those two possibilities?

reconstruction of single events: 
{channel1: {id1: {time: time1, energy: energy1}, id2: {time: time2, energy2}}, channel2: …}

distribution of binned events:
{list of times, list of number of events}

# source local-config.env; echo $ALERT_TOPIC; source run-containers.sh; hop subscribe -s EARLIEST --no-auth $ALERT_TOPIC
kafka://localhost:9092/snews-alert
766a41f9a039f6fdbc6e0ffd903d0b09b04d6befb3d4ae928c84cd1e9623801d
d24e63e3ebb0983f412688dd4fa7f05d4881af2cba34dd799bf74b215957fddb
%6|1612548695.721|FAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 5ms in state APIVERSION_QUERY)
internal kafka error: KafkaError{code=_TRANSPORT,val=-195,str="localhost:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 5ms in state APIVERSION_QUERY)"}
Traceback (most recent call last):
  File "/Users//snews/snews-venv/bin/hop", line 8, in <module>
    sys.exit(main())
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/hop/__main__.py", line 54, in main
    args.func(args)
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/hop/publish.py", line 42, in _main
    s.write(loader.load(message))
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/hop/io.py", line 343, in __exit__
    return self._producer.__exit__(*exc)
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/adc/producer.py", line 59, in __exit__
    n_unsent = self.close()
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/adc/producer.py", line 49, in close
    return self.flush()
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/adc/producer.py", line 40, in flush
    n = self._producer.flush(timeout.total_seconds())
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/adc/errors.py", line 22, in log_client_errors
    raise(KafkaException.from_kafka_error(kafka_error))
adc.errors.KafkaException: Error communicating with Kafka: code=_TRANSPORT localhost:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 5ms in state APIVERSION_QUERY)
%4|1612548695.749|TERMINATE|rdkafka#producer-1| [thrd:app]: Producer terminating with 1 message (108 bytes) still in queue or transit: use flush() to wait for outstanding message delivery
%6|1612548696.185|FAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 3ms in state APIVERSION_QUERY)
internal kafka error: KafkaError{code=_TRANSPORT,val=-195,str="localhost:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 3ms in state APIVERSION_QUERY)"}
Traceback (most recent call last):
  File "/Users//snews/snews-venv/bin/hop", line 8, in <module>
    sys.exit(main())
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/hop/__main__.py", line 54, in main
    args.func(args)
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/hop/publish.py", line 42, in _main
    s.write(loader.load(message))
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/hop/io.py", line 343, in __exit__
    return self._producer.__exit__(*exc)
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/adc/producer.py", line 59, in __exit__
    n_unsent = self.close()
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/adc/producer.py", line 49, in close
    return self.flush()
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/adc/producer.py", line 40, in flush
    n = self._producer.flush(timeout.total_seconds())
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/adc/errors.py", line 22, in log_client_errors
    raise(KafkaException.from_kafka_error(kafka_error))
adc.errors.KafkaException: Error communicating with Kafka: code=_TRANSPORT localhost:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 3ms in state APIVERSION_QUERY)
%4|1612548696.195|TERMINATE|rdkafka#producer-1| [thrd:app]: Producer terminating with 1 message (102 bytes) still in queue or transit: use flush() to wait for outstanding message delivery
Traceback (most recent call last):
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/adc/consumer.py", line 36, in subscribe
    topic_meta = self.describe_topic(topic, timeout)
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/adc/consumer.py", line 61, in describe_topic
    return cluster_meta.topics[topic]
KeyError: 'snews-alert'During handling of the above exception, another exception occurred:Traceback (most recent call last):
  File "/Users//snews/snews-venv/bin/hop", line 8, in <module>
    sys.exit(main())
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/hop/__main__.py", line 54, in main
    args.func(args)
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/hop/subscribe.py", line 55, in _main
    with stream.open(args.url, "r") as s:
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/hop/io.py", line 103, in open
    return _Consumer(
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/hop/io.py", line 246, in __init__
    self._consumer.subscribe(t)
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/adc/consumer.py", line 38, in subscribe
    raise ValueError(f"topic {topic} does not exist on the broker, so can't subscribe")
ValueError: topic snews-alert does not exist on the broker, so can't subscribe