oda-hub / dispatcher-plugin-integral Goto Github PK
View Code? Open in Web Editor NEWLicense: Other
License: Other
WARNING:[ InstrumentQueryBackEnd : instrument-not-set ]:after clear_temp_dir 2.7239949703216553 s
INFO:cdci_data_analysis.flask_app.dispatcher_query:constructed <class 'cdci_data_analysis.flask_app.dispatcher_query.InstrumentQueryBackEnd'>:[ InstrumentQueryBackEnd : isgri ] for data_server_call_back=False
ERROR:cdci_data_analysis.flask_app.app:Exception on /api/meta-data [GET]
Traceback (most recent call last):
File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/flask_restx/api.py", line 672, in error_router
return original_handler(e)
File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/flask_app/app.py", line 68, in run_api_meta_data
return query.get_meta_data()
File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/flask_app/dispatcher_query.py", line 656, in get_meta_data
l.append(self.instrument.get_parameters_list_as_json(prod_name=prod_name))
File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/analysis/instrument.py", line 396, in get_parameters_list_as_json
l.append(_query.get_parameters_list_as_json(prod_dict=self.query_dictionary))
File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/analysis/queries.py", line 247, in get_parameters_list_as_json
l.append(par.reprJSON())
File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/analysis/parameters.py", line 305, in reprJSON
return dict(name=self.name, units=self.units, value=self.value)
File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/analysis/parameters.py", line 218, in units
return self._units
AttributeError: 'OSAVersion' object has no attribute '_units'
Traceback (most recent call last):
File "/src/cdci-data-analysis/cdci_data_analysis/analysis/queries.py", line 581, in process_query_product
process_products_query_out=self.process_product(instrument,self.query_prod_list,api=api,config=config,**kwargs)
File "/src/cdci-data-analysis/cdci_data_analysis/analysis/queries.py", line 554, in process_product
query_out= self.process_product_method(instrument, query_prod_list, api=api, **kwargs)
File "/usr/local/lib/python3.8/site-packages/cdci_osa_plugin/osa_image_query.py", line 132, in process_product_method
html_fig = query_image.get_html_draw(
File "/src/cdci-data-analysis/cdci_data_analysis/analysis/products.py", line 386, in get_html_draw
_du=self.data.get_data_unit(ID=data_ID)
File "/usr/local/lib/python3.8/site-packages/oda_api/data_products.py", line 419, in get_data_unit
return self.data_unit[ID]
IndexError: list index out of range
curl "https://www.astro.unige.ch/mmoda/dispatch-data/inspect-state?token=$(cat ~/.oda-token | xargs)&job_id=62840af4b40af6c8" | jq
we should discuss if/where it makes sense
for single scw 066500220010.001 and E1 25, E2 80 ?
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/cdci_data_analysis/analysis/queries.py", line 581, in process_query_product
process_products_query_out=self.process_product(instrument,self.query_prod_list,api=api,config=config,**kwargs)
File "/usr/local/lib/python3.8/site-packages/cdci_data_analysis/analysis/queries.py", line 554, in process_product
query_out= self.process_product_method(instrument, query_prod_list, api=api, **kwargs)
File "/usr/local/lib/python3.8/site-packages/cdci_osa_plugin/osa_lightcurve_query.py", line 413, in process_product_method
_html_fig.append(query_lc.get_html_draw())
File "/usr/local/lib/python3.8/site-packages/cdci_osa_plugin/osa_lightcurve_query.py", line 233, in get_html_draw
dx = data['TIMEDEL']*0.5
File "/usr/local/lib/python3.8/site-packages/astropy/io/fits/fitsrec.py", line 507, in __getitem__
return self.field(key)
File "/usr/local/lib/python3.8/site-packages/astropy/io/fits/fitsrec.py", line 694, in field
column = self.columns[key]
File "/usr/local/lib/python3.8/site-packages/astropy/io/fits/column.py", line 1644, in __getitem__
key = _get_index(self.names, key)
File "/usr/local/lib/python3.8/site-packages/astropy/io/fits/column.py", line 2046, in _get_index
raise KeyError(f"Key '{key}' does not exist.")
KeyError: "Key 'TIMEDEL' does not exist."
ERROR:cdci_data_analysis.flask_app.dispatcher_query:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/astropy/io/fits/column.py", line 2037, in _get_index
indx = names.index(key.rstrip())
ValueError: 'TIMEDEL' is not in list
Related to oda-hub/dispatcher-app#250
Is it intended?
I guess it should not be, can this be confirmed?
check that those parameters which are always set in frontend, are always set in URLs, even if not provided in api request.
ile "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/flask_app/app.py", line 254, in run_analysis
r = query.run_query(disp_conf=app.config['conf'])
File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/flask_app/dispatcher_query.py", line 1433, in run_query
query_out = self.instrument.run_query(product_type,
File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/analysis/instrument.py", line 239, in run_query
query_out = query_obj.run_query(self, out_dir, job, run_asynch,
File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/analysis/queries.py", line 663, in run_query
query_out = self.process_query_product(instrument,
File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/analysis/queries.py", line 582, in process_query_product
process_products_query_out=self.process_product(instrument,self.query_prod_list,api=api,config=config,**kwargs)
File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/analysis/queries.py", line 555, in process_product
query_out= self.process_product_method(instrument, query_prod_list, api=api, **kwargs)
File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_osa_plugin/osa_spectrum_query.py", line 330, in process_product_method
query_spec.write()
File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/analysis/products.py", line 338, in write
self.data.write_fits_file(file_path, overwrite=overwrite)
File "/src/oda-api/oda_api/data_products.py", line 532, in write_fits_file
self.to_fits_hdu_list().writeto(filename,overwrite=overwrite)
@burnout87 could we make a test to request ISGRI LC for a single pointing, 066500220010.001 ?
j_ima_iros minimum time bin is 0.1 s with default to 4 s
the same is reflected in jemx_science_analysis
I have made a request from the API with 0.02 s bin job_id: 2bbfe99e
and one from the web interface with 0.02 s time bin job_id: e2c6aace
They both returned a light curve with 4 s time bin.
It is a good safety feature, but one should warn the user that the query parameters have been adapted to reflect limits.
[I do not know if this is even possible in automatic fashion, but we discussed something on weird inputs for ISGRI energy range]
the find search didn't yield any results
When extracting a long-term light curve of some sources, I noticed that occasionally, there are science windows with long duration. 10 ks should be safe value to enable one point per science window with ISGRI
Seems like no change to frontend, core dispatcher, or backend is needed.
but need to test if there are no hidden difficulties.
once released, send a GCN with explanations
I do not think this was done yet?
test would verify that condition set by the plugin is properly applied for the selected role.
The code that gets the source names from ddosa seem the one around this line, but it does not perform the string replacement that we search.
it's better than just relying on the token.
like "integral_data_rights" with two possible values "public" and "all-private"
that is, this parameter should be only allowed to be set to "all-private" if such the role "integral-private" is set.
I suspect parsing issue
In general, the rule is
However if one uploads a list of scw, the number is limited to 50 even if you are registered. There should be a unique rule to be applied.
e.g.:
it was added in a rush, and perhaps this one was forgotten. Easy hotfix is warranted.
When there are too many sources to consider, ii_skyimage
crashes with segfault. There is an automatic workaround (increasing sig threshold), but sometimes it still return an error. In such a case a pre-set catalog should be probably used.
added as a known issue
https://github.com/oda-hub/known-issues/blob/main/README.md#isgri
for public/private data status
to consider
HEASOFT lcurve needs specific keywords in the lightcurve output. dispatcher plugin should set them.
see there:
https://gitlab.astro.unige.ch/oda/api-clients/oda_api_wrapper/-/blob/master/oda_integral_wrapper/wrapper.py#L780
(lc is the object returned by oda_api)
and let's make a test for it!
https://github.com/oda-hub/dispatcher-app/runs/4368086679?check_suite_focus=true
maybe something is not set
for very large mosaic (>10k) making mosaic takes a while. Also every new added scw requires complete re-making of the mosaic.
Instead, we can make mosaics per-orbit, and then split merge them. With mimosa or varmosaic.
This will resolve both issues quoted above.
from oda_api.api import DispatcherAPI
disp=DispatcherAPI(url='http://dispatcher.staging.internal.odahub.io', instrument='mock')
par_dict={
"src_name": "4U 1700-377",
"RA": "270.80",
"DEC": "-29.80",
"T1": "2021-05-01",
"T2": "2021-05-05",
"T_format": "isot",
"instrument": "jemx",
"osa_version": "OSA11.0",
"radius": "4",
"max_pointings": "50",
"integral_data_rights": "all-private",
"jemx_num": "1",
"E1_keV": "3",
"E2_keV": "20",
"product_type": "Real",
"detection_threshold": "5",
"product": "jemx_lc",
"time_bin": "4",
"time_bin_format": "sec",
"catalog_selected_objects": "1,2,3",
"selected_catalog" :'{"cat_frame": "fk5", "cat_coord_units": "deg", "cat_column_list": [[0, 1, 2], ["GX 5-1", "MAXI SRC", "H 1820-303"], [96.1907958984375, 74.80066680908203, 66.31670379638672], [270.2771301269531, 270.7560729980469, 275.914794921875], [-25.088342666625977, -29.84027099609375, -30.366628646850586], [0, 1, 0], [0.05000000074505806, 0.05000000074505806, 0.05000000074505806]], "cat_column_names": ["meta_ID", "src_names", "significance", "ra", "dec", "FLAG", "ERR_RAD"], "cat_column_descr": [["meta_ID", "<i8"], ["src_names", "<U10"], ["significance", "<f8"], ["ra", "<f8"], ["dec", "<f8"], ["FLAG", "<i8"], ["ERR_RAD", "<f8"]], "cat_lat_name": "dec", "cat_lon_name": "ra"}',
"token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJlbWFpbCI6IkNhcmxvLkZlcnJpZ25vQHVuaWdlLmNoIiwibmFtZSI6ImNmZXJyaWdubyIsInJvbGVzIjoiYXV0aGVudGljYXRlZCB1c2VyLCBhZG1pbmlzdHJhdG9yLCBjb250ZW50IG1hbmFnZXIsIGdlbmVyYWwsIGludGVncmFsLXByaXZhdGUtcWxhLCBtYWdpYywgdW5pZ2UtaHBjLWZ1bGwsIHB1YmxpYy1wb29sLWhwYywgYW50YXJlcywgc2RzcyIsImV4cCI6MTYyMzEzNDg5M30.UVaNwQBGdcUly3bsQDH17X0Wa9fPz8nTWyphHP6yoBY"
}
data_collection_lc.show()
ID=0 prod_name=jemx_lc_0_H1820303 meta_data: {'src_name': 'H 1820 303', 'time_bin': 4.62963e-05, 'time': 'TIME', 'rate': 'RATE', 'rate_err': 'ERROR'}
ID=1 prod_name=jemx_lc_1_MAXISRC meta_data: {'src_name': 'MAXI SRC', 'time_bin': 4.62963e-05, 'time': 'TIME', 'rate': 'RATE', 'rate_err': 'ERROR'}
The "-" in the source name of "H 1820-303" disappears in the "show()" method.
to estimate if feasible
could be stored in the very same dummy directory, just called differently
might be easy
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.