opencadc / caom2service Goto Github PK

These should be position_dimension_naxis1[2] instead.

The current method of using TAP query processing is too subtle and isDownloadable() is deprecated.

The definitive correct way: get the plane release (dates) and read groups and use AccessUtil (from caom2 core lib) to determine permissions. Viable as of CAOM 2.4 and needed to add support for Artifact.contentRelease and Artifact.contentReadGroups

When loading a particular Plane into the caom2ops/meta service, some observations are throwing errors:

java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.Long at ca.nrc.cadc.caom2ops.Util.getLong(Util.java:86) at ca.nrc.cadc.caom2ops.mapper.PlaneMapper.mapRow(PlaneMapper.java:226) at ca.nrc.cadc.caom2ops.CaomTapQuery.buildObservation(CaomTapQuery.java:359) at ca.nrc.cadc.caom2ops.CaomTapQuery.performQuery(CaomTapQuery.java:165) at ca.nrc.cadc.caom2.meta.MetaQueryRunner.doIt(MetaQueryRunner.java:210) at ca.nrc.cadc.caom2.meta.MetaQueryRunner.run(MetaQueryRunner.java:159) at ca.nrc.cadc.uws.server.AbstractExecutor.executeSync(AbstractExecutor.java:208) at ca.nrc.cadc.uws.server.AbstractExecutor.execute(AbstractExecutor.java:171) at ca.nrc.cadc.uws.server.SimpleJobManager.execute(SimpleJobManager.java:220) at ca.nrc.cadc.uws.server.SyncServlet.processRequest(SyncServlet.java:410) at ca.nrc.cadc.uws.server.SyncServlet.access$000(SyncServlet.java:109) at ca.nrc.cadc.uws.server.SyncServlet$1.run(SyncServlet.java:298) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:421) at ca.nrc.cadc.uws.server.SyncServlet.doit(SyncServlet.java:293) at ca.nrc.cadc.uws.server.SyncServlet.doGet(SyncServlet.java:247) at javax.servlet.http.HttpServlet.service(HttpServlet.java:624) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423) at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:190) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748)

The Util.getLong() method is trying to cast a Long to an Integer.

Version 1.3.1 does not have this problem, and so was introduced in 1.3.2.

The PlaneMapper class throws a NullPointerException when caom2:Plane.energy.bounds.samples are missing, but caom2:Plane.time.bounds.samples are present.

In a request with FORMAT=foo the error handling fails to create a TableWriter impl that can output the error message correctly. The bug is in PluginFactory.getTableWriter being used to get the result TableWriter (should throw UnsupportedOperation exception) and to get the error tableWriter (which should always successfully return a writer.

Correct behaviour:

• a separate method getErrorTableWriter should be implemented in PluginFactory
• it should look at RESPONSEFORMAT/FORMAT and return the requested impl if possible
• it should return VOTableWriter if RESPONSEFORMAT/FORMAT are invalid and never throw

In the getURL() on line 195, the StorageResolver class is determined from the URI scheme of the given URI. That resolver does not have the authMethod from the CaomArtifactResolver set so that it can provide the appropriate URL (i.e. include /auth for basic authentication).

If the download URL does not provide a filename via the Content-Disposition header the tar generation fails with a null pointer exception.

eg ivo://cadc.nrc.ca/mirror/HST?ocqn3hytq/ocqn3hytq-RAW_STANDARD

This fails in toPlaneURI(PublisherID) because that code tries to extract the collection name from the path but gets mirror/HST instead of HST.

CADC code to generate publisherID values assumes a base URI and appends collection so a quick fix is to use the last path component, but that is not mandated anywhere.

trailing QUERY_STATUS = FAIL leads to particularly obscure bugs (not found in meta and datalink)

The following caom 2.4 fields are not populated from the ADQL query response by the mappers:

Observation.metaReadGroups
Plane.dataReadGroups
Plane.metaReadGroups
Artifact.contentReadGroups

Here is a search for a SITELLE observation:

https://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/en/search/?Observation.observationID=2384125p&Observation.proposal.id=19AC27&[email protected]=NONE&Plane.position.bounds=NGC4569&Observation.collection=CFHT&Observation.instrument.name=SITELLE&Plane.energy.bandpassName=SN3#sortCol=caom2%3APlane.time.bounds.lower&sortDir=dsc&col_1=_checkbox_selector;;;&col_2=caom2%3AObservation.uri;;;&col_3=caom2%3AObservation.collection;;;&col_4=caom2%3AObservation.observationID;;;&col_5=caom2%3AObservation.sequenceNumber;;;&col_6=caom2%3APlane.productID;;;&col_7=caom2%3APlane.position.bounds.cval1;;;&col_8=caom2%3APlane.position.bounds.cval2;;;&col_9=caom2%3AObservation.target.name;;;&col_10=caom2%3APlane.time.bounds.lower;;;&col_11=caom2%3APlane.time.exposure;;;&col_12=caom2%3AObservation.instrument.name;;;&col_13=caom2%3APlane.energy.bandpassName;;;&col_14=caom2%3APlane.calibrationLevel;;;&col_15=caom2%3AObservation.type;;;&col_16=caom2%3AObservation.proposal.id;;;&col_17=caom2%3AObservation.proposal.pi;;;&col_18=caom2%3APlane.dataRelease;;;

Using the package service endpoint (https://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/pkg?ID=ivo%3A%2F%2Fcadc.nrc.ca%2FCFHT%3F2384125p%2F2384125p) will result in a file containing an error message, but it has a .tar extension:

unexpected failure: java.lang.RuntimeException: entry size '8606365612' is too big ( > 8589934591 ).
CFHT-2384125p-2384125p/README0100644 0000000 0000000 00000000222 14043065762 013230 0ustar000000000 0000000 OK 2384125z.hdf5 1ca4483aac2b46e6281a80d41b36aa39 https://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/data/pub/CFHT/2384125z.hdf5

As the HD5 file is larger than 8GB.

Plane.position.bounds is a Shape and can be a circle or simple polygon. The TAP type should be char,*,shape (proposed for DALI-1.2)

Plane.position.bounds.samples is only set when Plane.position.bounds is a Polygon. The TAP type should be char,*,multipolygon (proposed for DALI-1.2)

A pair of Converter/Formatter is needed so that the position_bounds column is used in predicates but the position_bounds_points column should be used for polymorphic output.

Aside: It would have been nicer if the spoly column had the internal-only name (eg. position_bounds_spoly) and the real values were stored in position_bounds; then one would only need to modify predicate column reference in the CaomRegionConverter. Such a change spans caom2-tap-server and caom2persistence and would need to be coordinated