Implement API to get logs file about crabserver HOT 15 CLOSED

mcinquil: How do we want to behave on this?
The CRAB rest interface cannot-shouldn't read in the agent database (where the location of the job output is stored wmbs_job.cache_dir).
The CRAB rest interface could get the log files from couch. Currently the information stored in couch is only the FWJR.
Probably this implies to modify the JobAccountant to upload not just the FWJR into couch, but also the log files.

from crabserver.

evansde: Logfiles can be arbitrary large size, they shouldnt go into couch, it will add a bunch of unnecessary load on it.
LogArchives should be staged out by the job and if the user wants them back physically, they should be injected into the AsyncStageout system along with the other output. Logfiles are just another kind of of user output data.

from crabserver.

spiga: this include also view to get the FJR from couch.

from crabserver.

spiga: Please Review

from crabserver.

spiga: Please Review

from crabserver.

metson: Steve, can you take a look at patch-series-spiga-couch-view.tar.gz to check it doesn't add undue load to this design doc. It looks fine to me, but I also note that the design has a lot of what sound like large views. (we should totally write a view profiler...)

Daniele, patch-series-spiga-crabrest-logAPI doesn't apply to the HEAD, maybe I should have put this in before #1421? If so apologies...

{{{hilbert:CRABServer metson$ stg import --sign -s ~/Downloads/patch-series-spiga-crabrest-logAPI.tar.gz
Checking for changes in the working directory ... done
Importing patch "01-CS-LOG.patch" ... error: patch failed: src/python/CRABRESTModel.py:83
error: src/python/CRABRESTModel.py: patch does not apply
:115: new blank line at EOF.
+
stg import: Diff does not apply cleanly}}}

from crabserver.

spiga: Replying to [comment:9 metson]:

Steve, can you take a look at patch-series-spiga-couch-view.tar.gz to check it doesn't add undue load to this design doc. It looks fine to me, but I also note that the design has a lot of what sound like large views. (we should totally write a view profiler...)

Daniele, patch-series-spiga-crabrest-logAPI doesn't apply to the HEAD, maybe I should have put this in before #1421? If so apologies...

yes it was so. no Problem I'll update the patch tomorrow.

{{{hilbert:CRABServer metson$ stg import --sign -s ~/Downloads/patch-series-spiga-crabrest-logAPI.tar.gz
Checking for changes in the working directory ... done
Importing patch "01-CS-LOG.patch" ... error: patch failed: src/python/CRABRESTModel.py:83
error: src/python/CRABRESTModel.py: patch does not apply
:115: new blank line at EOF.
+
stg import: Diff does not apply cleanly}}}

from crabserver.

spiga: Please Review

from crabserver.

spiga: simon, with the 1421 you committed most of the things where in this patch.. the remaining part has been attached again.

from crabserver.

metson: thanks, Steve, can you comment on the view (patch-series-spiga-couch-view.tar.gz)? It looks fine but I don't currently have a way to know it's not going to add undue load to the FWJRDump application.
I'll commit the crab-server patch now, assuming no problems...

from crabserver.

metson: (In e174f34) add missing part to close the patch, fixes #1303

Signed-off-by: Simon Metson [email protected]

from crabserver.

metson: Reopening, the view hasn't been committed.

from crabserver.

spiga: Simon, Steve what about this view?

is that fine? can I check this in ?

from crabserver.

sfoulkes: (In 12673) Add a couch view to retrieve the PFNs of archived logs. Fixes #1303.

From: Danielle Spiga [email protected]
Signed-off-by: Steve Foulkes [email protected]

from crabserver.

sfoulkes: (In 12675) Add a couch view to retrieve the PFNs of archived logs. Fixes #1303.

From: Danielle Spiga [email protected]
Signed-off-by: Steve Foulkes [email protected]

from crabserver.