Comments (15)
mcinquil: How do we want to behave on this?
The CRAB rest interface cannot-shouldn't read in the agent database (where the location of the job output is stored wmbs_job.cache_dir).
The CRAB rest interface could get the log files from couch. Currently the information stored in couch is only the FWJR.
Probably this implies to modify the JobAccountant to upload not just the FWJR into couch, but also the log files.
from crabserver.
evansde: Logfiles can be arbitrary large size, they shouldnt go into couch, it will add a bunch of unnecessary load on it.
LogArchives should be staged out by the job and if the user wants them back physically, they should be injected into the AsyncStageout system along with the other output. Logfiles are just another kind of of user output data.
from crabserver.
spiga: this include also view to get the FJR from couch.
from crabserver.
spiga: Please Review
from crabserver.
spiga: Please Review
from crabserver.
metson: Steve, can you take a look at patch-series-spiga-couch-view.tar.gz to check it doesn't add undue load to this design doc. It looks fine to me, but I also note that the design has a lot of what sound like large views. (we should totally write a view profiler...)
Daniele, patch-series-spiga-crabrest-logAPI doesn't apply to the HEAD, maybe I should have put this in before #1421? If so apologies...
{{{hilbert:CRABServer metson$ stg import --sign -s ~/Downloads/patch-series-spiga-crabrest-logAPI.tar.gz
Checking for changes in the working directory ... done
Importing patch "01-CS-LOG.patch" ... error: patch failed: src/python/CRABRESTModel.py:83
error: src/python/CRABRESTModel.py: patch does not apply
:115: new blank line at EOF.
+
stg import: Diff does not apply cleanly}}}
from crabserver.
spiga: Replying to [comment:9 metson]:
Steve, can you take a look at patch-series-spiga-couch-view.tar.gz to check it doesn't add undue load to this design doc. It looks fine to me, but I also note that the design has a lot of what sound like large views. (we should totally write a view profiler...)
Daniele, patch-series-spiga-crabrest-logAPI doesn't apply to the HEAD, maybe I should have put this in before #1421? If so apologies...
yes it was so. no Problem I'll update the patch tomorrow.
{{{hilbert:CRABServer metson$ stg import --sign -s ~/Downloads/patch-series-spiga-crabrest-logAPI.tar.gz
Checking for changes in the working directory ... done
Importing patch "01-CS-LOG.patch" ... error: patch failed: src/python/CRABRESTModel.py:83
error: src/python/CRABRESTModel.py: patch does not apply
:115: new blank line at EOF.
+
stg import: Diff does not apply cleanly}}}
from crabserver.
spiga: Please Review
from crabserver.
spiga: simon, with the 1421 you committed most of the things where in this patch.. the remaining part has been attached again.
from crabserver.
metson: thanks, Steve, can you comment on the view (patch-series-spiga-couch-view.tar.gz)? It looks fine but I don't currently have a way to know it's not going to add undue load to the FWJRDump application.
I'll commit the crab-server patch now, assuming no problems...
from crabserver.
metson: (In e174f34) add missing part to close the patch, fixes #1303
Signed-off-by: Simon Metson [email protected]
from crabserver.
metson: Reopening, the view hasn't been committed.
from crabserver.
spiga: Simon, Steve what about this view?
is that fine? can I check this in ?
from crabserver.
sfoulkes: (In 12673) Add a couch view to retrieve the PFNs of archived logs. Fixes #1303.
From: Danielle Spiga [email protected]
Signed-off-by: Steve Foulkes [email protected]
from crabserver.
sfoulkes: (In 12675) Add a couch view to retrieve the PFNs of archived logs. Fixes #1303.
From: Danielle Spiga [email protected]
Signed-off-by: Steve Foulkes [email protected]
from crabserver.
Related Issues (20)
- initial CRAB UI page is almost empty HOT 1
- SUSPEND a Rucio rule before deleting
- bump WMCore to 2.3.6rc1 HOT 2
- Canary deployments - REST, TW, Publisher HOT 4
- schedulers - implement automatic reboot HOT 3
- change of memory in resubmit does not work HOT 12
- remove check for `username_CRAB` username for myproxy.
- #831: Test prod CRABClient using preprod REST instance and CMSSW_13_0_2 CMSSW release HOT 3
- test JobRouter on HTC 23.9 HOT 8
- fix ReportRecallQuota for new usage of Rucio account HOT 1
- add an API (URL) to retrieve list of SUBMITREFUSED in last N minutes
- Action Items from Udine Workshop
- add a GIT Etiquette section to crab-docs
- refactor crab-doc
- find a better way to prevent users from recalling PB's HOT 2
- use Stefano's OS token as S3 secrets in REST
- get rid of crab3/CRAB3ServerConfig
- Merge crab rest, log pipeline, and serviceaccount manifests together in single helm chart HOT 3
- Canary (QA) deployment for TW and Publishers
- CRAB monitoring diagram HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from crabserver.