Need new component, InputSandboxCache? about crabserver HOT 16 CLOSED

sfoulkes: This seems reasonable to me. I'd suggest using cherrypy to serve up the files as there is already support for that in WMCore and a cron'd script to prune older sandboxes as the disk fills up. Diagnostics and other bells and whistles would be built into the cherrypy server or the crab rest interface.

from crabserver.

ewv: A note to myself on how to implement this:

http://www.cherrypy.org/wiki/FileUpload

from crabserver.

ewv: Please review

Uses (modified) REST model for the uploading part, Page model for downloading.

from crabserver.

mmascher: Ouch... You are right it works. I' a moron...

from crabserver.

ewv: Simon, can you please review and then either check in or pass it on to someone else for further review?

from crabserver.

metson: The code in the patch looks fine from a quick look. However, shouldn't this be in CRAB and not WMCore? What other systems will have a UserFileCache?

from crabserver.

ewv: I don't have a strong opinion, but I put it in WMCore for two reasons.

1. I wanted it started with the Local WQ/Agent cluster of things
2. I figured it may be of more general use with MC workflows that have to ship big LHE files or whatever. Those could be run in production.

So make a decision and I will relocate it if needed.

from crabserver.

evansde: For 2, in the production case the LHE files will either be converted to EDM GEN files at CERN or shipped via squids or the DM system like normal data.
So I think that would make this Crab Only.

Question: Is there a maxmimum size limit on the input sandbox? The idea that a user could dump a couple of GB of data in there and send it to a batch system that copies it per job could lead to some issues with load, even with caching etc.

from crabserver.

ewv: At the moment there is no limit, but we can and should enforce something in the client, I think. I think CRAB2 enforces a 50 MB limit which comes from gLite. We had issues with PAT libraries being larger than that when they weren't in the release, but I haven't heard of that recently. So maybe 50 or 100 MB will be a good starting point.

So in answer to Simon's question, it sounds like I should relocate this to CRABServer.

from crabserver.

metson: Replying to [comment:11 ewv]:

So in answer to Simon's question, it sounds like I should relocate this to CRABServer.

Yeah, I think that's best. Also, the patch has no tests in it. Can you add them at the same time?

from crabserver.

ewv: Yeah. I'll have to find an example of tests for a web service.

from crabserver.

metson: https://svnweb.cern.ch/trac/CMSDMWM/browser/WMCore/trunk/test/python/WMCore_t/WebTools_t

from crabserver.

spiga: Replying to [comment:11 ewv]:

At the moment there is no limit, but we can and should enforce something in the client, I think. I think CRAB2 enforces a 50 MB limit which comes from gLite. We had issues with PAT libraries being larger than that when they weren't in the release, but I haven't heard of that recently. So maybe 50 or 100 MB will be a good starting point.

the limit we have now should be 100MB (the gLite limit was 10MB and apply to direct submission only). I agree to start with 100, also I'd made it configurable.

So in answer to Simon's question, it sounds like I should relocate this to CRABServer.

from crabserver.

ewv: Please review. New and improved with Unit tests

from crabserver.

ewv: Can this please be reviewed and checked in?

from crabserver.

spiga: As agreed I would give first the current stuff to integration and then move ahead.

Few things are still missing/not working on the deploy (including some problem I discovered yesterday which apparently doesn't show up it previous test?!?).

To be more precise: as soon as the next wmcore tag is cut we move on.

from crabserver.