Add stageout parameters to WMSpec about crabserver HOT 12 CLOSED

sfoulkes: Are we not using the async stageout stuff for this?

from crabserver.

riahi: For the async stageout stuff, there is only the async_dest attribute that was added, as user output parameter, in WMSpec to specify the remote destination.

from crabserver.

riahi: The remote directories of users outputs (equivalent to user_remote_dir in CRAB2) parameter is still missing in WMSpec.

from crabserver.

ewv: Yes, I know. I will get to it soon, I hope.

from crabserver.

spiga: what's giving problem here #1330 is what should be covered by this thicket too.

from crabserver.

ewv: Please review.

Patch propagates a setting of arguments['UserSubDir'] down to StageOut.py as self.step.userSubDir, similarly to asyncDest.

from crabserver.

ewv: Pre-requisite for fixing the LFN issue. Please review.

from crabserver.

spiga: It seems fine to me. Hassen could you have a look?

from crabserver.

riahi: The code seems ok to me too but I don't understand why we add this new parameter (publishName) similarly to the asyncDest parameter.

The asyncDest is propagated to the AsyncStageOut across couchDB (into job fwjr) and therefore the AsyncStageOut will pick it from there. The AsyncStageOut gets, from the fwjr, the path (LFN) of the output + site source + asyncDest to transfer the output to the final destination (asyncDest) removing only tmp if it is in the LFN (like /store/tmp/user/username/outputfile.root). Given that, the publishName will not be needed directly by the AsyncStageOut but it will be needed/used to build the LFN of the user output in the closeSE of the WN.

why and in that way this parameter will be propagated in

from crabserver.

riahi: Replying to [comment:14 riahi]:

The code seems ok to me too but I don't understand why we add this new parameter (publishName) similarly to the asyncDest parameter.

The asyncDest is propagated to the AsyncStageOut across couchDB (into job fwjr) and therefore the AsyncStageOut will pick it from there. The AsyncStageOut gets, from the fwjr, the path (LFN) of the output + site source + asyncDest to transfer the output to the final destination (asyncDest) removing only tmp if it is in the LFN (like /store/tmp/user/username/outputfile.root). Given that, the publishName will not be needed directly by the AsyncStageOut but it will be needed/used to build the LFN of the user output in the closeSE of the WN.

from crabserver.

ewv: I guess I don't understand. Aysnc stageout may not need this, but the StageOut step does, right, to know how to do the local stageout from whatever file the user wrote to /store/temp/user/username/PrimaryDS/username-publishName-procVer/ etc?

from crabserver.

riahi: Right. My comment was just to avoid confusion since I see that you have followed the asyncDest parameter implementation to implement the publishName parameter. But as you said, the publishName parameter is needed for the StageOut step.

from crabserver.