AsyncStageout component about crabserver HOT 10 CLOSED

metson: Hassen, is this still valid? I think all these changes went in to #95 or am I wrong?

from crabserver.

riahi: Not all changes went to #95. The patches, which went in #95, were to have AsyncStageout as component. I aligned the patches of this ticket with changes done in ticket #95.

from crabserver.

metson: ok, cool thanks Hassen! I'll bring my local repo up to sync with your changes and hopefully commit tonight.

from crabserver.

riahi: Perfect!...Thank you

Hassen

from crabserver.

metson: This patch series looks good, and has been committed.

I have a few questions:

1. should TransferManager be removed? (I think yes)
2. is ''defaultProxy'' in the config the servers certificate? (I think it should be)
3. cleanSpace should probably reuse some code (not sure where from) to issue the appropriate delete command (IIRC lcg-rm is better in some cases) and/or deal with fallback/retries
4. getFTServer should read the mapping from a file specified in the configuration, instead of a hardcoded list
5. the orderFileAlgo should be replaced by changes to the CouchDB view, and how that's queried.

1 & 2 aren't a big deal (but I'd like to clarify before moving on). 3-5 should probably be separate tickets. If you agree can you open them and add the numbers to this ticket?

from crabserver.

metson: in fact getFTServer could use a database in couch... maybe over complicated.

from crabserver.

metson: oh, and one last thing:

1. using subprocess.PIPE for stdin, stderr and stdout can lead to deadlocks when the command is verbose ([http://docs.python.org/library/subprocess.html#convenience-functions for more info]). ftscp is probably noisey enough to be a concern, so we should look at that (e.g. write to a log that is then parsed, rather than reading the stdout)

from crabserver.

metson: (In 8920) Change main class name, add defaultProxy parameter to config, implement ftscp machinery, references #311

From: Simon [email protected]

from crabserver.

riahi: I answer to your questions:

1- Yes, the TransferManager should be removed.
2- The defaultProxy is a valid user certificate in the server. I set this default value to be able to do tests and run transfers since a valid user certificate is needed to transfer files using ftscp as well as to remove files.
3- Yes maybe lcg-rm is better in some case. I'll open a new ticket to issue the appropriate delete command.
4- Yes, here we can use also a configuration file to have a mapping of FTS servers or get needed info from couch
5- I agree that orderFileAlgo should be replaced by changes to the CouchDB views to avoid complicated algorithms. I'll open a new ticket for this.
6- We should maybe evaluate this in an another ticket.

Cheers
Hassen

from crabserver.

metson: Ok, the other tickets are:
3: #337
4: #339
5: #338

Closing this one now.

from crabserver.