Comments (8)
riahi: The component is written in WMAgent style and ready to be commited. I expect is fine to check the code in cvs. That is fine? Any constraints about the agent naming convention? The following is the workflow:
- FileTransferPoller queries the couchDB docs to get needed information
- It loops on users to call the ftscpWrapper for them. It gives to ftsWrapper processes (working in parallel): mapping of pfn sources and destinations, userProxy, destination site, list of destination pfn
- Each ftscpWapper process cleans the storage space removing the destination pfn got
- create ftscp copy jobs splitting by source the mapping of pfn sources and destination in files (since different sources should be in different copy jobs to be transferred using ftscp command)
- Call ftscp command for the copy job files created
- remove the pfn for succeeded transfers
The following is what I see as next steps :
- Overload the mark_good method to remove succeeded transfers from couchDB (discuss couchDB fields to use)
- Discuss from where the user proxy can be got
from crabserver.
metson: Hi,
The code should go into SVN, can we do that next week after the MC code fest? Will that hold you up on other work? I'm not sure by what you mean by steps 2 and 4 - creating the input for ftscp was done via a view and list - have you changed this? I'd like to discuss that further before the code goes into version controll.
Can you open tickets up for the two issues you mention at the end of your comment?
Cheers
Simon
from crabserver.
riahi: Hi,
For the commit I'll wait your green light since you are the boss :-)
For the step 2, the input for ftscp wrapper is done via view and list: mapping of pfn sources and destinations, destination site and list of destination pfn are got from view and list.
For the step 4, due to the fact that different sources can't use the same FTS server, for e.g:
source X and destination Y uses fts server Z
source T and destination Y can't use the fts server Z
That is why the jobs should be split into different copy job files sorted by source. So, each ftscpWrapper process will create needed copy jobs file before calling ftscp command.
I will open a ticket for the two isseus.
Cheers
Hassen
from crabserver.
riahi: Patch to review and commit is enclosed to the ticket.
from crabserver.
metson: (In 8891) Default config file to fix ticket 95
From: Hassen Riahi [email protected]
from crabserver.
metson: (In 8892) Poller class to fix ticket 95
From: Hassen Riahi [email protected]
from crabserver.
metson: (In 8893) Main component class to fix ticket 95
From: Hassen Riahi [email protected]
from crabserver.
metson: (In 8894) Worker class to fix ticket 95
From: Hassen Riahi [email protected]
from crabserver.
Related Issues (20)
- remove getWebdirForDb from ServerUtilities and PreJob
- Remove TastType backward compatibility
- getTasks() fails to get tasks if tw_name is not initialised
- #830: Test prod CRABClient using test2 REST instance and CMSSW_13_0_2 CMSSW release HOT 13
- make the new SUBMITREFUSED status visible in Grafana
- make sure we do not use cmsRun -p
- variable used before it is defined HOT 7
- odd race condition ? HOT 12
- ZeroDivisionError from TapeRecallManager.py HOT 2
- small, scattered issues
- adapt CheckTapeRecall to "account=username" HOT 1
- improve error reporting when user indicates bad vogroup HOT 1
- CRAB Overview dashboard - requests panel is empty now HOT 2
- Make sure that we have space to put output files in eos
- initial CRAB UI page is almost empty HOT 1
- SUSPEND a Rucio rule before deleting
- bump WMCore to 2.3.6rc1 HOT 2
- Canary deployments - REST, TW, Publisher HOT 4
- schedulers - implement automatic reboot HOT 3
- change of memory in resubmit does not work HOT 12
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from crabserver.