fast-data-transfer / fdtcp Goto Github PK

Fast Data Transfer

License: Apache License 2.0

Java 4.35% Shell 2.96% Python 90.73% Perl 1.97%

fdtcp's Introduction

FDT - PhEDEx integration project (fdtcp)

This experimental integration project is aimed at linking CMS (CERN Compact Muon Solenoid [1]) data transfers system PhEDEx [2] and Caltech's high-perf data transfer application FDT (Fast Data Transfer) [3]. The ultimate goal is to bring dynamic network circuits provided by the DYNES (Dynamic Network Services) [4], which is an Internet2 project funded by NSF (National Science Foundation), to CMS data transfers. The DYNES plug-in for FDT application exists and the purpose of fdtcp is to integrate FDT with high-level and claimed to be network-agnostic system PhEDEx. The fdtcplib projects provides CLI utilities and daemon fdtd application which are Python wrappers for FDT application. Grid authentication via Grid Map File, third-party transfers and various other features necessary for integration with PhEDEx are implemented here. The test transfers were realized between Caltech and Nebraska Tier2 CMS centres utilizing HDFS (Hadoop Distributed File System).

More on: https://twiki.cern.ch/twiki/bin/view/Main/PhEDExFDTIntegration doc/fdtcp_chep2010_proceedings.pdf

[1] http://cms.web.cern.ch/ [2] https://cmsweb.cern.ch/phedex/ [3] http://monalisa.cern.ch/FDT/ [4] http://iopscience.iop.org/1742-6596/396/4/042065/pdf/1742-6596_396_4_042065.pdf

fdtcp's People

Contributors

Stargazers

Watchers

Forkers

fno2010

fdtcp's Issues

All TODOs

Solve all TODOs...

There should be fdtls -(laht)

Options to support:

-a do not ignore entries starting with .
-t sort by modification time, newest first
-l use a long listing format
-h with -l, print sizes in human readable format (e.g., 1K 234M 2G)

failure to clean up process

Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 764, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/site-packages/Pyro-3.16-py2.7.egg/Pyro/protocol.py", line 1126, in connectionHandler
    if self.getAdapter().handleConnection(conn, self):
  File "/usr/lib/python2.7/site-packages/Pyro-3.16-py2.7.egg/Pyro/protocol.py", line 836, in handleConnection
    ver,body,pflags = self.receiveMsg(conn)
  File "/usr/lib/python2.7/site-packages/Pyro-3.16-py2.7.egg/Pyro/protocol.py", line 633, in receiveMsg
    self.returnException(conn, ProtocolError(msg))
  File "/usr/lib/python2.7/site-packages/Pyro-3.16-py2.7.egg/Pyro/protocol.py", line 813, in returnException
    pic=Pyro.util.getXMLPickle('gnosis')
  File "/usr/lib/python2.7/site-packages/Pyro-3.16-py2.7.egg/Pyro/util.py", line 435, in getXMLPickle
    raise NotImplementedError('xml pickling implementation (%s) is not available' % impl)
NotImplementedError: xml pickling implementation (gnosis) is not available

whenever transfer happens, it does not report status to the executor

move to newest Pyro4 4.49

support local file transfer

should support fdtcp file:///tmp/file/name.root fdt://server-name... and fdt://server-name... file:///tmp/file/name.root to download file. Same for the list of files

move to newest psutil

to 4.4.2 version

"source" is an invalid command in sh

/usr/bin/wrapper_fdt.sh: 7: /usr/bin/wrapper_fdt.sh: source: not found

Since the shebang of wrapper_fdt.sh is !#/bin/sh, we need to use . to import another shell script.

Server will stay running in case client fails...

------------------------------------------------------------------------------
stdout:
running: '/usr//bin/java -XX:-HeapDumpOnOutOfMemoryError -Xms1024m -Xmx1024m -cp /usr/share/java/fdt.jar lia.util.net.copy.FDT -bs 64K -p 51085 -bio -wCount 5 -f 188.184.81.127 -nolock -S -noupdates -enable_apmon 131.215.207.20:8884 -monID 188.184.29.213:51085:10000 -notmp'
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000eaa80000, 358088704, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 358088704 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /tmp/jvm-4953/hs_error.log

------------------------------------------------------------------------------
stderr:

------------------------------------------------------------------------------
DEBUG     2016-11-01 16:36:19,702 transfer FDT executors container before removing attempt (total 66 items).
CRITICAL  2016-11-01 16:36:19,702 transfer Process of executor id 'fdtcp-vlad-vm-slc6.cern.ch--188.184.81.127-to-188.184.29.213--2016-11-01-16h:36m:16s-qtoqv' (process PID: 4952) still exists! Not removing from FDTD executors container.
DEBUG     2016-11-01 16:36:19,702 transfer Executor process PID: 4952 'sudo -u dynes /usr/bin/wrapper_fdt.sh -XX:-HeapDumpOnOutOfMemoryError -Xms1024m -Xmx1024m -cp $FDTJAR lia.util.net.copy.FDT -bs 64K -p 51085 -bio -wCount 5 -f 188.184.81.127 -nolock -S -noupdates -enable_apmon 131.215.207.20:8884 -monID 188.184.29.213:51085:10000 -notmp' id:'fdtcp-vlad-vm-slc6.cern.ch--188.184.81.127-to-188.184.29.213--2016-11-01-16h:36m:16s-qtoqv' has syncFlag not set, continue.
DEBUG     2016-11-01 16:36:19,702 transfer End of request CleanupProcessesAction serving.
------------------------------------------------------------------------------



WARNING   2016-11-01 16:36:19,702 transfer Logger closing.