how XIO works with tar? about gct HOT 6 CLOSED

Hi,
I'm not sure what you're trying to do exactly and have no personal experience with using globus-url-copy using the pipe option, but calling tar without arguments gives this error message.
From the options in the documentation for globus-url-copy I would think you probably need only -dst-pipe with the correct -x flags, probably something like tar -xf -.

from gct.

@Jin-Chao
Unfortunately I also never tried that popen feature, but there was a script in the GCT source that according to its description could perform directory copies via tar and globus-url-copy using that popen feature. It's still available in the repo's history:

https://github.com/gridcf/gct/blob/ab99a0e1a7ffe22295edf05ad545bb5175056223/gass/copy/source/globus-dir-copy

Checking the source should give you the correct usage for the corresponding use case (i.e. get, put, 3rd-party, etc.). For 3rd-party transfers the popen feature needs to be enabled on both servers.

One thing to note is, that for huge directories leading to huge tarballs and longer transfer times, the restart functionality you can use with globus-url-copy does not work as expected when the transfer is interrupted, as you can't seek in a stream (the tarball doesn't exist as file!). Because of that, the stream always needs to be retransferred from the beginning, in case of transfer problems.

from gct.

@Jin-Chao
Do you need any further assistance or can this issue be closed?

from gct.

@fscheiner
Sorry for late response.
Thank you very much for your help. We have reviewed globus-dir-copy, and it worked well. Please close this thread.

from gct.

@Jin-Chao
Glad we could work it out.

from gct.

@Jin-Chao @msalle
https://web.archive.org/web/20100612224141/http://www.mcs.anl.gov/~bresnaha/Stretch/ has some very interesting additional examples that use the "popen" functionality. Might come in handy sometime. :-)

from gct.