Comments (7)
I think implementing such an option specific to particular use cases would be a starting point for a mess. I don't intend to implement rsync -a
into mscp. scp and rsync are different programs because they focus on different things: transferring files and synchronizing directory trees. If I intended to implement a fast backup tool, I would choose a different design.
To enhance mscp, I need a strategy to cover broader cases before implementing such an option. Could you describe your use case? Is --noclobber
enough? (I guess it's not)
from mscp.
I understand your position and will try and give a standard use case that we frequently encounter and are looking at this tool for. We are currently implementing whole genome sequencing for diagnostics. This is approximately 120 Gbyte per patient and we aim to initially run batches of 63 samples. After analysis all this data needs to be shared securely with the participating medical centers. For this we currently use ssh/rsync transfers. However the speed through firewalls and the overall transfer speeds are too slow to transfer all this data. Therefore we are exploring alternatives and your tool looks very promising. One thing that I am worried about is transfer stability possibly leading to corrupt or partial transfers (after which a transfer would need to be restarted or resumed). Hope this helps for our use case, which as I understand might not necessary be yours.
from mscp.
Thank you for describing your use case. Transferring large amounts of genome/health/observed data is also common in my univ :).
One thing that I am worried about is transfer stability possibly leading to corrupt or partial transfers (after which a transfer would need to be restarted or resumed).
I understand that the first priority is the resume feature, for example, when all SSH connections fail, mscp writes out bytes and positions of transferred data to a checkpoint file, and mscp can restart the transfer from the checkpoint. --noclobber
option will not resolve your concern. It will not overwrite remote files, although they are corrupted due to the last transfer failing.
In addition, could you please tell me which rysnc options you use? I don't intend to implement all the rsync features, but a subset of them would be acceptable. I'd like to know what the options are crucial.
from mscp.
I will lookup our current transfer options and get back to you.
About the no-clobber, what rsync does for example is transfer to a temp file a .filename and "simply" moves it to the finale filename after successful transfer.
from mscp.
I will lookup our current transfer options and get back to you.
Thank you!
About the no-clobber, what rsync does for example is transfer to a temp file a .filename and "simply" moves it to the finale filename after successful transfer.
I see.
Anyway, I have implemented the resume option on the dev branch (not merged into main yet). Could you please see the doc and check -W
and -R
options and the Example section? Comments are welcome.
from mscp.
Thanks for sharing! Sounds like a nice option that solves resuming after an interrupted transfer.
Found a few typos:
resumeing - resuming
CHECKOPOINT - CHECKPOINT
chunkes - chunks
Infromation - information
pathces - patches
As to the rsync, right now we use the following option:
rsync -vzrta --perms --exclude 'Checkpoints.txt' --chmod='Du=rwx,Dg=rwsx,Fu=rw,Fg=rw,o-rwx' --progress
The permissions we set to make sure the receiving party can read and remove the files after the transfer. The exclude is used to exclude a file which is used as a transfer completed signal (in the example above it is the Checkpoints.txt file).
from mscp.
Thanks! v0.1.5, including the feature, is released.
Also, thanks for sharing the rsync options. I will consider whether to implement them. If you find any vital options (or features) that will be applicable to broader use cases, please open a new issue.
from mscp.
Related Issues (20)
- glob support HOT 1
- Hpn-ssh? HOT 3
- Possible to add Resume support? HOT 2
- Unable to parse IPv6 addresses HOT 2
- Configurable intervals between SSH connection attempts HOT 1
- Host runs out of memory resulting in mscp being killed HOT 7
- Possibility to provide password through sshpass HOT 3
- unable to transfer file between RHEL 7 and ubuntu using mscp HOT 6
- Static build will not resolve localhost HOT 5
- No match for method server host key algo HOT 2
- Add support to limit bandwidth HOT 4
- Proxyjump HOT 2
- Issue with resume transfer HOT 6
- Password needs to entered multiple times when resuming from a transfer HOT 2
- Characters in shell are invisible(?) after entering the passwords from resumed transfer HOT 2
- support for zstd compression HOT 1
- CLI documentation / usability / default values HOT 4
- Provide windows build HOT 1
- Prebuilt binary has very very very poor performance HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mscp.