upa / mscp Goto Github PK

mscp: transfer files over multiple SSH (SFTP) connections

License: GNU General Public License v3.0

CMake 5.52% C 71.75% Python 15.80% Shell 1.37% Dockerfile 5.56%

mscp's Introduction

mscp: multi-threaded scp

mscp, a variant of scp, copies files over multiple SSH (SFTP) connections by multiple threads. It enables transferring (1) multiple files simultaneously and (2) a large file in parallel, reducing the transfer time for a lot of/large files over networks.

You can use mscp like scp, for example:

$ mscp srcfile [email protected]:dstfile

Remote hosts only need to run standard sshd supporting the SFTP subsystem (e.g. openssh-server), and you need to be able to ssh to the hosts as usual. mscp does not require anything else.

mscp-demo.mp4

Major differences from scp on usage:

Remote-to-remote copy is not supported.
-r option is not needed to transfer directories.
Checkpointing for resuming failed transfer is supported.
and any other differences I have not implemented and noticed.

Paper:

Ryo Nakamura and Yohei Kuga. 2023. Multi-threaded scp: Easy and Fast File Transfer over SSH. In Practice and Experience in Advanced Research Computing (PEARC '23). Association for Computing Machinery, New York, NY, USA, 320–323. https://doi.org/10.1145/3569951.3597582

Install

macOS

brew install upa/tap/mscp

Ubuntu

sudo add-apt-repository ppa:upaa/mscp
sudo apt-get install mscp

RHEL-based distributions

sudo dnf copr enable upaaa/mscp
sudo dnf install mscp

Single binary mscp for x86_64 (not optimal performance)

wget https://github.com/upa/mscp/releases/latest/download/mscp.linux.x86_64.static -O /usr/local/bin/mscp
chmod 755 /usr/local/bin/mscp

Build

mscp depends on a patched libssh. The patch introduces asynchronous SFTP Write, which is derived from https://github.com/limes-datentechnik-gmbh/libssh (see Re: SFTP Write async).

We test building mscp on Linux (Ubuntu, Rocky, Alma, and Alpine), macOS, and FreeBSD.

# clone this repository
git clone https://github.com/upa/mscp.git
cd mscp

# prepare patched libssh
git submodule update --init
patch -d libssh -p1 < patch/$(git --git-dir=./libssh/.git describe).patch

# install build dependency
bash ./scripts/install-build-deps.sh

# configure mscp
mkdir build && cd build
cmake ..

# in macOS, you may need OPENSSL_ROOT_DIR for cmake:
# cmake .. -DOPENSSL_ROOT_DIR=$(brew --prefix)/opt/[email protected]

# build
make

# install the mscp binary to CMAKE_INSTALL_PREFIX/bin (usually /usr/local/bin)
make install

Source tar balls (mscp-X.X.X.tar.gz, not Source code) in Releases page contain the patched version of libssh. So you can start from cmake with it.

Documentation

manpage is available.

mscp's People

Contributors

Stargazers

Watchers

Forkers

k-akashi joeking11829 jaehansuperman takase1121 gerbenvoshol dongcai008 budiantoip

mscp's Issues

noclobber option

Hi,

Normally we use rsync and one of the advantages it the resume (already mentioned in another issue) option. Another is the option to not overwrite files if they exist. Would it be possible for you to add a --noclobber option to mscp?

Thanks!

Possible to add Resume support?

Configurable intervals between SSH connection attempts

NGFWs sometimes treat multiple SSH connection attempts at a short period as a brute force attack. For example, palo alto says 20 times in 60 seconds is a brute force attack by default (https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClmpCAC).

So, the interval between SSH connection attempts should be configurable.

Add support to limit bandwidth

Hello,

I would like to run multiple mscp concurrently and control the bandwith limit on each transfer started. mscp is too fast for some links and use case.

Similar to rsync --bwlimit or scp -l

Thanks
Bruno

support for zstd compression

According to the CLI help, currently only zlib compression is supported. There are a multitude of impressive benchmark that show the power and efficiency of zstd over zlib. Acknowledging that mscp is meant for slow/high-latency connections anyway, it would surely benefit from more powerful compression.

A different idea would be to support reading from stdin. Then, the user is free to transparently compress the file on their own.

Issue with resume transfer

Hi,

I encountered this when I tried to resume the transfer: "mscp_checkpoint_load: [checkpoint.c:366:checkpoint_load_chunk] path index 583 not found". Can u help me with this?

Thx!

unable to transfer file between RHEL 7 and ubuntu using mscp

I am trying to transfer file between RHEL 7.9 and Ubuntu 22.04. I am able to transfer file using scp. But when I am using mscp I am getting error message as below
Copy failed: sftp_async_write: sftp_async_write: Out of memory or Permission denied

Please let me know what are the steps required to fix it.

CLI documentation / usability / default values

Just a few minor things concerning the CLI help and some usability / default value stuff. More like a collection of things here.

Concerning the help:

-n has default floor(log(cores)*2)+1, but is that log10 or log2?
-s says it has default 64M, but as a user I cannot give mscp the number with units - I have to specify bytes. I'd like to be able to put something like -s 4M myself.
What are the allowed values for -g, -c, -M?

Concerning usability:

When the values for -s and -S are not divisible by 4096, mscp will give an error and exit. Can't it just round to the next allowed value?
Just a minor thing, but why does mscp not create the directory needed for -W itself?
Also, it would be great if I could just specify a flag and mscp would take care of the checkpoints automatically (maybe by making a properly named temporary folder? unison does just that. I don't know if the rsync way (--partial --append-verify) would work for mscp too. Browsers like chrome just leave unfinished downloads as they are, but they get an extension that signals incompleteness (.crdownload). What I like about all those ideas is that they happen automatically, increasing user friendlyness).
This might be an oversight on my side, but I specified my login name as user@host. What is -l for?

Concercing the default values:

MAX_CHUNK_SIZE is very large. For a file of hundreds of gigabytes, the transfer is not really interruptible with such a large chunk size (it would have to start over). The minimum chunk size is also relatively large. Wouldn't 16MB/32MB be more reasonable? I'm getting good performance with those and have a quite interruptible transmission.
Also, for my use case (a high-bandwidth high-latency connection), the number of connections is much more important that the number of ssh connections. In fact, I get full saturation with just -u 1 (but -n 12). The other way round (-u 12 -n 1), I get just plain rsync/scp performance which is horribly slow. The default values however seem to prefer high ssh connection count but low connection count. That does not really cater to the intended audience.

All in all, thank you for your software that makes my transfers fast again. I like it a lot despite the flood of tickets I reported.

Hpn-ssh?

Hey there, a colleague saw your presentation at PEARC and sent me a link. Have you looked at hpn-ssh in terms of mscp? If not please take a look GitHub.com/rapier1/openssh-portable. I'd be interested in talking to you if you like.

Host runs out of memory resulting in mscp being killed

System configuration:

NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

MSCP stack trace:

execve("/usr/local/bin/mscp", ["mscp", "-n", "1", "-i", "test.pem", "/localcache/", "[email protected]:/tmp/stash"], 0x7fffe1f95470 /* 13 vars */) = 0
mmap(NULL, 680, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f901349a000
arch_prctl(ARCH_SET_FS, 0x7f901349a1e0) = 0
set_tid_address(0x6dc2f0)               = 1265
brk(NULL)                               = 0x11ee000
brk(0x11f0000)                          = 0x11f0000
mmap(0x11ee000, 4096, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x11ee000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9013499000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9013498000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9013497000
pipe([3, 4])                            = 0
mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9013493000
fcntl(4, F_GETFL)                       = 0x1 (flags O_WRONLY)
fcntl(4, F_SETFL, O_WRONLY|O_APPEND)    = 0
ioctl(4, TIOCGWINSZ, 0x7ffd178d4828)    = -1 ENOTTY (Inappropriate ioctl for device)
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9013492000
getuid()                                = 0
open("/etc/passwd", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 5
fcntl(5, F_SETFD, FD_CLOEXEC)           = 0
fcntl(5, F_SETFD, FD_CLOEXEC)           = 0
read(5, "root:x:0:0:root:/root:/bin/bash\n"..., 1024) = 1024
lseek(5, -992, SEEK_CUR)                = 32
close(5)                                = 0
open("/root/.ssh/config", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/etc/ssh/ssh_config", O_RDONLY|O_LARGEFILE) = 5
read(5, "\n# This is the ssh client system"..., 1024) = 1024
open("/etc/ssh/ssh_config.d/*.conf", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
read(5, "#   AddressFamily any\n#   Connec"..., 1024) = 579
read(5, "", 1024)                       = 0
close(5)                                = 0
socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 5
fcntl(5, F_SETFL, O_RDONLY|O_NONBLOCK|O_LARGEFILE) = 0
setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0
connect(5, {sa_family=AF_INET, sin_port=htons(22), sin_addr=inet_addr("172.31.1.70")}, 16) = -1 EINPROGRESS (Operation now in progress)
poll([{fd=5, events=POLLIN|POLLOUT}], 1, 10000) = 1 ([{fd=5, revents=POLLOUT}])
fcntl(5, F_SETFL, O_RDONLY|O_LARGEFILE) = 0
sendto(5, "SSH-2.0-libssh_0.10.4\r\n", 23, MSG_NOSIGNAL, NULL, 0) = 23
poll([{fd=5, events=POLLIN|POLLOUT}], 1, 9999) = 1 ([{fd=5, revents=POLLOUT}])
poll([{fd=5, events=POLLIN}], 1, 9999)  = 1 ([{fd=5, revents=POLLIN}])
mmap(NULL, 28672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f901348b000
recvfrom(5, "SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ub"..., 4096, 0, NULL, NULL) = 41
getpid()                                = 1265
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f901348a000
getrandom("\xb1\x94\x4d\x01\xc4\x57\x2d\x8f\x46\xd9\x97\x3e\x5e\x6d\x5d\xbf\x94\x2a\xb9\x55\x14\x91\x6f\xb8\xb9\x02\x1d\x2d\xef\xa7\xc8\x6c", 32, 0) = 32
getpid()                                = 1265
getpid()                                = 1265
getpid()                                = 1265
getpid()                                = 1265
getpid()                                = 1265
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9013489000
open("/root/.ssh/known_hosts", O_RDONLY|O_LARGEFILE) = 6
read(6, "172.31.1.70 ssh-ed25519 AAAAC3Nz"..., 1024) = 93
read(6, "", 1024)                       = 0
close(6)                                = 0
open("/etc/ssh/ssh_known_hosts", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9013488000
sendto(5, "\0\0\3T\7\24L3\24\272[{\237w\224i]\215\5\364\216\33\0\0\0\361curve2"..., 856, MSG_NOSIGNAL, NULL, 0) = 856
poll([{fd=5, events=POLLIN|POLLOUT}], 1, 9982) = 1 ([{fd=5, revents=POLLIN|POLLOUT}])
recvfrom(5, "\0\0\4\34\n\24>\3310\377\2745\313\354\272+\217\261\270\316\35\246\0\0\0\346curve2"..., 4096, 0, NULL, NULL) = 1056
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9013486000
getpid()                                = 1265
getpid()                                = 1265
sendto(5, "\0\0\0,\6\36\0\0\0 \35\1\354\213\16deA-og\240D(\375T\307\274\311\210\350\207"..., 48, MSG_NOSIGNAL, NULL, 0) = 48
poll([{fd=5, events=POLLIN|POLLOUT}], 1, 9978) = 1 ([{fd=5, revents=POLLOUT}])
poll([{fd=5, events=POLLIN}], 1, 9977)  = 1 ([{fd=5, revents=POLLIN}])
recvfrom(5, "\0\0\0\274\10\37\0\0\0003\0\0\0\vssh-ed25519\0\0\0 \372i\367"..., 4096, 0, NULL, NULL) = 436
sendto(5, "\0\0\0\f\n\25\0\0\0\0\0\0\0\0\0\0", 16, MSG_NOSIGNAL, NULL, 0) = 16
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9013485000
munmap(0x7f9013485000, 4096)            = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9013485000
munmap(0x7f9013485000, 4096)            = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9013485000
munmap(0x7f9013485000, 4096)            = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9013485000
munmap(0x7f9013485000, 4096)            = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9013485000
munmap(0x7f9013485000, 4096)            = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9013485000
munmap(0x7f9013485000, 4096)            = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9013485000
munmap(0x7f9013485000, 4096)            = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9013485000
munmap(0x7f9013485000, 4096)            = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9013485000
munmap(0x7f9013485000, 4096)            = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9013485000
munmap(0x7f9013485000, 4096)            = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9013485000
getpid()                                = 1265
getpid()                                = 1265
poll([{fd=5, events=POLLIN|POLLOUT}], 1, -1) = 1 ([{fd=5, revents=POLLOUT}])
sendto(5, "#\f(\277\23\23J\25^%\277\212\331)M\5\361{\312\311\31\6*Mg]\300\211\251\230^\230"..., 44, MSG_NOSIGNAL, NULL, 0) = 44
poll([{fd=5, events=POLLIN|POLLOUT}], 1, -1) = 1 ([{fd=5, revents=POLLIN|POLLOUT}])
recvfrom(5, "\300~\315(\341i\257\250\v\253\215cO[K\2200\345kZ\210:8\371/\305\263\tGMN<"..., 4096, 0, NULL, NULL) = 44
getpid()                                = 1265
getpid()                                = 1265
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9013484000
munmap(0x7f9013484000, 4096)            = 0
sendto(5, "\325\202v\257,i \303\346\240v\25\336\363\n\245A\347\r85\345\257$\205\204\320\21)\344\277\246"..., 68, MSG_NOSIGNAL, NULL, 0) = 68
poll([{fd=5, events=POLLIN|POLLOUT}], 1, -1) = 1 ([{fd=5, revents=POLLOUT}])
poll([{fd=5, events=POLLIN}], 1, -1)    = 1 ([{fd=5, revents=POLLIN}])
recvfrom(5, "hn\17\246\213>\342Hcx'\220\315\205\373j(\314r$\216g\346_\212\377!\308\317/a"..., 4096, 0, NULL, NULL) = 44
open("test.pem.pub", O_RDONLY|O_LARGEFILE) = 6
fstat(6, {st_mode=S_IFREG|0664, st_size=404, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9013484000
readv(6, [{iov_base="ssh-rsa AAAAB3NzaC1yc2EAAAADAQAB"..., iov_len=403}, {iov_base="\n", iov_len=1024}], 2) = 404
close(6)                                = 0
getpid()                                = 1265
getpid()                                = 1265
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9013483000
munmap(0x7f9013483000, 4096)            = 0
sendto(5, "\n\332\313i|\216g\317\17\222\250\231\354\305RHO\350:8\264\312F\376\351.\320e\313\253%\336"..., 372, MSG_NOSIGNAL, NULL, 0) = 372
poll([{fd=5, events=POLLIN|POLLOUT}], 1, -1) = 1 ([{fd=5, revents=POLLOUT}])
poll([{fd=5, events=POLLIN}], 1, -1)    = 1 ([{fd=5, revents=POLLIN}])
recvfrom(5, "\321\264\262\273B\36o\377\343\263\206\211\207\314Z\341\200\\\3308\251r\212\313E\10v\206\354\245\10\200"..., 4096, 0, NULL, NULL) = 332
open("test.pem", O_RDONLY|O_LARGEFILE)  = 6
fstat(6, {st_mode=S_IFREG|0600, st_size=1679, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9013483000
readv(6, [{iov_base="-----BEGIN RSA PRIVATE KEY-----\n"..., iov_len=1678}, {iov_base="\n", iov_len=1024}], 2) = 1679
close(6)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9013482000
munmap(0x7f9013482000, 4096)            = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9013482000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9013480000
getpid()                                = 1265
getpid()                                = 1265
getpid()                                = 1265
getpid()                                = 1265
getpid()                                = 1265
getpid()                                = 1265
sendto(5, "#*\234?Q\377\206<\347\16`\346]\256Q\3143\247\202L\306]\315(M\3731\224\261#\362+"..., 652, MSG_NOSIGNAL, NULL, 0) = 652
poll([{fd=5, events=POLLIN|POLLOUT}], 1, -1) = 1 ([{fd=5, revents=POLLOUT}])
poll([{fd=5, events=POLLIN}], 1, -1)    = 1 ([{fd=5, revents=POLLIN}])
recvfrom(5, "\253Yf\34\211\204\273\240\6\rW\2719\311\275\277\337\323\203\345q\23\340^\372\1w@", 4096, 0, NULL, NULL) = 28
open("/root/.ssh/known_hosts", O_RDONLY|O_LARGEFILE) = 6
read(6, "172.31.1.70 ssh-ed25519 AAAAC3Nz"..., 1024) = 93
read(6, "", 1024)                       = 0
close(6)                                = 0
getpid()                                = 1265
getpid()                                = 1265
sendto(5, "^\236{Ji;>\323Ztc\246\215\225\27\240\300]X\247k\265\267\5\253\242\315\371\364\255\3074"..., 52, MSG_NOSIGNAL, NULL, 0) = 52
poll([{fd=5, events=POLLIN|POLLOUT}], 1, -1) = 1 ([{fd=5, revents=POLLOUT}])
poll([{fd=5, events=POLLIN}], 1, -1)    = 1 ([{fd=5, revents=POLLIN}])
recvfrom(5, "\266\311\316\30iJ^!Q0Ci}\211\366\"J\247y\254\204\264;3\357\r\27\252\236Zm0"..., 4096, 0, NULL, NULL) = 776
getpid()                                = 1265
getpid()                                = 1265
sendto(5, "L\"\214\250m\346\34\331\232\373Ts\264\6?\v\367\177w\363\377)\233\340Z\364\376\370\2006\3544"..., 36, MSG_NOSIGNAL, NULL, 0) = 36
poll([{fd=5, events=POLLIN|POLLOUT}], 1, -1) = 1 ([{fd=5, revents=POLLOUT}])
poll([{fd=5, events=POLLIN}], 1, -1)    = 1 ([{fd=5, revents=POLLIN}])
recvfrom(5, "\353\225p\367\317\327\250\371w\340}\266G\300\222\6h|8,~\313-\270`-\341Ki=I\0"..., 4096, 0, NULL, NULL) = 192
getpid()                                = 1265
getpid()                                = 1265
sendto(5, "1o\263\244\356\36\376\275\336\363\36\326\2\232\317\301\177\374\305z\276\22Xgg\376\256k\323m \373"..., 52, MSG_NOSIGNAL, NULL, 0) = 52
poll([{fd=5, events=POLLIN|POLLOUT}], 1, -1) = 1 ([{fd=5, revents=POLLOUT}])
poll([{fd=5, events=POLLIN}], 1, -1)    = 1 ([{fd=5, revents=POLLIN}])
recvfrom(5, "Q\340\340\363\303#\364:~\213\216\222A\23vUUl-\16g_\231\333\23^\25\t%\210\21\204"..., 4096, 0, NULL, NULL) = 72
getpid()                                = 1265
getpid()                                = 1265
sendto(5, "`\37~Y\326\313\252\\{7\200\327\1\344\223\215\312p\"\256\365\5\321K\216\335ve\362S#\4"..., 44, MSG_NOSIGNAL, NULL, 0) = 44
poll([{fd=5, events=POLLIN|POLLOUT}], 1, -1) = 1 ([{fd=5, revents=POLLOUT}])
poll([{fd=5, events=POLLIN}], 1, -1)    = 1 ([{fd=5, revents=POLLIN}])
recvfrom(5, "&]>\356.1@\305\320\335\313\23\355\204\255\356%Ny\345$yTq\340\200\256 P\335\325W"..., 4096, 0, NULL, NULL) = 220
getpid()                                = 1265
getpid()                                = 1265
sendto(5, "\232F\375\361k\274u\265\223\256\275*\17\267&\231%\270\333\323;\rKy\367\206%\fC\373\360B"..., 36, MSG_NOSIGNAL, NULL, 0) = 36
rt_sigprocmask(SIG_UNBLOCK, [RT_1 RT_2], NULL, 8) = 0
mmap(NULL, 143360, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f901345d000
mprotect(0x7f901345f000, 135168, PROT_READ|PROT_WRITE) = 0
rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1 RT_2], [], 8) = 0
clone(child_stack=0x7f901347f928, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID|0x400000, parent_tid=[1287], tls=0x7f901347fb30, child_tidptr=0x6dc2f0) = 1287
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
nanosleep({tv_sec=0, tv_nsec=100000}, 0x7ffd178d4880) = 0
nanosleep({tv_sec=0, tv_nsec=100000}, 0x7ffd178d4880) = 0
nanosleep({tv_sec=0, tv_nsec=100000}, 0x7ffd178d4880) = 0
nanosleep({tv_sec=0, tv_nsec=100000}, 0x7ffd178d4880) = 0
nanosleep({tv_sec=0, tv_nsec=100000}, 0x7ffd178d4880) = 0
nanosleep({tv_sec=0, tv_nsec=100000}, 0x7ffd178d4880) = 0
nanosleep({tv_sec=0, tv_nsec=100000}, 0x7ffd178d4880) = 0
nanosleep({tv_sec=0, tv_nsec=100000}, 0x7ffd178d4880) = 0
mmap(NULL, 143360, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f901343a000
mprotect(0x7f901343c000, 135168, PROT_READ|PROT_WRITE) = 0
rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1 RT_2], [], 8) = 0
clone(child_stack=0x7f901345c928, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID|0x400000, parent_tid=[1288], tls=0x7f901345cb30, child_tidptr=0x6dc2f0) = 1288
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigaction(SIGINT, {sa_handler=0x402a40, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x5ea1ae}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
mmap(NULL, 143360, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9013417000
mprotect(0x7f9013419000, 135168, PROT_READ|PROT_WRITE) = 0
rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1 RT_2], [], 8) = 0
clone(child_stack=0x7f9013439928, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID|0x400000, parent_tid=[1289], tls=0x7f9013439b30, child_tidptr=0x6dc2f0) = 1289
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
futex(0x7f901347fb68, FUTEX_WAIT_PRIVATE, 2, NULL) = ?
+++ killed by SIGKILL +++

Password needs to entered multiple times when resuming from a transfer

Hello,

I know that this issue can be mitigated easily by key-based authentication and an ssh-agent.
Still, I found the following inconsistency: For an initial transfer, I only need to provide my password once. For continuing a transfer (with -W), mscp asks for my password for each connection separately.

do not truncate files when open

https://bugzilla.mindrot.org/show_bug.cgi?id=3431

Proxyjump

Hi,

Thx for your great work! Does mscp support proxyjump? For example: scp -r -P 2222 -J user1@addr user2@addr:path path

Thx!

Unable to parse IPv6 addresses

Example usage :

mscp -n 48 test.zip root@[24X2:1fX0:8X00:4Xd::3:0]:/home/ returns the error :
mscp_connect: ssh.c:167:ssh_init_session: failed to connect ssh server: Failed to resolve hostname [2402 (Name does not resolve)

Whereas, the implementation of scp works on the square brackets.

glob support

Support glob for file paths in both local and remote sides.

Reference: remote_glob() for SFTP in openssh https://github.com/openssh/openssh-portable/blob/master/sftp-glob.c#L137-L180

Cisco DUO Support

mscp should support SSH authentication with MFA.

Duo Unix - Two-Factor Authentication for SSH with PAM Support (pam_duo): https://duo.com/docs/duounix

Possibility to provide password through sshpass

Hi, Thanks for writing this software. I performed an initial speed test and it provides an impressive increase versus scp and our normal parallel/rsync solution. For some of our transfers however, we require a user/passwd combination. Therefore we use sshpass. Unfortunately, this did not seem to work. Is it possible to provide the passwd through sshpass?

In the meantime, I tried some other tools and I noticed https://github.com/clarkwang/passh, can provide the passwd but sshpass still fails.

Static build will not resolve localhost

(deck@steamdeck 3DLiveScanner)$ ~/Downloads/Packages/mscp.linux.x86_64.static -P 8022 parker@localhost:/sdcard/Documents/ .
mscp_connect: [ssh.c:185:ssh_init_session] failed to connect ssh server: Connection refused
(255)(deck@steamdeck 3DLiveScanner)$ ssh -p 8022 parker@localhost
parker@localhost's password: 
Welcome to Termux!

I changed over to 127.0.0.1 and it worked.

Provide windows build

a windows build would be cool

No match for method server host key algo

When trying to connect to an embedded device running dropbear sshd, I get a host key algo error.

Here is my ssh config file at ~/.ssh/config

Host satellite-payload1
    HostName 192.168.102.3
    User root
    IdentityFile ~/.ssh/petalinux_dev
    KexAlgorithms +ssh-rsa
    HostKeyAlgorithms +ssh-rsa

Expected behavior:

mscp should respect the ssh config file and use the additional hostkey and kex algorithms and connect.

Actual behavior:

Fails to negotiate ssh algos

mscp -F .ssh/config 100MBfile satellite-payload1:/run/media/mmcblk0p1/
mscp_connect: [ssh.c:185:ssh_init_session] failed to connect ssh server: kex error : no match for method server host key algo: server [ssh-rsa], client [ssh-ed25519,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256,[email protected],[email protected],rsa-sha2-512,rsa-sha2-256]

Characters in shell are invisible(?) after entering the passwords from resumed transfer

I found this alongside #17.
When I'm finished either entering the passwords for the server or the passwords for my key pair and interrupt the transfer with Ctrl+C, the shell (bash in my case) would not print any characters anymore. I could still run commands in the shell, I just cannot see what I'm typing.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.