backuppc / rsync-bpc Goto Github PK
View Code? Open in Web Editor NEWRsync-bpc is a customized version of rsync that is used as part of BackupPC
License: GNU General Public License v3.0
Rsync-bpc is a customized version of rsync that is used as part of BackupPC
License: GNU General Public License v3.0
WHAT IS RSYNC-BPC? ------------------ Rsync-bpc is a customized version of rsync that is used as part of BackupPC, an open source backup system. The main change to rsync is adding a shim layer (in the subdirectory backuppc, and in bpc_sysCalls.c) that emulates the system calls for accessing the file system so that rsync can directly read/write files in BackupPC's format. Rsync-bpc is fully line-compatible with vanilla rsync, so it can talk to rsync servers and clients. Rsync-bpc serves no purpose outside of BackupPC. WHAT IS RSYNC? -------------- Rsync is a fast and extraordinarily versatile file copying tool for both remote and local files. Rsync uses a delta-transfer algorithm which provides a very fast method for bringing remote files into sync. It does this by sending just the differences in the files across the link, without requiring that both sets of files are present at one of the ends of the link beforehand. At first glance this may seem impossible because the calculation of diffs between two files normally requires local access to both files. A technical report describing the rsync algorithm is included with this package. USAGE ----- Basically you use rsync just like scp, but rsync has many additional options. To get a complete list of supported options type: rsync --help See the manpage for more detailed information. SETUP ----- Rsync normally uses ssh or rsh for communication with remote systems. It does not need to be setuid and requires no special privileges for installation. You must, however, have a working ssh or rsh system. Using ssh is recommended for its security features. Alternatively, rsync can run in `daemon' mode, listening on a socket. This is generally used for public file distribution, although authentication and access control are available. To install rsync, first run the "configure" script. This will create a Makefile and config.h appropriate for your system. Then type "make". Note that on some systems you will have to force configure not to use gcc because gcc may not support some features (such as 64 bit file offsets) that your system may support. Set the environment variable CC to the name of your native compiler before running configure in this case. Once built put a copy of rsync in your search path on the local and remote systems (or use "make install"). That's it! RSYNC DAEMONS ------------- Rsync can also talk to "rsync daemons" which can provide anonymous or authenticated rsync. See the rsyncd.conf(5) man page for details on how to setup an rsync daemon. See the rsync(1) man page for info on how to connect to an rsync daemon. WEB SITE -------- The main rsync web site is here: http://rsync.samba.org/ You'll find a FAQ list, downloads, resources, HTML versions of the manpages, etc. MAILING LISTS ------------- There is a mailing list for the discussion of rsync and its applications that is open to anyone to join. New releases are announced on this list, and there is also an announcement-only mailing list for those that want official announcements. See the mailing-list page for full details: http://rsync.samba.org/lists.html BUG REPORTS ----------- To visit this web page for full the details on bug reporting: http://rsync.samba.org/bugzilla.html That page contains links to the current bug list, and information on how to report a bug well. You might also like to try searching the Internet for the error message you've received, or looking in the mailing list archives at: http://mail-archive.com/[email protected]/ To send a bug report, follow the instructions on the bug-tracking page of the web site. Alternately, email your bug report to [email protected] . GIT REPOSITORY -------------- If you want to get the very latest version of rsync direct from the source code repository then you can use git: git clone git://git.samba.org/rsync.git See the download page for full details on all the ways to grab the source, including nightly tar files, web-browsing of the git repository, etc.: http://rsync.samba.org/download.html COPYRIGHT --------- Rsync was originally written by Andrew Tridgell and is currently maintained by Wayne Davison. It has been improved by many developers from around the world. Rsync may be used, modified and redistributed only under the terms of the GNU General Public License, found in the file COPYING in this distribution, or at: http://www.fsf.org/licenses/gpl.html AVAILABILITY ------------ The main web site for rsync is http://rsync.samba.org/ The main ftp site is ftp://rsync.samba.org/pub/rsync/ This is also available as rsync://rsync.samba.org/rsyncftp/
To my understanding, https://github.com/backuppc/rsync-bpc/blob/master/bpc_sysCalls.c#L784
has 2 reasons for rewriting a file
a) the file was removed from the pool
b) the file was marked for deletion and can be re-used.
I suggest to also rewrite if the pool file is zero-length. However, I am not able to understand the source code well enough to judge if chains / handling md5 collisions might play a role here.
Related to issues mentioned in backuppc/backuppc#412
Thanks,
Raoul
Looks like rsync_bpc grew to a very large size and was killed:
kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
kernel: [19879] 40001 19879 16685899 13895036 32541 2745607 0 rsync_bpc
rsync-bpc-3.1.3.0-1.el7.x86_64
I had inconsistent "prior too pooling and compression" stats since using BackupPC v4, as stated in backuppc/backuppc#207
I tried to manually rerun a full sync of a host presenting inconsistent stats (107GB reported by BackupPC for first full backup when host only uses 12GB and has no sparse file.
Here's the command used and the summary:
sudo -u backuppc /usr/local/bin/rsync_bpc --bpc-top-dir /mnt/bkp/pool-01/BackupPC/ --bpc-host-name $H --bpc-share-name / --bpc-bkup-num 0 --bpc-bkup-comp 3 --bpc-bkup-prevnum -1 --bpc-bkup-prevcomp -1 --bpc-bkup-inode0 2 --bpc-attrib-new --bpc-log-level 1 -e /usr/bin/ssh\ -q\ -x\ -l\ root\ --rsync-path=nice\ -n19\ rsync --super --recursive --protect-args --numeric-ids --perms --owner --group -D --times --links --hard-links --delete --delete-excluded --partial --log-format=log:\ %o\ %i\ %B\ %8U,%8G\ %9l\ %f%L --stats --checksum --timeout=72000 --exclude=/proc --exclude=/sys --exclude=/dev --exclude=.extraswap $H:/ /
...
Number of files: 166617
Number of files transferred: 138364
Total file size: 115539848767 bytes
Total transferred file size: 115343522111 bytes
Literal data: 138758723 bytes
Matched data: 0 bytes
File list size: 5963242
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 35428177
Total bytes received: 170458046
sent 35428177 bytes received 170458046 bytes 112969.12 bytes/sec
total size is 115539848767 speedup is 561.18
Killed by signal 1.
DoneGen: 0 errors, 6701 filesExist, 196326656 sizeExist, 127932697 sizeExistComp, 145181 filesTotal, 115539848767 sizeTotal, 0 filesNew, 0 sizeNew, 0 sizeNewComp, 332784 inode
Here are the real metrics on the client host:
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/rootfs 18G 10G 7.6G 57% /
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.9G 4.0K 1.9G 1% /dev/shm
tmpfs 1.9G 193M 1.7G 11% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/vda1 497M 254M 243M 52% /boot
tmpfs 380M 0 380M 0% /run/user/0
$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/rootfs 18317312 160391 18156921 1% /
devtmpfs 482698 408 482290 1% /dev
tmpfs 485251 2 485249 1% /dev/shm
tmpfs 485251 514 484737 1% /run
tmpfs 485251 13 485238 1% /sys/fs/cgroup
/dev/vda1 512000 351 511649 1% /boot
tmpfs 485251 1 485250 1% /run/user/0
As we can seen, rsync_bpc
reports 107GB of data, while the server really only use 12GB.
I also noticed something strange that could explain the stats, the backup lasted way too long compared to other hosts with the same amount of data.
I suspect there could be (a large amount of) garbage / extra data transfered, that used far more bandwidth, which would explain why it was so long (about 20 times more than others, with only 2 time more data in volume).
If I configure rsync-bpc with --disable-acl-support, ACL is still enabled in build process leading to #27
Doing sed -i 's,lib/sysacls.o,,' Makefile* in source lead to a successful build.
Furthermore, backuppc is still working after this change.
I have been trying to get rsyncd transfer working with one of our client machines, but the transfer never manages to write any data to the disk and continues eternally. Unfortunately, I cannot choose freely the Xfer method (I have sudo rights, but it's a CentOS 7 virtual server with many of our university, which has many limitations concerning policies).
After figuring out that there is rsync-bpc, I tried to compile it on Ubuntu 18.04, but get the following message:
gcc -I. -I. -I./zlib -I./popt -g -O2 -DHAVE_CONFIG_H -Wall -W -c lib/sysacls.c -o lib/sysacls.o
lib/sysacls.c:2761:2: error: #error No ACL functions defined for this platform!
#error No ACL functions defined for this platform!
^
Makefile:65: recipe for target 'lib/sysacls.o' failed
make: *** [lib/sysacls.o] Error 1
Any idea?
Hi rsync bpc is not build for debian. for the last release, the beta works ok
the error on make is.
gcc -I. -I. -I./zlib -I./popt -g -O2 -DHAVE_CONFIG_H -Wall -W -c lib/sysacls.c -o lib/sysacls.o
lib/sysacls.c:2761:2: error: #error No ACL functions defined for this platform!
#error No ACL functions defined for this platform!
^~~~~
make: *** [Makefile:65: lib/sysacls.o] Error 1
file has vanished: "/usr/local/lib/ffsroot/fusr/flocal/flib/attrib"
I am quite sure a file (or several files) has really vanished during the backup, but the logged file name is wrong. The reported file and the directory never existed on both client host and BackupPC server.
Contents of file /var/db/BackupPC/pc/mx_fsroot/XferLOG.2723, modified 2020-10-13 01:35:30 (Extracting only Errors)
XferLOG file /var/db/BackupPC/pc/mx_fsroot/XferLOG.2723 created 2020-10-13 01:17:25
>>>skipped<<<
Running: /usr/local/bin/rsync_bpc --bpc-top-dir /var/db/BackupPC --bpc-host-name mx_fsroot --bpc-share-name fsroot --bpc-bkup-num 2723 --bpc-bkup-comp 0 --bpc-bkup-prevnum -1 --bpc-bkup-prevcomp -1 --bpc-bkup-inode0 555084 --bpc-log-level 1 --bpc-attrib-new --super --recursive --protect-args --numeric-ids --perms --owner --group -D --times --links --hard-links --delete --delete-excluded --partial --log-format=log:\ %o\ %i\ %B\ %8U,%8G\ %9l\ %f%L --stats --checksum --timeout=72000 --password-file=/var/db/BackupPC/pc/mx_fsroot/.rsyncdpw98269 --exclude=/cdrom --exclude=/dev --exclude=/media --exclude=/mnt --exclude=/proc --exclude=/smb --exclude=/tmp --exclude=/usr/obj --exclude=/usr/ports --exclude=/usr/src --exclude=/var/db/portsnap --exclude=/var/tmp --exclude=/vmail --exclude=/coreland backuppc@mx::fsroot /
full backup started for directory fsroot
Xfer PIDs are now 98289
This is the rsync child about to exec /usr/local/bin/rsync_bpc
Xfer PIDs are now 98289,98290
xferPids 98289,98290
[ skipped 408 lines ]
file has vanished: "/usr/local/lib/ffsroot/fusr/flocal/flib/attrib"
[ skipped 649 lines ]
Done: 0 errors, 194 filesExist, 116904421 sizeExist, 116904421 sizeExistComp, 0 filesTotal, 0 sizeTotal, 123 filesNew, 467075837 sizeNew, 467075837 sizeNewComp, 555229 inode
>>>skipped<<<
DoneGen: 0 errors, 1 filesExist, 32 sizeExist, 32 sizeExistComp, 67806 filesTotal, 4981734188 sizeTotal, 17 filesNew, 554 sizeNew, 554 sizeNewComp, 555234 inode
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1685) [generator=3.1.3.0]
rsync_bpc exited with benign status 24 (6144)
/rsync-3.1.3
$ make
gcc -I. -I. -I./zlib -I./popt -g -O2 -DHAVE_CONFIG_H -Wall -W -Wno-unused-parameter -c acls.c -o acls.o
In file included from acls.c:23:
lib/sysacls.h:118:20: error: array type has incomplete element type ‘struct acl’
118 | struct acl acl[1];
| ^~~
acls.c: In function ‘unpack_smb_acl’:
lib/sysacls.h:108:33: error: ‘USER_OBJ’ undeclared (first use in this function); did you mean ‘ACL_USER_OBJ’?
108 | #define SMB_ACL_USER_OBJ USER_OBJ
| ^~~~~~~~
acls.c:286:22: note: in expansion of macro ‘SMB_ACL_USER_OBJ’
286 | case SMB_ACL_USER_OBJ:
| ^~~~~~~~~~~~~~~~
lib/sysacls.h:108:33: note: each undeclared identifier is reported only once for each function it appears in
108 | #define SMB_ACL_USER_OBJ USER_OBJ
| ^~~~~~~~
acls.c:286:22: note: in expansion of macro ‘SMB_ACL_USER_OBJ’
286 | case SMB_ACL_USER_OBJ:
| ^~~~~~~~~~~~~~~~
and so on... What do I do? I need to install this version?
When I run ./configure
, I get the following error:
Configure found the following issues:
- You need python3 and either the cmarkgfm OR commonmark python3 lib in order
to build manpages based on the git source (manpages are included in the
official release tar files).
even though I have python3
and commonmark
installed (similarly if I install cmarkgrfm
).
It seems though that the problem is that the rsync
source directory is missing the exectuable md2man
that is called as follows in configure.ac
:
md2man_out=`"$srcdir/md2man" --test "$srcdir/rsync-ssl.1.md" 2>&1
To get config
to work I needed to add --disable-md2man
to the command line.
Perhaps that should be changed in the config code...
master is based on rsync-3.1.3 ... Anno 2018
3.1.2 is based on rsync-3.1.2 ... Anno 2015
3.0.9 is based on rsync-3.0.9 ... Anno 2011
i like to go with the latest stable Version. but i dont like to have an 11 years old rsync
why is 3.0.9 the latest stable and not 3.1.2 ?
Is there any reason not to use 3.1.2 ?
thx flece
I have not find backuppc chat on web like discord or other, so I try to ask here.
sorry for my simple question:
I are talking about doing backup on the same backuppc host (the server)
On backuppc 3.0 server I use for localhost backup to install rsyncd daemon and configure some share;
and all work fine.
Now on backuppc +4.0 I have to use rsync-bpc.
Is rsync-bpc comunicating with rsyncd daemon or I cannot use daemon way on the server?
Can I install rsync-bpc and install rsyncd daemon on the server and use it like on pair (server, client)
(server backuppc rsync-bpc <---> client rsyncd daemon share) pair?
forgive me I have do this question because I have not well understand how rsyncd daemon work and how is use by backuppc.
best regards,
Leonardo
2761 | #error No ACL functions defined for this platform!
| ^~~~~
make: *** [Makefile:65: lib/sysacls.o] Error 1
OK so there are no ACLs defined. How would you suggest the user corrects that please?
I'm running Ubuntu 21.10 with an existing SAMBA server supporting MacOS shares and Time Machine.
Hi,
we got the following error when trying to backup a file which has 2.049 GiB size, i.e. slightly over 2 GiB:
Running: /usr/libexec/backuppc-rsync/rsync_bpc --bpc-top-dir /var/lib/backuppc --bpc-host-name […] --bpc-share-name /home --bpc-bkup-num 520 --bpc-bkup-comp 3 --bpc-bkup-prevnum 519 --bpc-bkup-prevcomp 3 --bpc-bkup-inode0 1270854 --bpc-log-level 1 --bpc-attrib-new -e /usr/bin/ssh\ -l\ root --rsync-path=/usr/bin/rsync --super --recursive --protect-args --numeric-ids --perms --owner --group -D --times --links --hard-links --delete --delete-excluded --one-file-system --partial --log-format=log:\ %o\ %i\ %B\ %8U,%8G\ %9l\ %f%L --stats --checksum --timeout=72000 vcc-prd-01:/home/ /
full backup started for directory /home
Xfer PIDs are now 26685
This is the rsync child about to exec /usr/libexec/backuppc-rsync/rsync_bpc
Xfer PIDs are now 26685,26687
xferPids 26685,26687
same recv >fc........ rw-r--r-- 841, 600 0 […]
rsync_bpc: write failed on "[…]": File too large (27)
Done: 0 errors, 1 filesExist, 1342177280 sizeExist, 1341849014 sizeExistComp, 0 filesTotal, 0 sizeTotal, 0 filesNew, 0 sizeNew, 0 sizeNewComp, 1270855 inode
rsync error: error in file IO (code 11) at receiver.c(449) [receiver=3.1.3.0]
same recv >fcst...... rw-r--r-- 0, 0 2200378428 […]
DoneGen: 0 errors, 0 filesExist, 0 sizeExist, 0 sizeExistComp, 457 filesTotal, 9774730 sizeTotal, 0 filesNew, 0 sizeNew, 0 sizeNewComp, 1270864 inode
rsync_bpc exited with fatal status 11 (2816) (rsync error: error in file IO (code 11) at receiver.c(449) [receiver=3.1.3.0])
Xfer PIDs are now
Got fatal error during xfer (rsync error: error in file IO (code 11) at receiver.c(449) [receiver=3.1.3.0])
Backup aborted (rsync error: error in file IO (code 11) at receiver.c(449) [receiver=3.1.3.0])
This issue is usually only present in file system related limitations. But since rsync_bpc
(as I understand) doesn't write to any real file system, this is probably related to the BackupPC-specific backend.
In any case, the file system of /var/lib/backuppc/
in this case is XFS:
# xfs_info /dev/mapper/datavg0-bkpc
meta-data=/dev/mapper/datavg0-bkpc isize=512 agcount=274, agsize=8388608 blks
= sectsz=4096 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1 bigtime=0
data = bsize=4096 blocks=2290676736, imaxpct=5
= sunit=64 swidth=64 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=131072, version=2
= sectsz=4096 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
# df -hT /var/lib/backuppc/
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/datavg0-bkpc xfs 8.6T 6.5T 2.2T 76% /var/lib/backuppc
Both, the BackupPC server as well as the to be backed up host are running on Debian GNU/Linux 11 Bullseye, amd64 (x86_64) architecture.
Some more information on the used version:
# /usr/libexec/backuppc-rsync/rsync_bpc --version
rsync_bpc version 3.1.3.0 protocol version 31
Copyright (C) 1996-2018 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
append, ACLs, xattrs, iconv, symtimes, no prealloc
rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you
are welcome to redistribute it under certain conditions. See the GNU
General Public Licence for details.
It would be great if beta release had prerelease
flag set, so that https://api.github.com/repos/backuppc/rsync-bpc/releases/latest returned latest stable version (3.1.2.2 instead of 3.1.3beta1). This way we could avoid beta version during automatic deployment.
Hello, I've been asked to review rsync-bpcas part of an Ubuntu Main Inclusion Request; it's not a full audit, just a quick look. Part of this includes running the project through Coverity. I've reviewed some of the results and while there's many false positives due to the way global storage is used, I think it's found several real results.
I didn't see any SECURITY*
files in the repo, and the README
doesn't mention any security contacts. I don't think these issues are particularly sensitive and they may not even be part of rsync-bpc's threat model but I often lack imagination when it comes to attacks. I could post things to github issues, or email them to you, as you wish, please let me know.
Or, you could compile rsync-bpc in Coverity's free service and get all the results. I'm pretty sure their free service has the nice interface to explore results. Marking all the false positives might be a bit of work, but I think in the long run it would pay dividends by reporting bugs before users do.
Thanks
i.e.
Thanks,
Raoul
rsync-bpc fails backup of PC after upgrading to Ubuntu 22.04 LTS (Jammy Jellyfish)
backuppc log for windows backup:
Error: Got fatal error during xfer (rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.3.0]) in the backuppc log.
/usr/local/bin/rsync_bpc --version
rsync_bpc version 3.1.3.0 protocol version 31
Copyright (C) 1996-2018 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
append, ACLs, xattrs, iconv, symtimes, no prealloc
rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you
are welcome to redistribute it under certain conditions. See the GNU
General Public Licence for details.
This is the error I am getting:
` rsync_bpc 3.1.3.beta0 configuration successful
[root@BackupPC rsync-bpc]# cd rsync-bpc/^C
[root@BackupPC rsync-bpc]# make
perl ./mkproto.pl ./*.c ./lib/compat.c
gcc -std=gnu99 -I. -I. -I./zlib -I./popt -g -O2 -DHAVE_CONFIG_H -Wall -W -c flist.c -o flist.o
gcc -std=gnu99 -I. -I. -I./zlib -I./popt -g -O2 -DHAVE_CONFIG_H -Wall -W -c rsync.c -o rsync.o
gcc -std=gnu99 -I. -I. -I./zlib -I./popt -g -O2 -DHAVE_CONFIG_H -Wall -W -c generator.c -o generator.o
gcc -std=gnu99 -I. -I. -I./zlib -I./popt -g -O2 -DHAVE_CONFIG_H -Wall -W -c receiver.c -o receiver.o
gcc -std=gnu99 -I. -I. -I./zlib -I./popt -g -O2 -DHAVE_CONFIG_H -Wall -W -c cleanup.c -o cleanup.o
gcc -std=gnu99 -I. -I. -I./zlib -I./popt -g -O2 -DHAVE_CONFIG_H -Wall -W -c sender.c -o sender.o
gcc -std=gnu99 -I. -I. -I./zlib -I./popt -g -O2 -DHAVE_CONFIG_H -Wall -W -c exclude.c -o exclude.o
gcc -std=gnu99 -I. -I. -I./zlib -I./popt -g -O2 -DHAVE_CONFIG_H -Wall -W -c util.c -o util.o
gcc -std=gnu99 -I. -I. -I./zlib -I./popt -g -O2 -DHAVE_CONFIG_H -Wall -W -c util2.c -o util2.o
gcc -std=gnu99 -I. -I. -I./zlib -I./popt -g -O2 -DHAVE_CONFIG_H -Wall -W -c main.c -o main.o
gcc -std=gnu99 -I. -I. -I./zlib -I./popt -g -O2 -DHAVE_CONFIG_H -Wall -W -c checksum.c -o checksum.o
gcc -std=gnu99 -I. -I. -I./zlib -I./popt -g -O2 -DHAVE_CONFIG_H -Wall -W -c match.c -o match.o
gcc -std=gnu99 -I. -I. -I./zlib -I./popt -g -O2 -DHAVE_CONFIG_H -Wall -W -c syscall.c -o syscall.o
syscall.c: In function ‘do_punch_hole’:
syscall.c:341: error: ‘FALLOC_FL_PUNCH_HOLE’ undeclared (first use in this function)
syscall.c:341: error: (Each undeclared identifier is reported only once
syscall.c:341: error: for each function it appears in.)
syscall.c:341: error: ‘FALLOC_FL_KEEP_SIZE’ undeclared (first use in this function)
make: *** [syscall.o] Error 1`
-=============-
My kernel version: Linux 2.6.32-754.9.1.el6.x86_64
Centos 6.10 64 bit minimal install
Hello,
We have a lot of servers what we are backing up via backuppc and now we have installed new backuppc system on centos 8 and noticed that on only few servers we have issues when backup stops and freezes and then gives error:
rsync_bpc: connection unexpectedly closed (42360 bytes received so far) [receiver]
Done: 0 errors, 0 filesExist, 0 sizeExist, 0 sizeExistComp, 0 filesTotal, 0 sizeTotal, 0 filesNew, 0 sizeNew, 0 sizeNewComp, 6735 inode
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [receiver=3.1.3.0]
[receiver] _exit_cleanup(code=12, file=io.c, line=226): about to call exit(12)
rsync_bpc: connection unexpectedly closed (51093 bytes received so far) [generator]
DoneGen: 0 errors, 0 filesExist, 0 sizeExist, 0 sizeExistComp, 355 filesTotal, 15231605 sizeTotal, 0 filesNew, 0 sizeNew, 0 sizeNewComp, 6734 inode
rsync error: unexplained error (code 255) at io.c(226) [generator=3.1.3.0]
[generator] _exit_cleanup(code=12, file=io.c, line=226): about to call exit(255)
So i have started backup from cli with command /usr/share/BackupPC/bin/BackupPC_dump -f -vvv target_host and collected rsync_bpc at what rsync_bpc stucked:
/usr/bin/rsync_bpc -vvv --bpc-top-dir /data/BackupPC/ --bpc-host-name target_host --bpc-share-name /var/lib/mysql --bpc-bkup-num 1 --bpc-bkup-comp 3 --bpc-bkup-prevnum -1 --bpc-bkup-prevcomp -1 --bpc-bkup-inode0 6734 --bpc-log-level 1 --bpc-attrib-new -e /usr/bin/ssh\ -p\ 11111\ -q\ -x\ -l\ root --rsync-path=/usr/bin/rsync --super --recursive --protect-args --numeric-ids --perms --owner --group -D --times --links --hard-links --delete --delete-excluded --partial --log-format=log:\ %o\ %i\ %B\ %8U,%8G\ %9l\ %f%L --stats --checksum --timeout=72000 target_host:/var/lib/mysql/ /
And then started rcync process with this command and rsync process stucked at (Please keep in mind that this is not first directory for what backuppc was runing on same server, but last one and as i noticed backuppc for each directory generate rsync command. Also x@0024waits_global_by_latency.frm,374 is the last file from folder):
recv_generator(sys/x@0024wait_classes_global_by_avg_latency.frm,370)
recv_generator(sys/x@0024wait_classes_global_by_latency.frm,371)
recv_generator(sys/x@0024waits_by_host_by_latency.frm,372)
recv_generator(sys/x@0024waits_by_user_by_latency.frm,373)
recv_generator(sys/x@0024waits_global_by_latency.frm,374)
generate_files phase=1
rsync_bpc: connection unexpectedly closed (42360 bytes received so far) [receiver]
Done: 0 errors, 0 filesExist, 0 sizeExist, 0 sizeExistComp, 0 filesTotal, 0 sizeTotal, 0 filesNew, 0 sizeNew, 0 sizeNewComp, 6735 inode
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [receiver=3.1.3.0]
[receiver] _exit_cleanup(code=12, file=io.c, line=226): about to call exit(12)
rsync_bpc: connection unexpectedly closed (51093 bytes received so far) [generator]
DoneGen: 0 errors, 0 filesExist, 0 sizeExist, 0 sizeExistComp, 355 filesTotal, 15231605 sizeTotal, 0 filesNew, 0 sizeNew, 0 sizeNewComp, 6734 inode
rsync error: unexplained error (code 255) at io.c(226) [generator=3.1.3.0]
[generator] _exit_cleanup(code=12, file=io.c, line=226): about to call exit(255)
Second server what has same issue:
bash-4.4$ /usr/share/BackupPC/bin/BackupPC_dump -f -vvv target2_host
Backup type: type = full, needs_full = 1, needs_incr = 1, lastFullTime = 0, opts{f} = 1, opts{i} = , opts{F} =
cmdSystemOrEval: about to system /bin/echo -c 1 target2_host
cmdSystemOrEval: about to system /bin/echo -c 1 target2_host
CheckHostAlive: can't extract round-trip time (not fatal)
CheckHostAlive: ran '/bin/echo -c 1 target2_host'; returning 0
XferLOG file /data/BackupPC//pc/target2_host/XferLOG.0.z created 2021-02-17 15:18:18
Backup prep: type = full, case = 1, inPlace = 1, doDuplicate = 0, newBkupNum = 0, newBkupIdx = 0, lastBkupNum = , lastBkupIdx = (FillCycle = 0, noFillCnt = )
bpc_progress_state backup share "/var/lib"
Running: /usr/bin/rsync_bpc --bpc-top-dir /data/BackupPC/ --bpc-host-name target2_host --bpc-share-name /var/lib --bpc-bkup-num 0 --bpc-bkup-comp 3 --bpc-bkup-prevnum -1 --bpc-bkup-prevcomp -1 --bpc-bkup-inode0 2 --bpc-log-level 1 --bpc-attrib-new -e /usr/bin/ssh\ -p\ 11111\ -q\ -x\ -l\ root --rsync-path=/usr/bin/rsync --super --recursive --protect-args --numeric-ids --perms --owner --group -D --times --links --hard-links --delete --delete-excluded --partial --log-format=log:\ %o\ %i\ %B\ %8U,%8G\ %9l\ %f%L --stats --checksum --timeout=72000 --exclude=/proc --exclude=/tmp --exclude=/var/spool/exim --exclude=/var/spool/pmta --exclude=/var/spool/postfix --exclude=*.log --exclude=*.log.* --exclude=*.pid --exclude=sess_* --exclude=*.avi --exclude=*.deb --exclude=*.dmp target2_host:/var/lib/ /
full backup started for directory /var/lib
started full dump, share=/var/lib
Xfer PIDs are now 20910
xferPids 20910
This is the rsync child about to exec /usr/bin/rsync_bpc
cmdExecOrEval: about to exec /usr/bin/rsync_bpc --bpc-top-dir /data/BackupPC/ --bpc-host-name target2+host --bpc-share-name /var/lib --bpc-bkup-num 0 --bpc-bkup-comp 3 --bpc-bkup-prevnum -1 --bpc-bkup-prevcomp -1 --bpc-bkup-inode0 2 --bpc-log-level 1 --bpc-attrib-new -e /usr/bin/ssh\ -p\ 11111\ -q\ -x\ -l\ root --rsync-path=/usr/bin/rsync --super --recursive --protect-args --numeric-ids --perms --owner --group -D --times --links --hard-links --delete --delete-excluded --partial --log-format=log:\ %o\ %i\ %B\ %8U,%8G\ %9l\ %f%L --stats --checksum --timeout=72000 --exclude=/proc --exclude=/tmp --exclude=/var/spool/exim --exclude=/var/spool/pmta --exclude=/var/spool/postfix --exclude=*.log --exclude=*.log.* --exclude=*.pid --exclude=sess_* --exclude=*.avi --exclude=*.deb --exclude=*.dmp target2_host:/var/lib/ /
Xfer PIDs are now 20910,20913
xferPids 20910,20913
xferPids 20910,20913
new recv cd+++++++++ rwx------ 0, 0 4096 .
new recv cd+++++++++ rwxr-xr-x 0, 0 4096 1052039
bpc_progress_fileCnt 1
bpc_progress_fileCnt 21
bpc_progress_fileCnt 41
bpc_progress_fileCnt 61
bpc_progress_fileCnt 81
bpc_progress_fileCnt 101
bpc_progress_fileCnt 121
bpc_progress_fileCnt 141
same recv >f..tp..... rw-r--r-- 0, 0 0 1052039/fastboot
rsync_bpc: connection unexpectedly closed (38533 bytes received so far) [receiver]
Done: 0 errors, 0 filesExist, 0 sizeExist, 0 sizeExistComp, 0 filesTotal, 0 sizeTotal, 0 filesNew, 0 sizeNew, 0 sizeNewComp, 7 inode
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [receiver=3.1.3.0]
rsync_bpc: connection unexpectedly closed (54016 bytes received so far) [generator]
DoneGen: 0 errors, 159 filesExist, 3935 sizeExist, 4941 sizeExistComp, 160 filesTotal, 3935 sizeTotal, 0 filesNew, 0 sizeNew, 0 sizeNewComp, 2146 inode
rsync error: unexplained error (code 255) at io.c(226) [generator=3.1.3.0]
rsync_bpc exited with fatal status 255 (65280) (rsync error: unexplained error (code 255) at io.c(226) [generator=3.1.3.0])
Xfer PIDs are now
xferPids
Got fatal error during xfer (rsync error: unexplained error (code 255) at io.c(226) [generator=3.1.3.0])
cmdSystemOrEval: about to system /bin/echo -c 1 target2_host
cmdSystemOrEval: about to system /bin/echo -c 1 target2_host
CheckHostAlive: can't extract round-trip time (not fatal)
CheckHostAlive: ran '/bin/echo -c 1 target2_host'; returning 0
Backup aborted (rsync error: unexplained error (code 255) at io.c(226) [generator=3.1.3.0])
bpc_progress_state fail cleanup
BackupFailCleanup: nFilesTotal = 160, type = full, BackupCase = 1, inPlace = 1, lastBkupNum =
Keeping non-empty backup #0 (/data/BackupPC//pc/target2_host/0)
bpc_progress_state fsck
Running BackupPC_refCountUpdate -h target2_host -f on target2_host
cmdSystemOrEval: about to system /usr/share/BackupPC/bin/BackupPC_refCountUpdate -h target2_host -f
Xfer PIDs are now 20922
xferPids 20922
bpc_progress_state refCnt #0
BackupPC_refCountUpdate: target2_host #0 inodeLast set to 2132 (was 1)
bpc_progress_state cntUpdate #0
bpc_progress_state rename #0
bpc_progress_state sumUpdate
bpc_progress_fileCnt 0/128
bpc_progress_fileCnt 8/128
bpc_progress_fileCnt 16/128
bpc_progress_fileCnt 24/128
bpc_progress_fileCnt 32/128
bpc_progress_fileCnt 40/128
bpc_progress_fileCnt 48/128
bpc_progress_fileCnt 56/128
bpc_progress_fileCnt 64/128
bpc_progress_fileCnt 72/128
bpc_progress_fileCnt 80/128
bpc_progress_fileCnt 88/128
bpc_progress_fileCnt 96/128
bpc_progress_fileCnt 104/128
bpc_progress_fileCnt 112/128
bpc_progress_fileCnt 120/128
bpc_progress_state rename total
BackupPC_refCountUpdate: host target2_host got 0 errors (took 0 secs)
Xfer PIDs are now
xferPids
Finished BackupPC_refCountUpdate (running time: 0 sec)
Xfer PIDs are now
xferPids
dump failed: rsync error: unexplained error (code 255) at io.c(226) [generator=3.1.3.0]
if need to collect any additional data please tell me
rsync-bpc/backuppc/bpc_poolWrite.c
Line 699 in 1b59d68
Hello, the function bpc_poolWrite_copyToPool()
appears to perform repeated read and write operations eight bytes at a time:
nt bpc_poolWrite_copyToPool(bpc_poolWrite_info *info, char *poolPath, char *fileName)
{
int fdRead, fdWrite;
int nRead, nWrite;
if ( (fdWrite = open(poolPath, O_WRONLY | O_CREAT | O_EXCL, 0666)) < 0 ) {
info->errorCnt++;
bpc_logErrf("bpc_poolWrite_copyToPool: can't open/create %s for writing", poolPath);
return -1;
}
if ( (fdRead = open(fileName, O_RDONLY)) < 0 ) {
info->errorCnt++;
bpc_logErrf("bpc_poolWrite_copyToPool: can't open %s for reading", fileName);
return -1;
}
while ( (nRead = read(fdRead, info->buffer, sizeof(info->buffer))) > 0 ) { /* sizeof(pointer) */
uchar *p = info->buffer;
int thisWrite;
nWrite = 0;
while ( nWrite < nRead ) {
do {
thisWrite = write(fdWrite, p, nRead - nWrite);
} while ( thisWrite < 0 && errno == EINTR );
...
(Of course on 32-bit platforms it'd be four bytes at a time, but those are dwindling.)
Depending upon how much data is being copied and how often this function is used, this could be very slow. There's a few possible approaches I can think of off the top of my head:
sizeof
with BPC_POOL_WRITE_BUF_SZ
if the buffer size is well and truly fixedfdopen(3)
on these file descriptors to use stdio's buffered io routinesfopen(3)
on these file names to use stdio's buffered io routines (is mode 0666
correct for the output file? this seems very permissive)Also, if the fdRead = open(fileName, ...);
operation fails, fdWrite
could be leaked.
Thanks
Hi,
I have installed cygwin-rsyncd-3.2.3.1 on my Windows 10 v20H2 and i have some issues with rsync authentication.
I have completed rsyncd.conf and rsyncd.secrets as expected.
I have completed RsyncdUserName et RsyncdPasswd on BackupPC side. Version is : 4.4.0
When i try to backup my computer on the server i have thir error :
This is the rsync child about to exec /usr/local/bin/rsync_bpc
ERROR: password file must not be other-accessible
Done: 0 errors, 0 filesExist, 0 sizeExist, 0 sizeExistComp, 0 filesTotal, 0 sizeTotal, 0 filesNew, 0 sizeNew, 0 sizeNewComp, 2 inode
rsync error: syntax or usage error (code 1) at authenticate.c(196) [Receiver=3.1.3.0]
rsync_bpc exited with fatal status 1 (256) (rsync error: syntax or usage error (code 1) at authenticate.c(196) [Receiver=3.1.3.0])
On the computer in rsyncd.log i have :
2021/03/21 13:36:44 [211] rsyncd version 3.2.4dev-no-msgs2stderr starting, listening on port 873
2021/03/21 13:36:58 [212] name lookup failed for 192.168.1.2: Name or service not known
2021/03/21 13:36:58 [212] connect from UNKNOWN (192.168.1.2)
2021/03/21 13:36:58 [212] rsync allowed access on module users from UNKNOWN (192.168.1.2)
2021/03/21 13:36:58 [212] auth failed on module users from UNKNOWN (192.168.1.2): invalid challenge response
2021/03/21 13:37:36 [211] sent 0 bytes received 0 bytes total size 0
I don't understand the next message because i am on Winwows 10 :
This is the rsync child about to exec /usr/local/bin/rsync_bpc ERROR: password file must not be other-accessible
Any idea ?
Thank you.
Gerald.
Hi,
i get the following error:
backuppc@backup1:~$ /usr/bin/rsync_bpc --bpc-top-dir /backuppc/backups/ --bpc-host-name server123_hourly --bpc-share-name / --bpc-bkup-num 24669 --bpc-bkup-comp 0 --bpc-bkup-prevnum 24668 --bpc-bkup-prevcomp 0 --bpc-bkup-inode0 2147515806 --bpc-log-level 2 --bpc-attrib-new -e /usr/bin/ssh\ -q\ -x\ -l\ backupuser --rsync-path=sudo\ /usr/bin/rsync --super --recursive --protect-args --numeric-ids --perms --owner --group -D --times --links --hard-links --delete --delete-excluded --partial --log-format=log:\ %o\ %i\ %B\ %8U,%8G\ %9l\ %f%L --stats --timeout=72000 --include=/home --include=/home/backupuser --include=/home/backupuser/mysqlbackup_hourly 10.0.2.91:/ /
rsync_bpc: 2147515806: number too large or too small
rsync error: syntax or usage error (code 1) at main.c(1439) [client=3.0.9.15]
The backuppc file of the client with the jobs etc:
backuppc@backup1:~$ cat /backuppc/backups/pc/server123_hourly/backups
21993 full 1696752565 1696752934 50691 12397857860 0 0 4 981148366 0 0 0 0 0 0 981148366 0 1 rsync 04.4.0 1885302250 0 {}
21995 full 1696756156 1696756525 50691 12397834257 0 0 4 981124763 0 0 0 0 0 0 981124763 0 1 rsync 04.4.0 1885529184 0 {}
21998 full 1696759783 1696760150 50691 12397808456 0 0 4 981098962 0 0 0 0 0 0 981098962 0 1 rsync 04.4.0 1885857502 0 {}
21999 full 1696762884 1696763291 50691 12397779098 0 0 4 981069604 0 0 0 0 0 0 981069604 0 1 rsync 04.4.0 1885983052 0 {}
22000 full 1696766557 1696766932 50691 12397761486 0 0 4 981051992 0 0 0 0 0 0 981051992 0 1 rsync 04.4.0 1886108602 0 {}
22001 full 1696770290 1696770692 50691 12397769622 0 0 4 981060128 0 0 0 0 0 0 981060128 0 1 rsync 04.4.0 1886234152 0 {}
22002 full 1696773687 1696774078 50691 12397755007 0 0 4 981045513 0 0 0 0 0 0 981045513 0 1 rsync 04.4.0 1886359702 0 {}
22003 full 1696777348 1696777716 50691 12397768857 0 0 4 981059363 0 0 0 0 0 0 981059363 0 1 rsync 04.4.0 1886485252 0 {}
22004 full 1696780944 1696781315 50688 12397604736 0 0 4 981072050 0 0 0 0 0 0 981072050 0 1 rsync 04.4.0 1886610796 0 {}
22005 full 1696784547 1696784914 50688 12397617027 0 0 4 981084341 0 0 0 0 0 0 981084341 0 1 rsync 04.4.0 1886736340 0 {}
22006 full 1696788212 1696788583 50688 12397623766 0 0 4 981091080 0 0 0 0 0 0 981091080 0 1 rsync 04.4.0 1886861884 0 {}
22007 full 1696791671 1696792040 50688 12397641391 0 0 4 981108705 0 0 0 0 0 0 981108705 0 1 rsync 04.4.0 1886987428 0 {}
22008 full 1696795410 1696795790 50688 12397665100 0 0 4 981132414 0 0 0 0 0 0 981132414 0 1 rsync 04.4.0 1887112972 0 {}
22009 full 1696798872 1696799239 50688 12397686256 0 0 4 981153570 0 0 0 0 0 0 981153570 0 1 rsync 04.4.0 1887238516 0 {}
22010 full 1696802604 1696802968 50688 12397692178 0 0 4 981159492 0 0 0 0 0 0 981159492 0 1 rsync 04.4.0 1887364060 0 {}
22014 full 1696806575 1696806935 50688 12397701642 0 0 4 981168956 0 0 0 0 0 0 981168956 0 1 rsync 04.4.0 1887793738 0 {}
22017 full 1696810215 1696810573 50688 12397714204 0 0 4 981181518 0 0 0 0 0 0 981181518 0 1 rsync 04.4.0 1888122038 0 {}
22019 full 1696813766 1696814148 50688 12398436397 0 0 4 981903711 0 0 0 0 0 0 981903711 0 1 rsync 04.4.0 1888348960 0 {}
22021 full 1696817374 1696817744 50688 12401158822 0 0 4 984626136 0 0 0 0 0 0 984626136 0 1 rsync 04.4.0 1888575882 0 {}
22023 full 1696820989 1696821368 50688 12399860034 0 0 4 983327348 0 0 0 0 0 0 983327348 0 1 rsync 04.4.0 1888802804 0 {}
22025 full 1696824567 1696824933 50688 12398617365 0 0 4 982084679 0 0 0 0 0 0 982084679 0 1 rsync 04.4.0 1889029726 0 {}
22026 full 1696827671 1696828038 50688 12397633313 0 0 4 981100627 0 0 0 0 0 0 981100627 0 1 rsync 04.4.0 1889155270 0 {}
22028 full 1696831777 1696832157 50688 12397609011 0 0 4 981076325 0 0 0 0 0 0 981076325 0 1 rsync 04.4.0 1889382192 0 {}
22030 full 1696835392 1696835782 50689 12397712002 0 0 4 981184430 0 0 0 0 0 0 981184430 0 1 rsync 04.4.0 1889609118 0 {}
24435 incr 1708567201 1708567551 54130 13689493060 54124 12609544901 4 1079948159 0 0 0 0 0 12609544901 1079948159 1 1rsync 1 4.4.0 2142316755 0 {}
24436 incr 1708570801 1708571168 54130 13694010234 54124 12609544901 4 1084465333 0 0 0 0 0 12609544901 1084465333 1 1rsync 1 4.4.0 2142425017 0 {}
24437 incr 1708574401 1708574741 54130 13692738496 54124 12609544901 4 1083193595 0 0 0 0 0 12609544901 1083193595 1 1rsync 1 4.4.0 2142533279 0 {}
24438 incr 1708578001 1708578338 54130 13691327157 54124 12609544901 4 1081782256 0 0 0 0 0 12609544901 1081782256 1 1rsync 1 4.4.0 2142641541 0 {}
24439 incr 1708581601 1708581949 54130 13690073633 54124 12609544901 4 1080528732 0 0 0 0 0 12609544901 1080528732 1 1rsync 1 4.4.0 2142749803 0 {}
24440 incr 1708585201 1708585548 54131 13689447552 54124 12609544901 5 1079902651 0 0 0 0 0 12609544901 1079902651 1 1rsync 1 4.4.0 2142858067 0 {}
24441 incr 1708588801 1708589203 54133 13689566243 54125 12609560629 6 1080005614 0 0 0 0 0 12609560629 1080005614 1 1rsync 1 4.4.0 2142966335 0 {}
24442 incr 1708592400 1708592787 54141 13690396330 54127 12609599371 12 1080796959 0 0 0 0 0 12609599371 1080796959 1 1rsync 1 4.4.0 2143074619 0 {}
24443 incr 1708596001 1708596358 54137 13690372807 54130 12610093775 5 1080279032 0 0 0 0 0 12610093775 1080279032 1 1rsync 1 4.4.0 2143182895 0 {}
24444 incr 1708597801 1708598165 54138 13690591600 54131 12610112401 5 1080479199 0 0 0 0 0 12610112401 1080479199 1 1rsync 1 4.4.0 2143291173 0 {}
24445 incr 1708599601 1708600001 54139 13690652170 54132 12610248423 5 1080403747 0 0 0 0 0 12610248423 1080403747 1 1rsync 1 4.4.0 2143399453 0 {}
24446 incr 1708603631 1708604046 54140 13690745100 54135 12610267407 3 1080477693 0 0 0 0 0 12610267407 1080477693 1 1rsync 1 4.4.0 2143507735 0 {}
24447 incr 1708632001 1708632390 54158 13694912794 54125 12609955690 31 1084957104 0 0 0 0 0 12609955690 1084957104 1 1rsync 1 4.4.0 2143616053 0 {}
24448 incr 1708643161 1708643522 54158 13694929529 54152 12613602352 4 1081327177 0 0 0 0 0 12613602352 1081327177 1 1rsync 1 4.4.0 2143724371 0 {}
24449 incr 1708644601 1708644952 54158 13694938656 54152 12613602352 4 1081336304 0 0 0 0 0 12613602352 1081336304 1 1rsync 1 4.4.0 2143832689 0 {}
24450 incr 1708646401 1708646755 54158 13694918110 54152 12613602352 4 1081315758 0 0 0 0 0 12613602352 1081315758 1 1rsync 1 4.4.0 2143941007 0 {}
24451 incr 1708648519 1708648905 54158 13694915995 54153 12613603739 3 1081312256 0 0 0 0 0 12613603739 1081312256 1 1rsync 1 4.4.0 2144049325 0 {}
24452 incr 1708650002 1708650374 54158 13694923026 54152 12613602352 4 1081320674 0 0 0 0 0 12613602352 1081320674 1 1rsync 1 4.4.0 2144157643 0 {}
24453 incr 1708653601 1708653973 54158 13694982974 54152 12613602352 4 1081380622 0 0 0 0 0 12613602352 1081380622 1 1rsync 1 4.4.0 2144265961 0 {}
24454 incr 1708657201 1708657585 54158 13699445111 54152 12613602352 4 1085842759 0 0 0 0 0 12613602352 1085842759 1 1rsync 1 4.4.0 2144374279 0 {}
24455 incr 1708660801 1708661137 54158 13698129782 54152 12613602352 4 1084527430 0 0 0 0 0 12613602352 1084527430 1 1rsync 1 4.4.0 2144482597 0 {}
24456 incr 1708664401 1708664745 54158 13696688167 54152 12613602352 4 1083085815 0 0 0 0 0 12613602352 1083085815 1 1rsync 1 4.4.0 2144590915 0 {}
24457 incr 1708668001 1708668342 54158 13695502399 54152 12613602352 4 1081900047 0 0 0 0 0 12613602352 1081900047 1 1rsync 1 4.4.0 2144699233 0 {}
24458 incr 1708671601 1708671941 54158 13694938081 54152 12613602352 4 1081335729 0 0 0 0 0 12613602352 1081335729 1 1rsync 1 4.4.0 2144807551 0 {}
24459 incr 1708675201 1708675558 54160 13694998316 54152 12613602352 6 1081395964 0 0 0 0 0 12613602352 1081395964 1 1rsync 1 4.4.0 2144915873 0 {}
24460 incr 1708678800 1708679152 54160 13695109730 54154 12613611778 4 1081497952 0 0 0 0 0 12613611778 1081497952 1 1rsync 1 4.4.0 2145024195 0 {}
24461 incr 1708682401 1708682747 54162 13695231078 54154 12613611778 6 1081619300 0 0 0 0 0 12613611778 1081619300 1 1rsync 1 4.4.0 2145132521 0 {}
24462 incr 1708684201 1708684539 54162 13695315789 54156 12613627578 4 1081688211 0 0 0 0 0 12613627578 1081688211 1 1rsync 1 4.4.0 2145240847 0 {}
24463 incr 1708686001 1708686365 54162 13695386394 54156 12613627578 4 1081758816 0 0 0 0 0 12613627578 1081758816 1 1rsync 1 4.4.0 2145349173 0 {}
24464 incr 1708689601 1708689954 54162 13695455658 54156 12613627578 4 1081828080 0 0 0 0 0 12613627578 1081828080 1 1rsync 1 4.4.0 2145457499 0 {}
24465 incr 1708730015 1708730406 54164 13696147128 54159 12613668023 3 1082479105 0 0 0 0 0 12613668023 1082479105 1 1rsync 1 4.4.0 2145565829 0 {}
24466 incr 1708731001 1708731383 54164 13696149214 54158 12613666637 4 1082482577 0 0 0 0 0 12613666637 1082482577 1 1rsync 1 4.4.0 2145674159 0 {}
24467 incr 1708732801 1708733148 54164 13696121043 54158 12613666637 4 1082454406 0 0 0 0 0 12613666637 1082454406 1 1rsync 1 4.4.0 2145782489 0 {}
24468 incr 1708734601 1708734934 54164 13696127391 54158 12613666637 4 1082460754 0 0 0 0 0 12613666637 1082460754 1 1rsync 1 4.4.0 2145890819 0 {}
24469 incr 1708736401 1708736722 54164 13696127127 54158 12613666637 4 1082460490 0 0 0 0 0 12613666637 1082460490 1 1rsync 1 4.4.0 2145999149 0 {}
24470 incr 1708740001 1708740342 54164 13696189284 54158 12613666637 4 1082522647 0 0 0 0 0 12613666637 1082522647 1 1rsync 1 4.4.0 2146107479 0 {}
24471 incr 1708743601 1708743959 54164 13700699616 54158 12613666637 4 1087032979 0 0 0 0 0 12613666637 1087032979 1 1rsync 1 4.4.0 2146215809 0 {}
24472 incr 1708747201 1708747537 54164 13699372110 54158 12613666637 4 1085705473 0 0 0 0 0 12613666637 1085705473 1 1rsync 1 4.4.0 2146324139 0 {}
24473 incr 1708750800 1708751137 54164 13697947142 54158 12613666637 4 1084280505 0 0 0 0 0 12613666637 1084280505 1 1rsync 1 4.4.0 2146432469 0 {}
24474 incr 1708754400 1708754745 54165 13696745607 54158 12613666637 5 1083078970 0 0 0 0 0 12613666637 1083078970 1 1rsync 1 4.4.0 2146540801 0 {}
24475 incr 1708758001 1708758327 54165 13696158482 54159 12613685041 4 1082473441 0 0 0 0 0 12613685041 1082473441 1 1rsync 1 4.4.0 2146649133 0 {}
24476 incr 1708761601 1708761928 54165 13696130892 54159 12613685041 4 1082445851 0 0 0 0 0 12613685041 1082445851 1 1rsync 1 4.4.0 2146757465 0 {}
24477 incr 1708765201 1708765532 54165 13696120109 54159 12613685041 4 1082435068 0 0 0 0 0 12613685041 1082435068 1 1rsync 1 4.4.0 2146865797 0 {}
24478 incr 1708768801 1708769128 54165 13696105228 54159 12613685041 4 1082420187 0 0 0 0 0 12613685041 1082420187 1 1rsync 1 4.4.0 2146974129 0 {}
24479 incr 1708770601 1708770924 54165 13696138574 54159 12613685041 4 1082453533 0 0 0 0 0 12613685041 1082453533 1 1rsync 1 4.4.0 2147082461 0 {}
24480 incr 1708772401 1708772735 54165 13696115129 54159 12613685041 4 1082430088 0 0 0 0 0 12613685041 1082430088 1 1rsync 1 4.4.0 2147190793 0 {}
24481 incr 1708776338 1708776708 54165 13696083539 54160 12613686428 3 1082397111 0 0 0 0 0 12613686428 1082397111 1 1rsync 1 4.4.0 2147299125 0 {}
24482 incr 1708815961 1708816347 54169 13701322060 54153 12613605630 14 1087716430 0 0 0 0 0 12613605630 1087716430 1 1rsync 1 4.4.0 2147407465 0 {}
24669 incr 1708817401 1708817749 54169 13701327840 54163 12618836274 4 1082491566 0 0 0 0 0 12618836274 1082491566 1 1rsync 1 4.4.0 2147515805 0 {}
24670 partial 1709715600 00 1 rsync 1 4.4.0 2147515805 0
Used version of rsync_bpc:
backuppc@backup1:~$ rsync_bpc --version
rsync_bpc version 3.1.3.0 protocol version 31
Copyright (C) 1996-2018 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
append, ACLs, xattrs, iconv, symtimes, prealloc
....
Also tried version 3.1.3.1 but i get the same error than:
root@backup1:~/install/rsync-bpc# rsync_bpc --version
rsync_bpc version 3.1.3.1 protocol version 31
Copyright (C) 1996-2018 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
append, ACLs, xattrs, iconv, symtimes, prealloc
...
Is it possible to have a downloadable windows build?
Building on the latest version of cygwin64, I'm getting the following error. It looks like the acl structure definition is not found. This is on latest source and updated cygwin distro:
gcc -I. -I. -g -O2 -DHAVE_CONFIG_H -Wall -W -Wno-unused-parameter -c acls.c -o acls.o
In file included from acls.c:23:
lib/sysacls.h:118:20: error: array type has incomplete element type ‘struct acl’
118 | struct acl acl[1];
| ^~~
make: *** [Makefile:65: acls.o] Error 1
$ gcc --version
gcc (GCC) 11.2.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
I recently encountered this issue:
Backup aborted (rsync error: error allocating core memory buffers (code 22) at io.c(820) [generator=3.1.3.0])
I tried to add --max-alloc=2147483648 to rsync options, and I got: rsync_bpc: --max-alloc=2147483648: unknown option
RSYNC_MAX_ALLOC env variable seems to be ignored as well.
I am running newest version of BackupPC.
Hi I am using backuppc 4.0.0.alpha3 with rsync-bpc 3.0.9.3 and 3.0.9.4.
in both cases I keep getting the following error when trying to do full backup of hosts with 570 000 files.
2017-01-05 14:50:00 full backup started for directory /
2017-01-05 16:41:54 Got fatal error during xfer (rsync error: error in rsync protocol data stream (code 12) at io.c(629) [generator=3.0.9.4])
2017-01-05 16:41:59 Backup aborted (rsync error: error in rsync protocol data stream (code 12) at io.c(629) [generator=3.0.9.4])
Initially I was continuing on top of the old v3 backups, but then I removed them hoping the problem lies somewhere between the v3 vs. v4 incompatibility. Unfortunately it did not help and I keep getting the same error somewhere around 180 000 files being processed.
Building rsync_bpc under alpine linux, many warnings are emitted, although the compilation finally succeeds. See the attached file:
warnings.txt
I run four BackupPC systems backing up a handful of clients each and have been getting these errors on two of the hosts for several months:
From host identified as "f-az" on client "db8-az"
incr backup started for directory /
Xfer PIDs are now 5893
This is the rsync child about to exec /usr/bin/rsync_bpc
Xfer PIDs are now 5893,5897
xferPids 5893,5897
[ skipped 83 lines ]
file has vanished: "/usr/lib/.build-id/5e/f%2f/fusr/flib/f.build-id/f5e/attrib"
file has vanished: "/usr/lib/.build-id/b3/f%2f/fusr/flib/f.build-id/fb3/attrib"
file has vanished: "/usr/lib/firmware/3com/f%2f/fusr/flib/ffirmware/f3com/attrib"
file has vanished: "/usr/lib/firmware/advansys/f%2f/fusr/flib/ffirmware/fadvansys/attrib"
file has vanished: "/usr/lib/firmware/amd-ucode/f%2f/fusr/flib/ffirmware/famd-ucode/attrib"
[ skipped 27 lines ]
Done: 0 errors, 6 filesExist, 15685422 sizeExist, 3480429 sizeExistComp, 0 filesTotal, 0 sizeTotal, 83 filesNew, 460150033 sizeNew, 371747412 sizeNewComp, 621521 inode
Number of files: 288,082 (reg: 271,192, dir: 11,347, link: 5,535, special: 8)
Number of created files: 26 (reg: 26)
Number of deleted files: 0
Number of regular files transferred: 95
Total file size: 23,859,454,943 bytes
Total transferred file size: 476,422,270 bytes
Literal data: 91,275,589 bytes
Matched data: 384,559,866 bytes
File list size: 5,199,215
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 562,483
Total bytes received: 98,890,812
sent 562,483 bytes received 98,890,812 bytes 57,338.31 bytes/sec
total size is 23,859,454,943 speedup is 239.91
DoneGen: 0 errors, 0 filesExist, 0 sizeExist, 0 sizeExistComp, 276735 filesTotal, 23859454943 sizeTotal, 0 filesNew, 0 sizeNew, 0 sizeNewComp, 621468 inode
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1685) [generator=3.1.3.0]
rsync_bpc exited with benign status 24 (6144)
That means the client rsync had errors on some files. Please check the XferLOG.
It likely means that rsync's delete cleanup (which deletes files on the backup
server that are no longer on the client) was skipped. You should fix the error(s)
that rsync can run cleanly. You can also specify the --ignore-errors option
which will still do the delete even if there are rsync errors, but do that with caution.
Xfer PIDs are now
incr backup 213 complete, 276735 files, 23859454943 bytes, 7 xferErrs (0 bad files, 0 bad shares, 7 other)
BackupExpire: cntFull = 11, cntIncr = 6, firstFull = 0, firstIncr = 8, oldestIncr = 11.9838657407407, oldestFull = 141.025520833333
Removing filled backup 109
Xfer PIDs are now 11290
BackupPC_backupDelete: removing #109
BackupPC_backupDelete: No prior backup for merge
Xfer PIDs are now 11290,12498
BackupPC_refCountUpdate: host db8-az got 0 errors (took 48 secs)
Xfer PIDs are now 11290
Xfer PIDs are now
Finished BackupPC_backupDelete, status = 0 (running time: 446 sec)
Xfer PIDs are now
BackupExpire: cntFull = 10, cntIncr = 6, firstFull = 0, firstIncr = 7, oldestIncr = 11.9890277777778, oldestFull = 113.03068287037
Running BackupPC_refCountUpdate -h db8-az on db8-az
Xfer PIDs are now 12670
BackupPC_refCountUpdate: host db8-az got 0 errors (took 2 secs)
Xfer PIDs are now
Finished BackupPC_refCountUpdate (running time: 2 sec)
Xfer PIDs are now
From host identified as "f-az" on client "o8-az"
[ skipped 15 lines ]
file has vanished: "/usr/lib/.build-id/5a/f%2f/fusr/flib/f.build-id/f5a/attrib"
file has vanished: "/usr/lib/.build-id/5c/f%2f/fusr/flib/f.build-id/f5c/attrib"
file has vanished: "/usr/lib/.build-id/5d/f%2f/fusr/flib/f.build-id/f5d/attrib"
file has vanished: "/usr/lib/.build-id/5f/f%2f/fusr/flib/f.build-id/f5f/attrib"
file has vanished: "/usr/lib/.build-id/65/f%2f/fusr/flib/f.build-id/f65/attrib"
file has vanished: "/usr/lib/.build-id/c5/f%2f/fusr/flib/f.build-id/fc5/attrib"
[ skipped 26 lines ]
Done: 0 errors, 2 filesExist, 35 sizeExist, 48 sizeExistComp, 0 filesTotal, 0 sizeTotal, 22 filesNew, 806064920 sizeNew, 777075184 sizeNewComp, 189671 inode
Number of files: 84,345 (reg: 68,007, dir: 10,882, link: 5,449, special: 7)
From host identified as "f-ev" on client "fd8-a"
file has vanished: "/etc/alternatives/f%2f/fetc/falternatives/attrib"
[ skipped 104 lines ]
Done: 0 errors, 13 filesExist, 1730988 sizeExist, 1546568 sizeExistComp, 0 filesTotal, 0 sizeTotal, 43 filesNew, 52405157 sizeNew, 18028323 sizeNewComp, 622993 inode
rsync versions on the servers:
[Cent-7:root@f-ev ~]# rpm -qa | grep -ie rsync
rsync-bpc-3.1.3.0-1.el7.x86_64
rsync-3.1.2-10.el7.x86_64
[Cent-7:user@f-az ~]$ rpm -qa | grep -ie rsync
rsync-bpc-3.1.3.0-1.el7.x86_64
rsync-3.1.2-10.el7.x86_64
rsync versions on the clients:
ssh db8-az 'rpm -qa | grep -ie rsync'
rsync-3.1.3-12.el8.x86_64
ssh o8-az 'rpm -qa | grep -ie rsync'
rsync-3.1.3-12.el8.x86_64
ssh fd8-a 'rpm -qa | grep -ie rsync'
rsync-3.1.3-12.el8.x86_64
@craigbarratt - when you suggested enabling verbose logging in issue #18 you said 'the backup seems quite small'. I'm not sure how mine compares so wanted to wait before going to the next step.
Thank you!
while hunting down my problem that permissions are not correct on rsyncd restores,
I came across following error :
while playing around I noticed that the message "BackupPC options are compulsory for running rsync_bpc\n" is not printed.
options.c, Line 1880:
if ( !bpc_top_dir || !bpc_host_name || !bpc_share_name || bpc_backup_num < 0 ) {
snprintf(err_buf, sizeof(err_buf), "BackupPC options are compulsory for running rsync_bpc\n");
exit_cleanup(RERR_SYNTAX);
}
I am not a "C" Programmer, but I guess it should be :
snprintf(err_buf, sizeof err_buf, "BackupPC options are compulsory for running rsync_bpc\n");
Rsync-bpc does not compile on Ubuntu 20.04.1 on Raspberry Pi 4 Model B. Seems to have an ACL problem.
Linux backup 5.4.0-1019-raspi #21-Ubuntu SMP PREEMPT Mon Sep 14 07:20:34 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux
./configure --disable-acl-support
does not help.
...
checking for _acl... no
checking for __acl... no
checking for _facl... no
checking for __facl... no
checking whether to support ACLs... no
...
gcc -I. -I. -g -O2 -DHAVE_CONFIG_H -Wall -W -Wno-unused-parameter -c lib/sysacls.c -o lib/sysacls.o
lib/sysacls.c:2761:2: error: #error No ACL functions defined for this platform!
2761 | #error No ACL functions defined for this platform!
| ^~~~~
make: *** [Makefile:65: lib/sysacls.o] Error 1
It compiles (and works!) after removing "lib/sysacls.o" from LIBOBJ in Makefile.
I just upgraded a V3 systems that I've been using for years. I also installed rsync_bpc. I was able to migrate all the pools to V4 without issue. I did a few backups without problems. Restores work from these new backups. However when I try to restore from one of the old backups, (using the 1. Direct rsync_bpc method) it fails. Using zip and tar method works without issues. This problem seems pretty consistent across all my nodes.
Originally, I was using an epel install of rsync_bpc (3.1.2), then I compiled the latest 3.1.3beta0 and got the same results.
Here is the logs from a simple test to restore a single file /var/www/html/logo.gif.
*** Failed restore from a migrated V3 Backup
Trimming /html from remoteDir -> /var/www/
Wrote source file list to /backup/BackupPC/pc/backupsvr/.rsyncFilesFrom6060: /html/logo.gif
Running: /usr/local/bin/rsync_bpc --bpc-top-dir /backup/BackupPC --bpc-host-name backupsvr --bpc-share-name /var/www --bpc-bkup-num 2612 --bpc-bkup-comp 7 --bpc-bkup-merge 2612/7/3 --bpc-attrib-new --bpc-log-level 1 -e /usr/bin/ssh\ -q\ -x\ -l\ root --rsync-path=/usr/bin/rsync --recursive --super --protect-args --numeric-ids --perms --owner --group -D --times --links --hard-links --delete --partial --log-format=log:\ %o\ %i\ %B\ %8U,%8G\ %9l\ %f%L --stats --files-from=/backup/BackupPC/pc/backupsvr/.rsyncFilesFrom6060 / backupsvr:/var/www/
This is the rsync child about to exec /usr/local/bin/rsync_bpc
rsync_bpc: change_dir "/" failed: No such file or directory (2)
Done: 0 errors, 0 filesExist, 0 sizeExist, 0 sizeExistComp, 0 filesTotal, 0 sizeTotal, 0 filesNew, 0 sizeNew, 0 sizeNewComp, 0 inode
rsync error: errors selecting input/output files, dirs (code 3) at flist.c(2100) [sender=3.1.3.beta0]
rsync_bpc exited with fatal status 3 (768) (rsync error: errors selecting input/output files, dirs (code 3) at flist.c(2100) [sender=3.1.3.beta0])
restore failed: rsync error: errors selecting input/output files, dirs (code 3) at flist.c(2100) [sender=3.1.3.beta0]
*** Successful restore from a recent V4 Backup
Trimming /html from remoteDir -> /var/www/
Wrote source file list to /backup/BackupPC/pc/backupsvr/.rsyncFilesFrom5917: /html/logo.gif
Running: /usr/local/bin/rsync_bpc --bpc-top-dir /backup/BackupPC --bpc-host-name backupsvr --bpc-share-name /var/www --bpc-bkup-num 2634 --bpc-bkup-comp 9 --bpc-bkup-merge 2634/9/4 --bpc-attrib-new --bpc-log-level 1 -e /usr/bin/ssh\ -q\ -x\ -l\ root --rsync-path=/usr/bin/rsync --recursive --super --protect-args --numeric-ids --perms --owner --group -D --times --links --hard-links --delete --partial --log-format=log:\ %o\ %i\ %B\ %8U,%8G\ %9l\ %f%L --stats --files-from=/backup/BackupPC/pc/backupsvr/.rsyncFilesFrom5917 / backupsvr:/var/www/
This is the rsync child about to exec /usr/local/bin/rsync_bpc
same send .d..t...... rwxr-xr-x 0, 0 4096 html
send <f+++++++++ r--r--r-- 501, 501 1394 html/logo.gif
Number of files: 2 (reg: 1, dir: 1)
Number of created files: 1 (reg: 1)
Number of deleted files: 0
Number of regular files transferred: 1
Total file size: 1,394 bytes
Total transferred file size: 1,394 bytes
Literal data: 1,394 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 1,521
Total bytes received: 46
sent 1,521 bytes received 46 bytes 1,044.67 bytes/sec
total size is 1,394 speedup is 0.89
Done: 0 errors, 0 filesExist, 0 sizeExist, 0 sizeExistComp, 1 filesTotal, 1394 sizeTotal, 0 filesNew, 0 sizeNew, 0 sizeNewComp, 0 inode
Hello,
Is it possible to update rsync-bpc to a more recent rsync version (like 3.2.7) ?
I could not understand how current rsync-bpc is created from rsync source code. Is it possible to share knowledge on this point ?
Main goal here is to continue to have backuppc in Debian, but there is a Debian bug for build backuppc-rsync package.
Note: After having seen issue #29 ; and Debian bug 1066438 report , I decided to open this issue.
Feel free to close it and tell what is appropriate way to handle this matter.
After upgrading rsync_bpc to 3.0.9.9 (and also 3.0.9.10) I find a segfault just for one singe Solaris machine. Backup is broken for this machine.
BackupPC is 4.1.5, BackupPC-XS is 0.57.
backuppc log:
XferLOG file /var/lib/backuppc/pc/ubsun02/XferLOG.138.z created 2017-12-06 10:55:53
Backup prep: type = incr, case = 4, inPlace = 0, doDuplicate = 0, newBkupNum = 138, newBkupIdx = 30, lastBkupNum = 137, lastBkupIdx = 29 (FillCycle = 0, noFillCnt = 1)
Running: /usr/local/bin/rsync_bpc --bpc-top-dir /var/lib/backuppc --bpc-host-name ubsun02 --bpc-share-name /etc --bpc-bkup-num 138 --bpc-bkup-comp 3 --bpc-bkup-prevnum 137 --bpc-bkup-prevcomp 3 --bpc-bkup-inode0 225396 --bpc-attrib-new --bpc-log-level 1 -e /usr/bin/ssh\ -oKexAlgorithms=+diffie-hellman-group1-sha1\ -l\ root --rsync-path=/usr/bin/rsync --recursive --numeric-ids --perms --owner --group -D --times --links --hard-links --delete --delete-excluded --one-file-system --partial --log-format=log:\ %o\ %i\ %B\ %8U,%8G\ %9l\ %f%L --stats --timeout=72000 ubsun02:/etc/ /
incr backup started for directory /etc
Xfer PIDs are now 868
This is the rsync child about to exec /usr/local/bin/rsync_bpc
rsync_bpc exited with fatal status 0 (139) (This is the rsync child about to exec /usr/local/bin/rsync_bpc)
syslog:
Dec 6 10:55:54 backup kernel: [525132.318971] rsync_bpc[868]: segfault at 14 ip 0000000000430c47 sp 00007fff081d2ae0 error 4 in rsync_bpc[400000+7c000]
On the solaris machine
/usr/bin/rsync --recursive --numeric-ids --perms --owner --group -D --times --links --hard-links --delete --delete-excluded --one-file-system --partial --log-format=log:\ %o\ %i\ %B\ %8U,%8G\ %9l\ %f%L --stats --timeout=72000 /etc/ /tmp/test
runs without any problem.
Also no problem for any other machine (including some more solaris machines).
BackupPC-4 creates directories under cpool with world-writable permission (no problem in BackupPC-3).
The creation of these directories has been traced to the backuppc/bpc_dirOps .c procedure of rsync-bpc (the configuration variable $Conf{UmaskMode} does not have any influence on the permission of the created directories).
This issue has relative urgency since the Mageia distro is finalising its new Mageia-7 release and would like to include BackupPC 4.3.0 in this release - see https://bugs.mageia.org/show_bug.cgi?id=24441
Sorry for the typo in the titile: create -> created
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.