perlundq / yajsync Goto Github PK

On MacOS X, I'm trying to run:

rsync ~/tmp/test rsync://127.0.0.1:14415/Uploads

I'm getting back an error and this message on server side:

déc. 28, 2015 7:37:39 PM com.github.perlundq.yajsync.client.YajSyncServer$8 call
GRAVE: 
com.github.perlundq.yajsync.session.RsyncProtocolException: Unsupported protocol version: @RSYNCD: 29
    at com.github.perlundq.yajsync.session.SessionConfig.receivePeerVersion(SessionConfig.java:174)
    at com.github.perlundq.yajsync.session.SessionConfig.exchangeProtocolVersion(SessionConfig.java:93)
    at com.github.perlundq.yajsync.session.ServerSessionConfig.handshake(ServerSessionConfig.java:100)
    at com.github.perlundq.yajsync.RsyncServer.serve(RsyncServer.java:95)
    at com.github.perlundq.yajsync.client.YajSyncServer$8.call(YajSyncServer.java:221)
    at com.github.perlundq.yajsync.client.YajSyncServer$8.call(YajSyncServer.java:198)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Is MacOS supported?

Thanks!

Rsync supports connection and send/receive timeouts according to the following options:

--timeout=SECONDS
This option allows you to set a maximum I/O timeout in seconds. If no data is transferred for the specified time then rsync will exit. The default is 0, which means no timeout.
Exit-Code 30: Timeout in data send/receive

--contimeout=SECONDS
This option allows you to set the amount of time that rsync will wait for its connection to an rsync daemon to succeed. If the timeout is reached, rsync exits with an error.
Exit-Code 35: Timeout waiting for daemon connection

I'd like to discuss the ways how to implement these timeouts being non-familiar with NIO blocking/non-blocking.

In principle I could imagine to change the factories for plain/ssl sockets in the following way:

public class StandardChannelFactory implements ChannelFactory
{
    @Override
    public DuplexByteChannel open(String address, int remotePort)
        throws IOException
    {
        InetSocketAddress socketAddress = new InetSocketAddress(address,
                                                                remotePort);
        // SocketChannel socketChannel = SocketChannel.open(socketAddress);

        SocketChannel socketChannel = SocketChannel.open();
        socketChannel.socket().setSoTimeout(sndRcvTimeout);
        socketChannel.socket().connect(socketAddress, connectionTimeout);

        return new StandardSocketChannel(socketChannel);
    }
}

public class SSLChannel implements DuplexByteChannel
{
...
public static SSLChannel open(String address, int port) throws IOException
    {
        SocketFactory factory = SSLSocketFactory.getDefault();
        // Socket sock = factory.createSocket(address, port);
        InetSocketAddress socketAddress = new InetSocketAddress(address, port);
        Socket sock = factory.createSocket();
        sock.setSoTimeout(sndRcvTimeout);
        sock.connect(socketAddress, connectionTimeout);
        return new SSLChannel((SSLSocket) sock);
    }
...
}

After reading https://technfun.wordpress.com/2009/01/29/networking-in-java-non-blocking-nio-blocking-nio-and-io/ I think at least com.github.perlundq.yajsync.channels.net.StandardSocketChannel might have to be changed so read/write is executed on streams (seems like r/w works on the socket directly right now).

@perlundq: I didn't dig into this yet, may you can give some advice first?

$ java -ea -Dumask=0002 -XX:+UseConcMarkSweepGC -XX:+AggressiveOpts -jar /home/perl/projects/yajsync/build/jar/yajsyncd.jar -vv --config=/home/perl/yajsyncd.conf --port=14415 &
$ java -ea -Dumask=0002 -XX:+UseConcMarkSweepGC -XX:+AggressiveOpts -jar build/jar/yajsync.jar --port=14415 localhost:: &>/dev/null
Feb 17, 2014 12:12:01 AM com.github.perlundq.yajsync.session.RsyncServerSession startSession
FINE: Got connection from /127.0.0.1:49296
Feb 17, 2014 12:12:01 AM com.github.perlundq.yajsync.session.ServerSessionConfig handshake
FINE: sending module listing and exiting
Feb 17, 2014 12:12:01 AM com.github.perlundq.yajsync.ui.YajSyncServer$8 call
FINE: Thread exit status: ERROR

From the API, I could understand that it transfers data in one direction and pushes/uploads data from source to destination.

From the below example illustrated:
java -Dumask=$(umask) -jar yajsync-app/target/yajsync-app-0.9.0-SNAPSHOT-full.jar client --port=14415 -r example localhost::Uploads

it seems that we are uploading data from Client(from example directory) to server(in uploads directory).

Please confirm if this is true ?

As I tested the yajsync , I find it update the whole file to server if the file has been changed on client,
so I am confused if I am missing some argument settings or the yajsync dose not support incremental update.

I was testing yajsync and try to sync a directory between local and remote directory. Basically, I would like to do:
rsync -a /home/temp/test user@remote/home/temp

For rsync, I can use one command to complete this task, with yajsync, do I have to start a client on local, and start a server on remote, in order to do this?

yajsync currently supports a minimal subset of rsync protocol version 30.0, with the additional constraint that the peer must also support rsync safe file lists.

How can I assure my native rsync client supports "safe file lists" ?

$ rsync --version
rsync  version 3.1.0  protocol version 31
Copyright (C) 1996-2013 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, symtimes, prealloc

It would be great to add event listener to client API in order to track rsync progress.
-v console output may be done from listener implementation.

compareTo() in https://github.com/perlundq/yajsync/blob/master/src/main/com/github/perlundq/yajsync/filelist/FileInfo.java#L80 relies on _pathNameBytes (in compareUnixFileNamesBytes(...)). Is there any reason for not using a compare based on _normalizedPath instead (like in equals(...)) ?

Minor priority: com.github.perlundq.yajsync.session.RsyncClientSession and com.github.perlundq.yajsync.session.RsyncLocal could implement/extend a common interface/class to emphasize the mutual extension if local or remote classes are modified.

There are comments like the following that declare why some pathnames have to be saved as String not Path in the current implementation. To be able to support URIs of different FileSystems there should be a dual approach:

• allow URIs locally
• use according String representations for remote processing

Is there anything that could interfere with such a local/remote differentiation?

/**
 * a class for rsync file information, we must keep the path name as a string
 * as opposed to storing the Path directly since the latter may fail
 * (when being the receiver) and we must keep the file list identical to the
 * peer's file list when being receiver
 */
public class FileInfo implements Comparable<FileInfo>

The following code snippet could be an example how to support more file attribute views beyond POSIX and Basic, for e.g. support of S_ISUID, S_ISGID and S_ISVTX and without the necessity to map via int toMode(...):

public static RsyncFileAttributes stat(Path path) throws IOException
{
    if (Files.getFileStore(path).supportsFileAttributeView("unix")) {
        Map<String,Object> attrMap = Files.readAttributes(path, "unix:*"); 
        int mode = (Integer) attrMap.get("mode");
        long size = (Long) attrMap.get("size");
        FileTime modTime = (FileTime) attrMap.get("lastModifiedTime");
        return new RsyncFileAttributes(mode, size, modTime.to(TimeUnit.SECONDS));
    }

...

In case a --port=... is defined in the client YajSyncClient.updateRemotePort(Path, int, RsyncUrls, RsyncUrl) is called and RsyncUrl.toRemotePathName(String, String) is executed once more. In line https://github.com/perlundq/yajsync/blob/master/src/main/com/github/perlundq/yajsync/ui/RsyncUrl.java#L179 an already built pathname is appended to the moduleName again. Supposed there was no path given the target path will be after the remote port update.

In certain circumstances one thread may hang indefinitely if one thread exits prematurely due to an error. E.g. rsync file file/ is supposed to exit with an error, but due to this bug yajsync will also hang forever:

$ echo file > file
$ echo file2 > file2
$ yajsync file file2/
Warning: this software is still unstable and there might be data corruption bugs hiding. So use it only carefully at your own risk.
Oct 07, 2014 10:14:56 AM com.github.perlundq.yajsync.session.Receiver receive
SEVERE: unable to stat /tmp/file2/.: java.nio.file.FileSystemException: /tmp/file2/.: Not a directory
...hangs indefinitely...

This is only a problem for a local running rsync. It is caused by failing to properly close the channel(s) being used by the exiting thread - the peer thread on the other end is never notified that the thread on the other end isn't there anymore.

$ mkdir dir
$ echo file > dir/file
$ yajsync -r dir/ dir.new
Warning: this software is still unstable and there might be data corruption bugs hiding. So use it only carefully at your own risk.
$ ls -l dir dir.new
dir:
total 4
-r--r--r-- 1 perl perl 5 Oct  6 23:26 file

dir.new:
total 4
-rw------- 1 perl perl 5 Oct  6 23:28 file

in this case dir.new/file is expected to have perms 0444

rsync sender sends trailing data when specifying a recursive remote file transfer from yajsync -> rsync daemon and a nonexisting path on the sender side. E.g.

yajsync -rvvv rsync://rsync.kernel.org/pub/no_such_path
FINER: deferring exception raised by task 1/2: java.util.concurrent.ExecutionException: com.github.perlundq.yajsync.session.RsyncProtocolException: Unexpectedly got 2 bytes from peer during connection tear down: [0xff, 0x01]
Mar 23, 2016 11:44:57 PM com.github.perlundq.yajsync.session.RsyncTaskExecutor exec
FINER: waiting for result from task 2/2
Mar 23, 2016 11:44:57 PM com.github.perlundq.yajsync.session.RsyncTaskExecutor exec
FINER: task 2/2 finished OK
Mar 23, 2016 11:44:57 PM com.github.perlundq.yajsync.RsyncClient$FileListing$2 call
FINE: shutting down java.util.concurrent.ThreadPoolExecutor@1c80acf[Running, pool size = 3, active threads = 1, queued tasks = 0, completed tasks = 2]
Exception in thread "main" com.github.perlundq.yajsync.session.RsyncProtocolException: Unexpectedly got 2 bytes from peer during connection tear down: [0xff, 0x01]
at com.github.perlundq.yajsync.session.Receiver.readAllMessagesUntilEOF(Receiver.java:1913)
at com.github.perlundq.yajsync.session.Receiver.call(Receiver.java:368)
at com.github.perlundq.yajsync.session.Receiver.call(Receiver.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

# already running yajsyncd on port 14415
$ rsync rsync://localhost:14415
Downloads      
Uploads        
Downloads2     
$ rsync rsync://localhost:14415/Downloads
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [Receiver=3.0.9]

# yajsyncd error log:
Jan 10, 2014 7:42:35 PM com.github.perlundq.yajsync.ui.YajSyncServer$8 call
SEVERE: 
com.github.perlundq.yajsync.session.RsyncProtocolException: com.github.perlundq.yajsync.util.ArgumentParsingError: d - unknown option
        at com.github.perlundq.yajsync.session.ServerSessionConfig.handshake(ServerSessionConfig.java:123)
        at com.github.perlundq.yajsync.session.RsyncServerSession.startSession(RsyncServerSession.java:155)
        at com.github.perlundq.yajsync.ui.YajSyncServer$8.call(YajSyncServer.java:198)
        at com.github.perlundq.yajsync.ui.YajSyncServer$8.call(YajSyncServer.java:190)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)
Caused by: com.github.perlundq.yajsync.util.ArgumentParsingError: d - unknown option
        at com.github.perlundq.yajsync.util.ArgumentParser.getOptionForName(ArgumentParser.java:309)
        at com.github.perlundq.yajsync.util.ArgumentParser.parse(ArgumentParser.java:129)
        at com.github.perlundq.yajsync.session.ServerSessionConfig.parseArguments(ServerSessionConfig.java:261)
        at com.github.perlundq.yajsync.session.ServerSessionConfig.handshake(ServerSessionConfig.java:118)
        ... 8 more

see options.c:server_options() and options.c:parse_arguments()

xfer_dirs is initialized to -1 and if doing non-recursive module listing it is set to 1...

What's the meaning of
"Message MessageHeader ERROR_XFER length=108 java.nio.HeapByteBuffer[pos=0 lim=108 cap=108]" ?

java.lang.RuntimeException: TODO: not implemented: Sender.handleMessage(Message MessageHeader ERROR_XFER length=108 java.nio.HeapByteBuffer[pos=0 lim=108 cap=108])
at com.github.perlundq.yajsync.session.Sender.handleMessage(Sender.java:211)
at com.github.perlundq.yajsync.channels.TaggedInputChannel.readNextMessage(TaggedInputChannel.java:92)
at com.github.perlundq.yajsync.channels.TaggedInputChannel.readNextAvailable(TaggedInputChannel.java:62)
at com.github.perlundq.yajsync.channels.PrefetchedTaggedInputChannel.ensureMinimumPrefetched(PrefetchedTaggedInputChannel.java:161)
at com.github.perlundq.yajsync.channels.PrefetchedTaggedInputChannel.getByte(PrefetchedTaggedInputChannel.java:85)
at com.github.perlundq.yajsync.channels.IndexDecoderImpl.decodeIndex(IndexDecoderImpl.java:40)
at com.github.perlundq.yajsync.channels.RsyncInChannel.decodeIndex(RsyncInChannel.java:48)
at com.github.perlundq.yajsync.channels.AutoFlushableRsyncDuplexChannel.decodeIndex(AutoFlushableRsyncDuplexChannel.java:60)
at com.github.perlundq.yajsync.session.Sender.sendFiles(Sender.java:272)
at com.github.perlundq.yajsync.session.Sender.send(Sender.java:170)
at com.github.perlundq.yajsync.session.RsyncServerSession$1.call(RsyncServerSession.java:86)
at com.github.perlundq.yajsync.session.RsyncServerSession$1.call(RsyncServerSession.java:79)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

Hi there

Would it be possible to licence the core component as Apache or MIT?

Regards

I did some research about wrapping ReadableByteChannel and
WritableByteChannel by SSL: the common way is to use http://docs.oracle.com/javase/7/docs/api/javax/net/ssl/SSLEngine.html but some developers warned this could be hard stuff ;)

A sample implementation can be found in the package org.apache.tomcat.util.net/ at http://svn.apache.org/repos/asf/tomcat/trunk/java/org/apache/tomcat/util/net/
See esp. SecureNioChannel.java, it is the ssl secured implementation of a NIO channel.

Maybe you could check this to confirm if this should fit in the current yajsync architecture.

Example:

$ stat -c %s file.2G
2147483648
$ yajsync file.2G out
Warning: this software is still unstable and there might be data corruption bugs hiding. So use it only carefully at your own risk.
$ yajsync file.2G out
Warning: this software is still unstable and there might be data corruption bugs hiding. So use it only carefully at your own risk.
$ stat -c %s file.2G+1
2147483649
$ yajsync file.2G+1 out
Warning: this software is still unstable and there might be data corruption bugs hiding. So use it only carefully at your own risk.
$ yajsync file.2G+1 out
Warning: this software is still unstable and there might be data corruption bugs hiding. So use it only carefully at your own risk.
Exception in thread "main" java.lang.IllegalArgumentException: Negative position
        at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:676)
        at com.github.perlundq.yajsync.session.Receiver.matchReplica(Receiver.java:1474)
        at com.github.perlundq.yajsync.session.Receiver.combineDataToFile(Receiver.java:1371)
        at com.github.perlundq.yajsync.session.Receiver.mergeDataFromPeerAndReplica(Receiver.java:1281)
        at com.github.perlundq.yajsync.session.Receiver.matchData(Receiver.java:882)
        at com.github.perlundq.yajsync.session.Receiver.receiveFiles(Receiver.java:747)
        at com.github.perlundq.yajsync.session.Receiver.call(Receiver.java:308)
        at com.github.perlundq.yajsync.session.Receiver.call(Receiver.java:65)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

I tested the server and noticed that the synced directory needs to be predefined in yajsyncd.conf file. Is there a way that I can randomly give the directory that are not defined in the conf file?

Rsync program has the --progress options which can show the progress of the rsync, does current yajsync support this option as well?

The native rsync implementation sends a MSG_DATA message in the last protocol version to keep a connection alive. This message is sent in different i/o scenarios (checks are executed at fixed places in i/o processing) depending on the time when the last packages were sent. The The following comment comes from io.c:

/* Older rsync versions used to send either a MSG_NOOP (protocol 30) or a
 * raw-data-based keep-alive (protocol 29), both of which implied forwarding of
 * the message through the sender.  Since the new timeout method does not need
 * any forwarding, we just send an empty MSG_DATA message, which works with all
 * rsync versions.  This avoids any message forwarding, and leaves the raw-data
 * stream alone (since we can never be quite sure if that stream is in the
 * right state for a keep-alive message). */
void maybe_send_keepalive(time_t now, int flags)

Bug reports like https://bugzilla.samba.org/show_bug.cgi?id=7757 or hints like the following from https://download.samba.org/pub/rsync/rsync.html reveal that the implementation in native rsync is improvable:

--delete-before [...] Deleting before the transfer is helpful if the filesystem is tight for space and removing extraneous files would help to make the transfer possible. However, it does introduce a delay before the start of the transfer, and this delay might cause the transfer to timeout (if --timeout was specified).

Considering MSG_DATA is stable and compatible with different rsync protocol versions I propose to use scheduled threads (ScheduledExecutorService.scheduleAtFixedRate) to send out MSG_DATA packages independently from sender processing. This may result in more packages than required but assures timeouts are not met as long as the sender is running.

Do you see any better way to solve this?

I added usrflo@42425a3 to run the same system tests with a yajsync or rsync client for simplified rsync compatibility checks.

With the following test I experienced unexpected behaviour with the rsync client. These problems do not exist with the yajsync client.

@Test
public void testClientDirCopyDotted() throws IOException
{
    Path src = _tempDir.newFolder().toPath();
    Path dst = Paths.get(src.toString() + ".dst");

    Path srcDir1 = src.resolve("dir");
    Path srcDir2 = src.resolve("dir.2");
    Path srcFile1 = srcDir1.resolve("file1");
    Path srcFile2 = srcDir2.resolve("file2");
    Files.createDirectory(srcDir1);
    Files.createDirectory(srcDir2);
    FileUtil.writeToFiles(7, srcFile1);
    FileUtil.writeToFiles(8, srcFile2);

    Files.createDirectory(dst);

    ReturnStatus status = fileCopy(src, dst, "--recursive");

    assertTrue(status.rc == 0);
}

With srcDir2 set on "dir.2" or "dir-2" the path inclusion check inside
com.github.perlundq.yajsync.filelist.Filelist.SegmentBuilder.add(FileInfo) reports an error:

/tmp/junit2086544845478406680/junit7751461368613446554.dst/junit7751461368613446554/dir should be a path prefix to: /tmp/junit2086544845478406680/junit7751461368613446554.dst/junit7751461368613446554/dir.2/file2

OR

/tmp/junit5154228388051712892/junit3843589390619568255.dst/junit3843589390619568255/dir should be a path prefix to: /tmp/junit5154228388051712892/junit3843589390619568255.dst/junit3843589390619568255/dir-2/file2

With srcDir2 set to other values like "dir2" or "dir_2" everything is ok.

There seems to be a special handling of directory names in rsync? Does this have to be addressed in YajSync? (please be aware I am working on a branch that I last merged in November)

$ echo file > file
$ ln -s file link
$ yajsync file link target/
Warning: this software is still unstable and there might be data corruption bugs hiding. So use it only carefully at your own risk.
$ ls -l target/
total 8
-rw-rw-r-- 1 perl perl 5 Oct  7 19:22 file
-rw-rw-r-- 1 perl perl 5 Oct  7 19:22 link

In this case link should be skipped. This bug is caused by reading contents/attributes of the file the link is referring to, rather than the link itself.

A deadlock or a broken pipe may occur if Generator tries to notify Sender after Sender and Receiver has agreed to go to the stopped connection state.

Generator will try to notify Sender if any of the deferred attribute updates fails, but Sender is gone and is no longer receiving the messages.

1. E.g. a broken pipe occurs when doing remote recursive transfers as an unprivileged user and trying to preserve ownership:

   $ yajsync -ro localhost::etc etc.copy

   ...snip...
   WARNING: received I/O error while applying attributes on etc.copy/libvirt/qemu/networks/autostart: etc.copy/libvirt/qemu/networks/autostart: Operation not permitted
   Mar 21, 2016 4:41:12 PM com.github.perlundq.yajsync.ui.YajSyncClient remoteTransfer
   SEVERE: Error: communication closed with peer:
   com.github.perlundq.yajsync.channels.ChannelException: java.io.IOException: Broken pipe
   at com.github.perlundq.yajsync.channels.BufferedOutputChannel.send(BufferedOutputChannel.java:70)
   at com.github.perlundq.yajsync.channels.BufferedOutputChannel.flush(BufferedOutputChannel.java:79)
   at com.github.perlundq.yajsync.channels.TaggedOutputChannel.flush(TaggedOutputChannel.java:78)
   at com.github.perlundq.yajsync.channels.TaggedOutputChannel.putMessage(TaggedOutputChannel.java:55)
   at com.github.perlundq.yajsync.session.Generator$8.process(Generator.java:1014)
   at com.github.perlundq.yajsync.session.Generator$2.process(Generator.java:417)
   at com.github.perlundq.yajsync.session.Generator.processJobQueueBatched(Generator.java:355)
   at com.github.perlundq.yajsync.session.Generator.call(Generator.java:373)
   at com.github.perlundq.yajsync.session.Generator.call(Generator.java:74)
   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   at java.lang.Thread.run(Thread.java:745)
   Caused by: java.io.IOException: Broken pipe
   at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
   at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
   at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
   at sun.nio.ch.IOUtil.write(IOUtil.java:51)
   at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
   at com.github.perlundq.yajsync.channels.net.StandardSocketChannel.write(StandardSocketChannel.java:62)
   at com.github.perlundq.yajsync.channels.BufferedOutputChannel.send(BufferedOutputChannel.java:60)
   ... 14 more

2. Likewise, a deadlock may occur if doing the same kind of transfer locally (yajsync -ro /etc etc.copy).
   But: A deadlock does not always occur. And it seems like this also should always result in a broken pipe, but it does not (though it uses Java pipes rather than a tcp socket).

Writing a system test for timeout detection I'm ending up with the logging output of the stacktrace below (which is correct and expected). In this test I'm using the YajSyncClient that does not propagate exceptions but returns the result code like the native rsync client is doing. What options do you see to enable exception propagation if YajSyncClient/Server should be testable with annotations like

@test(expected=SocketTimeoutException.class)

?

com.github.perlundq.yajsync.channels.ChannelException: java.net.SocketTimeoutException
    at com.github.perlundq.yajsync.channels.SimpleInputChannel.get(SimpleInputChannel.java:134)
    at com.github.perlundq.yajsync.channels.SimpleInputChannel.getByte(SimpleInputChannel.java:62)
    at com.github.perlundq.yajsync.channels.BufferedDuplexChannel.getByte(BufferedDuplexChannel.java:92)
    at com.github.perlundq.yajsync.channels.AutoFlushableDuplexChannel.getByte(AutoFlushableDuplexChannel.java:41)
    at com.github.perlundq.yajsync.session.SessionConfig.readLine(SessionConfig.java:112)
    at com.github.perlundq.yajsync.session.SessionConfig.receivePeerVersion(SessionConfig.java:167)
    at com.github.perlundq.yajsync.session.SessionConfig.exchangeProtocolVersion(SessionConfig.java:93)
    at com.github.perlundq.yajsync.session.ClientSessionConfig.handshake(ClientSessionConfig.java:83)
    at com.github.perlundq.yajsync.RsyncClient$ModuleListing$1.call(RsyncClient.java:247)
    at com.github.perlundq.yajsync.RsyncClient$ModuleListing$1.call(RsyncClient.java:1)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException
    at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:229)
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
    at com.github.perlundq.yajsync.channels.net.StandardSocketChannel.read(StandardSocketChannel.java:83)
    at com.github.perlundq.yajsync.channels.SimpleInputChannel.get(SimpleInputChannel.java:122)
    ... 13 more

In native rsync, io.c, the sleep_for_bwlimit function is commented as follows:

/* Sleep after writing to limit I/O bandwidth usage.
 *
 * @todo Rather than sleeping after each write, it might be better to
 * use some kind of averaging.  The current algorithm seems to always
 * use a bit less bandwidth than specified, because it doesn't make up
 * for slow periods.  But arguably this is a feature.  In addition, we
 * ought to take the time used to write the data into account.
 *
 * During some phases of big transfers (file FOO is uptodate) this is
 * called with a small bytes_written every time.  As the kernel has to
 * round small waits up to guarantee that we actually wait at least the
 * requested number of microseconds, this can become grossly inaccurate.
 * We therefore keep track of the bytes we've written over time and only
 * sleep when the accumulated delay is at least 1 tenth of a second. */

When adding --bwlimit to yajsync the mentioned improvement of the @todo section above (averaging) should be considered. The token bucket algorithm would fit the requirements of such an improvement.

Taking a look at https://github.com/bbeck/token-bucket I guess this would be an appropriate implementation to simply integrate bandwidth limits. Taking in account you like to keep dependencies low:

• would you propose to re-implement such an algorithm?
• would you propose to re-use existing code but integrate it with its license to the yajsync source code as long as being compatible?
• would you be open for a maven dependency?

The comment of the method FileOps.atomicMove() already tells that this method might fail:

        // this can fail in many ways, maybe we can try to recover some of them?

If the file systems in both path arguments differ the option StandardCopyOption.ATOMIC_MOVE produces an error: java.nio.file.AtomicMoveNotSupportedException: Atomic move between providers is not supported

My proposal to recover this case in https://github.com/perlundq/yajsync/blob/master/src/main/com/github/perlundq/yajsync/util/FileOps.java#L270 :

        if (tempFile.getFileSystem().equals(path.getFileSystem())) {
            Files.move(tempFile, path, StandardCopyOption.ATOMIC_MOVE);
        } else {
            Files.move(tempFile, path);
        }

Please take a look at usrflo@378754a
These classes implement http://rsync.samba.org/ftp/rsync/rsync.html, section FILTER RULES, INCLUDE/EXCLUDE PATTERN RULES and MERGE-FILE FILTER RULES.

I could integrate this filter rule processing in yajsync if you don't have other plans. The integration consists of:

• reading option arguments
• initialize the filter rule configuration classes according to the rsync filter rule syntax
• check the rules on dir/file list processing

Please tell me if you are interested.

Connection is prematurely closed by receiver at connection teardown. Reproduced by:

• start yajsyncd server on port 14415:
  yajsyncd -vvvv --port=14415 --config=yajsyncd.conf
• upload some file using rsync or yajsync:

rsync --port=14415 -r projects/coreutils localhost::upload
rsync: connection unexpectedly closed (15 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [sender=3.1.1pre2]

• yajsyncd output:

...snip...
FINE: Received index -1
Oct 23, 2014 10:33:13 AM com.github.perlundq.yajsync.session.Receiver receiveFiles
FINE: tearing down at phase ConnectionState stopped
Oct 23, 2014 10:33:13 AM com.github.perlundq.yajsync.session.Receiver receive
FINE: Receiver returned 0 errors
Oct 23, 2014 10:33:13 AM com.github.perlundq.yajsync.session.Generator processJobQueueBatched
FINE: (Generator) got 1 job(s)
Oct 23, 2014 10:33:13 AM com.github.perlundq.yajsync.session.Generator processJobQueueBatched
FINE: (Generator) processing sendSegmentDone()
Oct 23, 2014 10:33:13 AM com.github.perlundq.yajsync.session.Generator processJobQueueBatched
FINE: (Generator) awaiting next jobs...
Oct 23, 2014 10:33:13 AM com.github.perlundq.yajsync.session.Generator processJobQueueBatched
FINE: (Generator) got 1 job(s)
Oct 23, 2014 10:33:13 AM com.github.perlundq.yajsync.session.Generator processJobQueueBatched
FINE: (Generator) processing stop()
Oct 23, 2014 10:33:13 AM com.github.perlundq.yajsync.ui.YajSyncServer$7 call
SEVERE: Error: communication closed with peer: java.nio.channels.ClosedChannelException
Oct 23, 2014 10:33:13 AM com.github.perlundq.yajsync.ui.YajSyncServer$7 call
FINE: Thread exit status: ERROR

Consider to add a SortedList implementation like http://stackoverflow.com/a/12638571/1894655 to yajsync and change the LinkedList in https://github.com/perlundq/yajsync/blob/master/src/main/com/github/perlundq/yajsync/filelist/Filelist.java#L42 to a SortedList to reduce O(n) mentioned in https://github.com/perlundq/yajsync/blob/master/src/main/com/github/perlundq/yajsync/filelist/Filelist.java#L90 to O(log n).

cloning latest master as of today and running the tests gives me the following error on Linux x86-64 (Ubuntu 16.04) kernel 4.5.1-040501-generic, Oracle JDK 8 (build 1.8.0_91-b14, 64bit server), Apache Maven 3.3.9:

testConnectionTimeout(com.github.perlundq.yajsync.test.SystemTest): Unexpected exception, expected<java.net.SocketTimeoutException> but was<java.net.SocketException>

Running com.github.perlundq.yajsync.test.SystemTest
drwxrwxr-x           2 2016/06/24 17:27:34 ./
test            a test module
Jun 24, 2016 5:27:35 PM com.github.perlundq.yajsync.internal.session.ServerSessionConfig handshake
WARNING: failed to authenticate 
drwxrwxr-x           2 2016/06/24 17:27:35 ./
drwxrwxr-x           3 2016/06/24 17:27:35 ./
-rw-rw-r--           0 2016/06/24 17:27:35 ./file
Tests run: 32, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.672 sec <<< FAILURE!
testConnectionTimeout(com.github.perlundq.yajsync.test.SystemTest)  Time elapsed: 0.004 sec  <<< ERROR!
java.lang.Exception: Unexpected exception, expected<java.net.SocketTimeoutException> but was<java.net.SocketException>
    at org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:28)
    at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:274)
    at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:268)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketException: Network is unreachable
    at sun.nio.ch.Net.connect0(Native Method)
    at sun.nio.ch.Net.connect(Net.java:454)
    at sun.nio.ch.Net.connect(Net.java:446)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
    at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:102)
    at com.github.perlundq.yajsync.net.StandardSocketChannel.open(StandardSocketChannel.java:53)
    at com.github.perlundq.yajsync.net.StandardChannelFactory.open(StandardChannelFactory.java:27)
    at com.github.perlundq.yajsync.test.SystemTest.testConnectionTimeout(SystemTest.java:1000)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:19)
    ... 4 more

$ yajsync -r /home target
Warning: this software is still unstable and there might be data corruption bugs hiding. So use it only carefully at your own risk.
Exception in thread "main" java.lang.IllegalArgumentException
        at sun.nio.fs.UnixPath.subpath(UnixPath.java:350)
        at sun.nio.fs.UnixPath.subpath(UnixPath.java:43)
        at com.github.perlundq.yajsync.util.PathOps.parentPath(PathOps.java:75)
        at com.github.perlundq.yajsync.util.PathOps.subtractPath(PathOps.java:81)
        at com.github.perlundq.yajsync.session.Sender.getLocalPathOf(Sender.java:1024)
        at com.github.perlundq.yajsync.session.Sender.expand(Sender.java:519)
        at com.github.perlundq.yajsync.session.Sender.expandAndSendSegments(Sender.java:626)
        at com.github.perlundq.yajsync.session.Sender.sendFiles(Sender.java:242)
        at com.github.perlundq.yajsync.session.Sender.send(Sender.java:164)
        at com.github.perlundq.yajsync.session.RsyncLocal$3.call(RsyncLocal.java:156)
        at com.github.perlundq.yajsync.session.RsyncLocal$3.call(RsyncLocal.java:152)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

There is no problem if source has more than one name component, e.g. yajsync -r /home/dir target works OK.

I guess this feature could be added by the modification of https://github.com/perlundq/yajsync/blob/master/src/main/com/github/perlundq/yajsync/session/Generator.java#L858 :

if (isDataModified(fileInfo.attrs(), curAttrs)) {
-->
if (_isForceChecksum || isDataModified(fileInfo.attrs(), curAttrs)) {

Is there anything else to consider?

$ ant
Buildfile: /tmp/yajsync/build.xml

init:
[mkdir] Created dir: /tmp/yajsync/build/classes
[mkdir] Created dir: /tmp/yajsync/build/jar

build:
[echo] YajSync: /tmp/yajsync/build.xml
[javac] Compiling 81 source files to /tmp/yajsync/build/classes
[javac] /tmp/yajsync/src/test/com/github/perlundq/yajsync/session/IntegerCoderTest.java:3: error: package org.junit does not exist
[javac] import static org.junit.Assert.assertEquals;
[javac] ^
...snip...

There is some prepared code for user authentication in ClientSessionConfig.printLinesAndGetReplyStatus and some comment at ServerSessionConfig.setModule that the authentication request should be done depending on the module configuration. As far as I can see the authentication on server side is not implemented yet?

The prepared source code in ClientSessionConfig.printLinesAndGetReplyStatus hashes the user password by MD5.

Some ideas regarding the server-side implementation:

• the current rsync way to save a clear-text password in the secrets-file besides the module configuration is not favourable. I appreciate your choice to send the MD5-hashed password to the server.
• for being able to provide a flexible configuration I'd prefer an optional config syntax like it was used in the postfix mailserver configuration, e.g.

user = mysql:/yajsync/user-sql.cfg
OR
path = mysql:/yajsync/path-sql.cfg
OR
path = ldap:/yajsync/path-ldap.cfg
and so on.

The config-modules (mysql or ldap or text or whatever) would have a generic interface to set (a) the current module (b) the username to select the configuration value from the config-module database.

The implementation of config-modules could be separate (in separate JARs on the classpath) and each with its custom configuration, e.g. a mysql module would refer to a path-sql.cfg that could look like this:
user = mydbuser
password = mySecPas$
hosts = 127.0.0.1
dbname = yajsync
table = userconfig
select_field = path
where_field_module = module
where_field_user = user

Regarding http://linux.die.net/man/5/rsyncd.conf this flexibility would allow to
a) use one module configuration with different paths for different users
b) user specific exclude/include options if once implemented
c) user specific 'allow host' directives
...

• considering a SSL enhancement for transport security I think users could use a server-side truststore optionally for client-authentication. I'd personally would not focus on that (maybe this part could be left to default JSSE security options).

What about these proposals?

Hi,

How stable is the code (both API and reliability) for general-purpose use? I note it hasn't had any tags yet.

I'm interested in using for a couple of projects (including https://github.com/PhilLello/rsync-maven-wagon).

Phil