Giter Club home page Giter Club logo

rsyncwrapper's Introduction

rsyncwrapper

An async wrapper to the rsync command line utility for Node.js. Also available as the Grunt task grunt-rsync.

Release notes

  • 3.0.1
    • Add TypeScript types
  • 3.0.0
    • Node LTS Carbon v8.11.3 now minimum Node version
    • Remove lodash dependency
    • Port tests to Jest
  • 2.0.1 Upgrade to modern lodash ~4.15.0
  • 2.0.0 Escape non-escaped spaces in argument paths #47
  • 1.0.1 Improved error handling in #45. Added a times option for preserving modification times in #44.
  • 1.0.0 Bumped to version 1! Package now considered stable ๐ŸŽ‰ With this comes a minor API change, the module no longer exports an object with an rsync function. Rather the rsync function is exported as the main value of the module. Check the usage section for an example.
  • 0.5.1 Added the sshCmdArgs option to enable additional customisation of the rsh ssh subcommand. See #37 and #38. Made the tests pass on modern versions of Node, see #36 - this package is now tested to work on all versions of Node from 0.10.25 to 5.3.0.
  • 0.4.3 Added the excludeFirst option. See #34.
  • 0.4.2 Add default chmod arguments on Windows to fix NTFS permissions, see #28.
  • 0.4.1 Renamed syncDest and syncDestIgnoreExcl options to the more scary sounding deleteAll and delete options in an effort to avoid potential user data loss due to misconfiguration. Updated docs to use the new option names. The old option names will continue to work.
  • 0.4.0 Reworking the way stdin is passed from the process to the rsync child process to facilitate Windows clients being able to read a passphrase from stdin.
  • 0.3.0 Swapping include/exclude order in the generated rsync command. Includes now come before excludes to faciliate the normal way of excluding file patterns with exceptions in rsync. See #16.
  • 0.2.0 Now launching the rsync command in a shell like child_process.exec does in Node Core. This enables us to use spawn, and avoid exec maxBuffer, while retaining full shell wildcard expansion.
  • 0.1.0 Now using child_process.exec as opposed to child_process.spawn to enable proper shell wildcard expansions in the options.src value. SSH option handling has been improved.
  • 0.0.1 Initial release.

Prerequisites

A reasonably modern version of rsync (>=2.6.9) in your PATH.

Installation

$ npm install rsyncwrapper

Usage

var rsync = require('rsyncwrapper')
rsync(options, [callback])

The callback function gets four arguments (error,stdout,stderr,cmd).

error: An Error object if rsync failed, otherwise null.

stdout: stdout from rsync.

stderr: stderr from rsync.

cmd: The command string that was used to invoke rsync for debugging purposes.

The options argument is an object literal. See below for possible fields.

Options

src [String|Array<String>] *required

Path to src. Can be a single filename, or an array of filenames. Shell wildcard expansion is supported. Examples:

src: "./dist/"
src: ["./dir-a/file1","./dir-b/file2"]
src: "./*.foo"
src: "foo{1,2,3}.txt"
etc.
dest [String] *required

Path to destination. Example, "/var/www/mysite.tld".

ssh [Bool] default: false

Run rsync over ssh. This is false by default. To use this you need to have public/private key passwordless ssh access setup and working between your workstation and your host. If set to true, you should specify ssh host data as part of your src or dest values, e.g. [email protected]:/var/www/site.

Another good approach is to define a host alias in your ssh config and just reference that alias in your rsync options.

port [String]

If your ssh host uses a non standard SSH port then set it here. Example, "1234".

privateKey [String]

To specify an SSH private key other than the default for this host. Example, "~/.ssh/aws.pem"

sshCmdArgs [Array]

Add an array of arbitrary additional args to the rsh ssh subcommand. This can be useful for turning off strict host key checking to avoid typing yes or no when connecting to new hosts. See SSH's clients man page for more. Example, ["-o StrictHostKeyChecking=no"]

recursive [Boolean] default: false

Recurse into directories. This is false by default which means only files in the src root are copied. Equivalent to the --recursive rsync command line flag.

โ— deleteAll [Boolean] default: false

Take care with this option, since misconfiguration could cause data loss. Deletes objects in dest that aren't present in src, also deletes files in dest that have been specifically excluded from transfer. Equivalent to setting both the --delete and --delete-excluded rsync flags.

โ— delete [Boolean] default: false

Take care with this option, since misconfiguration could cause data loss. The same as deleteAll, but without the --delete-excluded behaviour. One use case for using this option could be while syncing a Node app to a server: you want to exclude transferring the local node_modules folder while retaining the remote node_modules folder.

compareMode [String] enum: "checksum"|"sizeOnly"

By default files will be compared by modified date and file size. Set this value to checksum to compare by a 128bit checksum, or sizeOnly to compare only by file size.

include [Array<String>]

Optional array of rsync patterns to include in the transfer, if previously excluded. Include patterns are defined before exclude patterns when building the rsync command.

exclude [Array<String>]

Optional array of rsync patterns to exclude from transfer. Include patterns are defined before exclude patterns when building the rsync command.

excludeFirst [Array<String>]

Optional array of rsync patterns to exclude from transfer. These are defined before the include patterns when building the rsync command. This is useful to exclude specific files or folders that would be included by the include patterns.

dryRun [Boolean] default: false

Buffer verbose information to stdout about the actions rsyncwrapper would take without modifying the filesystem. Equivalent to setting both the --dry-run and --verbose rsync command line flags.

onStdout [Function]

Optional callback function. Called every time rsync outputs to stdout. Use this to print rsync output as it happens, rather than all at the end. Example: function (data) { console.log(data) }.

onStderr [Function]

Optional callback function. Called every time rsync outputs to stderr. Use this to print rsync error output as it happens, rather than all at the end. Example: function (data) { console.log(data) }.

times [Boolean]

Uses the rsync --times flag to transfer modification times along with the files and update them on the remote system.

args [Array<String>]

Array of additional arbitrary rsync command line options and flags.

The above options are provided for convenience and are designed to cover the most common use cases for rsync, they don't necessarily map directly to single rsync arguments with the same names. If you'd like to handcraft your rsync command then just use the src, dest and args options.

For extra information and subtlety relating to rsync options please consult the rsync manpages.

Tests

Basic tests are run on Vows Async BDD via this package's Gruntfile. To test rsyncwrapper clone the repo and ensure that the devDependancies are present. Additionally ensure that Grunt and Vows are installed globally, and then invoke:

$ npm test

Examples

Copy a single file to another location. If the dest folder doesn't exist rsync will do a mkdir and create it. However it will only mkdir one missing directory deep (i.e. not the equivalent of mkdir -p).

rsync(
  {
    src: 'file.txt',
    dest: 'tmp/file.txt',
  },
  function(error, stdout, stderr, cmd) {
    if (error) {
      // failed
      console.log(error.message)
    } else {
      // success
    }
  }
)

Copy the contents of a directory to another folder, while excluding txt files. Note the trailing / on the src folder and the absence of a trailing / on the dest folder - this is the required format when copying the contents of a folder. Again rsync will only mkdir one level deep:

rsync(
  {
    src: 'src-folder/',
    dest: 'dest-folder',
    recursive: true,
    exclude: ['*.txt'],
  },
  function(error, stdout, stderr, cmd) {
    if (error) {
      // failed
      console.log(error.message)
    } else {
      // success
    }
  }
)

Synchronize the contents of a directory on a remote host with the contents of a local directory. The deleteAll option will delete all files on the remote host that either aren't present in the local folder or have been excluded from transfer.

rsync(
  {
    src: 'local-src/',
    dest: '[email protected]:/var/www/remote-dest',
    ssh: true,
    recursive: true,
    deleteAll: true, // Careful, this could cause data loss
  },
  function(error, stdout, stderr, cmd) {
    if (error) {
      // failed
      console.log(error.message)
    } else {
      // success
    }
  }
)

rsyncwrapper's People

Contributors

arch1t3ct avatar beppler avatar bitzl avatar briangreenhill avatar bryanberger avatar j1anb1n avatar jedrichards avatar jonapgar avatar juriejan avatar kohlmannj avatar ksere avatar mrclay avatar nuxlli avatar reidransom avatar tjschuck avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

rsyncwrapper's Issues

Seems to interfere with console colors on Windows

I have noticed that after I run rsyncwrapper, my console colors get "messed up."

capture

It looks like this is happening here. When I change the the stdio option to inherit the colors work perfectly, but then the following child.stdout.on(...) methods error because child.stdout is no longer defined.

I tried changing child.stdout to process.stdout as suggested in the comment here, but that didn't seem to work (I got a cryptic uncaught error exception).

It's a minor thing, but it would be nice to keep the console looking nice. Thanks for the very helpful module, btw.

syncDest behaviour with excluded files

Am I correct in thinking that if I add a folder, say node_modules, to exclude list, the folder itself and the content will not be sent to the receiving end? And if I enable syncDest, the folder should still be not sync'ed?

As far as I understand syncDest is suppose to delete folders that are not on the sender side, but exist on the receiver side. I cite: "Optional boolean, delete objects in dest that aren't present in src." If that is correct, this make me wonder, why, when node_modules exists on both sides, the receiving end keeps deleting it? To put it simply, when I enable syncDest, all of the excluded folders content gets deleted.

After some digging I located the culprit. It's the --delete-excluded option. Now I'm wondering if this is a desired behaviour and the documentation is not clear enough, or if this needs a fix. Simply commenting out args.push("--delete-excluded"); helped me. If this is considered as a bug, the fix is clear. If not, I think an additional option for example syncDestExclude would be a great feature for this module.

cannot provide multiple -f args

because _.union(args) removes duplicate '-f' argument, which makes resulting command line invalid, because rsync allows multiple '-f' arguments

When specifying port cannot also specify user.

The changes to fix the port usage issues reported in .15 seem to have disabled the ability to specify a user as part of the rsync command.

in .15 I was using the following rsync config.

rsync: {
          options: {
              args:["-cvtpg", "--delete-after"],
              exclude:[".git*", ".idea*"],
              recursive: true,
              port: 2222
          },
          dev: {
              options: {
                  src: "./",
                  dest: "[email protected]:/blat/dir"
              }
          }

the code in .19 will only set the port if options.host is also set, and there does not seem to be a place to specify the user. It would be nice to keep the user@host:dir notation as it is much easier to specify as an argument on the command line.

Issue with escaping space - space in src or dest

It seems that the spaces in a path with a space - space are being escaped incorrectly.

eg path "/foo/bar - test.txt"

was being sent to rysnc as: "/foo/bar\ - test.txt"

and not as expected: "/foo/bar\ -\ test.txt"

I added a test and proposed an fix to the escape regex:

#55

SSH command wrapped in quotes breaks on Windows

I know you don't have a Windows box to debug, but hopefully we could work together to figure out what the issue might be.

I'm getting the following error when I try to run the rsync task:

rsync _site/ user@staging-host:/var/www/site --rsh=ssh --recursive --delete --delete-excluded -e "ssh -p 1234"
Error: rsync exited with code 12. rsync: Failed to exec ssh -p 1234: No such file or directory (2)

Both rsync and ssh are in my PATH. Confirmed multiple times :)

rsync  version 3.0.7  protocol version 30

If I take the command that grunt-rsync is trying to run and paste it myself, it no longer occurs (it simply rejects the connection because of the goofy credentials)

I've narrowed it down to line #62 of rsyncwrapper.js

args.push("-e ssh -p " + options.port + "\"");

Wrapping the ssh command in quotes seems to break it on windows. Removing the quotes seems to fix it.

Globbing patterns dosen't work in 'excludes' option

I have rsync-grunt module with the following task:

rsync: {
  'prepare-release': {
    src: '../',
    dest: 'rev',
    exclude: ['node_modules/*', '!node_modules/mymodule'],
    recursive: true,
    syncDest: true
  }
}

I need don't include all subfolders from node_modules except 'mymodule'. But it seems doesn't work. Is it bug?

Rename option `syncDest` to `delete` or similar

Just used this tool for the first time, and the example code I was using had the syncDest option set to true. Sadly I didn't realise in time what this did, as I was syncing to a directory which contained user added content, and thus the command quickly wiped about 100,000 photos uploaded by users.

Not in any way casting blame, but I think a flag with a chance of causing such a huge problem should be renamed so that the end user can't make this kind of mistake as naively.

Feature Request: Progress

Is it possible to add the option to show progress...

It's quite annoying when rsync is just "starting" and you need to wait until it complets, it would be useful to view a progress in the console per file that is uploading, and percentages if possible.

Like

uploading index.html 34kb 100% completed...
uploading mainphoto.jpg 3mb 35% in progress...
etc ...

Allow output for progress info

There's currently no way to see the progress of a transfer. This is a suggestion to add this functionality.

I.e.

rsync -P -v foo.tar.gz /bar/

An option to either print stdout directly from the process or using an EventEmitter with a progress event would be cool, so we know the status of the transfer.

Thanks for the project!

Args Issue

I'm having trouble getting the args parameter to work. I'm trying to output the progress onto the screen by:

rsync({
src:'./',
dest: config.devarea.username + '@' + config.devarea.host + ':' + config.devarea.remoteFilePath + '/',
port: config.devarea.port,
ssh: true,
recursive: true,
excludeFirst: excludes,
include: ['./*/'],
args: [
"--verbose",
"--progress"
]
}, function (error,stdout,stderr,cmd) {

  if(error) {
    reportError(error);
  }

  console.log(cmd);

});

it outputs the rsync command being used and if I copy that and run in on command line it works fine but nothing is printed out to the console when running it through my gulp task

Order of include exclude

I am having trouble using exclude and include. What I want to do is copy everything from the webroot folder except the webroot/scss folder.

I can do it from the commandline like

rsync . user@host:/path/to.files --rsh ssh --recursive --dry-run --verbose --exclude="webroot/scss" --include="webroot/***" --exclude="*"

But I cannot figure out how to do it using rsyncwrapper since include[] always comes before exclude[].
For now I am doing this as a workaround

options: {
    recursive: true,
    src: ".",
    args: ["--exclude=webroot/scss", "--include=webroot/***", "--exclude=*"],
    dryRun: true,
    dest: "/path/to/files",
    host: "[email protected]",
}

Would it make sense to allow something like

excludeFirst=[],
include=[].
exclude=[]

Or is there some clever way to write the include/exclude rules to make it work as is?

I can't install the module

My dev machine runs OSX, I am getting this error on npm install, node v 0.8.9/npm v1.1.0

module.js:340
throw err;
      ^
Error: Cannot find module 'XXXX/node_modules/rsyncwrapper/node_modules/lodash/build/post-install'
at Function.Module._resolveFilename (module.js:338:15)
at Function.Module._load (module.js:280:25)
at Module.runMain (module.js:492:10)
at process.startup.processNextTick.process._tickCallback (node.js:244:9)

Missing doc on noExec option

I've been trying to use the callback functionality to trigger a secondary rsync call when the first rsync has completed. However despite copying the examples exactly the callback wasn't firing.

I looked in the code for the callback and found it wrapped in a conditional relating to a undocumented (as far as I can tell) option call noExec.

https://github.com/jedrichards/rsyncwrapper/blob/master/lib/rsyncwrapper.js#L102

Perhaps I missed something obvious but could you please explain this option and then document it? Much appreciated.

import issue

import * as rsync from 'rsyncwrapper';

This module can only be referenced with ECMAScript imports/exports by turning on the 'esModuleInterop' flag and referencing its default export.

 compareMode?: "checksum" | "sizeOnly";
           ~~~~~~~~~~~
    The expected type comes from property 'compareMode' which is declared here on type 'RsyncOptions'

I have to use const rsync = require('rsyncwrapper'); for now.
Any other import workarounds?

Publish a new version to use lodash 4.15.0

Hi,

I noticed a commit (8e06629) which updated the lodash version to 4.15.0.

Do you mind publishing a new version based on this commit? I'm using lodash 4.15.0 everywhere in my project. It will save me a lot time if you could do kindly do this.

Thanks!
Matthew

maxBuffer exceeded.

I am using the example "Copy the contents of a directory to another folder" on Mac OS X to copy a folder with a lot of binary files including very large .psd files (500mb - 2gb) in size. I get a "maxBuffer exceeded" output when I enabled verbose mode for this package.

I read some documentation about using 'spawn' instead of 'exec' to get rid of this problem. Can you clarify?

"Reverse" rsync

I've been using

https://github.com/jedrichards/grunt-rsync

...to sync an /uploads/ directory from my local to a remote.

However my team have requested a way to reverse this (ie: pulling down remote directory to a local) using rsync.

I know this is possible because I've done it using

rsync -e ssh user@host:~/public_html/wp-content/uploads/ . --recursive -avz --progress

But I cannot get this to work using Grunt. Is this possible with your two grunt rsync scripts?

windows rsync exited with code 11

Hi
I am running window 7, 64bit, i have installed cwRsync - Rsync for Windows, i have rsync in my path, and i am able to run rsync from anywhere.

my setup:
gulp.task('deploy', function() {
rsync({
ssh: true,
src: './public/',
dest: '****@
_._.compute.amazonaws.com:/node/deploytest',
recursive: true,
privateKey: '../docs/
**.pem',
args: ['--verbose']
}, function(error, stdout, stderr, cmd) {
gutil.log(error);
gutil.log(stdout);
gutil.log(stderr);
});
});

I am getting the following error:
[00:40:10] [Error: rsync exited with code 11]
[00:40:10] sending incremental file list
Could not create directory '/home/Fathead/.ssh'.
Failed to add the host to the list of known hosts (/home/Fathead/.ssh/known_hosts).
rsync: mkdir "/node/test33" failed: No such file or directory (2)
rsync error: error in file IO (code 11) at main.c(674) [Receiver=3.1.0]

Any help would be much appreciated.
/Dennis

Specify custom (ssh) port

Hi there,

I'm just having trouble selecting a custom port with e.g.

options = { host: 'me@my_host:555', src: "./src", dest: "~user/dest" };
rsync(options, cb);

This produces something like rsync ./src me@my_host:555:~user/dest --rsh=ssh --, which attempts to connect to the ssh standard port, 22.

The expected behaviour when providing a port number on as part of the host would be to respect the port. :)

One would ordinarily accomplish this with something like rsync -e "ssh -p $destPort" src dest.

Rsync failed when custom port is used

I got this error when I use custom port:

rsync: Failed to exec ssh -p 1222: No such file or directory (2)

If I type the whole rsync command - it is working flowless

rsync /local-folder/ user@example.com:/home/example.com/public_html --rsh=ssh --recursive --delete --dry-run --verbose --exclude=.git* --exclude=.* --exclude=assets/files --exclude=assets/logs/* -e "ssh -p 1222" --checksum -av

is it something related to path ssh is installed?

Using rsync on Windows returns syntax error

This is likely a small use case but I am wondering if there's a solution I just don't know about considering my limited knowledge of *nix commands.

I have defined the issue here:
mattbanks/WordPress-Starter-Theme#10

Running Node.js in command prompt works after installation, thus saving me from having to download another terminal. However, this problem forced me to install Cygwin in hopes that it fixed the problem with no success.

I might not understand this correctly but I'm guessing the problem is that Windows doesn't understand what ./ type directory structure is. I can't prove this to be the case b/c I do not know what to use instead of ./. How can I troubleshoot this to make it work cross-platform? Or could it be that I need to install SSH?

Doesn't escape src option

Setting the ssh setting to true, I noticed it does not escape the value of src

src: 'user@server:/abs_path/Vader 018 (2016).cbr'

even when i tried wrapping src in double quotes ' "<value>" ', I still seem to be getting this error

[Error: rsync exited with code 12]
bash: -c: line 0: syntax error near unexpected token `('
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [Receiver=3.1.1]

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.