rook2pawn / node-filecompare Goto Github PK

View Code? Open in Web Editor NEW

3.0 3.0 4.0 24 KB

Asynchronous File Compare

License: Other

JavaScript 100.00%

node-filecompare's Introduction

Project Git Url

filecompare

Asynchronous file compare. Now using native Promises and native BufferTools (alloc and Buffer comparisons)

install

npm install --save filecompare

test

npm test

Example

var fc = require('filecompare');
var cb = function(isEqual) {
  console.log("equal? :" + isEqual);
}
fc(path1,path1,cb);

command line usage

npm install -g filecompare

filecompare a.txt b.txt

Features

perfect for high stress systems
works with binary data
perfect for comparing very large files

Advanced usage

Specify read size and buffer size.

var fc = require('filecompare');
var cb = function(isEqual) {
  console.log("equal? :" + isEqual);
}
const readSize = 4096;
const bufferSize = 8192;
fc(path1,path1,cb,readSize, bufferSize);

Why is this useful?

Bytes are read into a small buffer, then compared.

Each step is independently asynchronous yet only steps forward after confirming buffers are identical.

This means if there is an unforseen process spike from some other processes, the file compare will exscuse itself until CPU load becomes more available. This means you can compare arbitrarily sized multi-gigabyte files all the time without worry about locking up the computer.

LICENSE

MIT

node-filecompare's People

Contributors

Stargazers

Watchers

Forkers

yclod willjones7 bcruzzn rafaelkr

node-filecompare's Issues

Callback called multiple times

Async mocha tests require that done() is called only once. Filecompare calls it twice. The following mocha code:

const filecompare = require('filecompare');

describe('mymodule', function () {
  it('should call once', function (done) {
    console.log('pre');
      filecompare('index.js', 'LICENSE', function (isEqual) {
        console.log('post');
        return done();
      });
  });
});

Produces the following output:

    mymodule
pre
post
    ✓ should call once

post
    1) should call once

1 passing (14ms)
1 failing

1) filterxml cli should call once:
   Error: done() called multiple times

Callback should be called only once!

EMFILE, too many open files

Im using asnyc (https://www.npmjs.org/package/async) - eachSeries to iterate through my files one at a time, and also logging open and close just for sanity. However after a 1021 files, I get the following error.
Error: EMFILE, too many open files

I'm guessing something is not being closed properly by the plugin?

async.eachSeries(self.items, function (value, callback) {
    console.log('open')
    fc(value.originalFile, value.targetFile, function (isEqual) {
        if (!isEqual) {
            self.invalidate(report);
        }
        console.log('close');
        callback();
    });

}, function (err) {
    if (err) {
        console.error(err)
    }
    console.log('done')
});

open
close
open
close
open

Error: EMFILE, too many open files *****************************
at Object.fs.openSync (fs.js:438:18)
    at Object.exports.file (c:\Users\Sam\Desktop\node-dir-compare\node-dir-compare\node-dir-compare\node_modules\filecompare\lib\index.js:5:17)
    at module.exports (c:\Users\Sam\Desktop\node-dir-compare\node-dir-compare\node-dir-compare\node_modules\filecompare\index.js:4:18)
    at c:\Users\Sam\Desktop\node-dir-compare\node-dir-compare\node-dir-compare\pro-compare.js:76:17
    at iterate (c:\Users\Sam\Desktop\node-dir-compare\node-dir-compare\node-dir-compare\node_modules\async\lib\async.js:149:13)
    at c:\Users\Sam\Desktop\node-dir-compare\node-dir-compare\node-dir-compare\node_modules\async\lib\async.js:160:25
    at null._onTimeout (c:\Users\Sam\Desktop\node-dir-compare\node-dir-compare\node-dir-compare\pro-compare.js:82:25)
    at Timer.listOnTimeout [as ontimeout] (timers.js:112:15)

An error caused by the new version of buffertools

Hi @rook2pawn , I got an error when using this repository:

/THE/PATH/TO/filecompare/lib/index.js:32
        var isEqual = s1.equals(s2);
                         ^
TypeError: Object 1
 has no method 'equals'
    at Object.h2 (/THE/PATH/TO/filecompare/lib/index.js:32:26)
    at Object.wrapper [as oncomplete] (fs.js:459:17)

I installed it with npm install, and the version is 0.0.2, as described on the npm site. After some debugging and searching, I found this repository has a dependency buffertools, which has changed its API now. buffertools stop extending Buffer's prototype by default. But we can use another way to do that. If a Buffer needs to use .equals(), it should add require('buffertools').extend(); instead of require('buffertools');.

You can find more details here -> extend Buffer.prototype.

Wish you can update the repository and publish it to npm, thanks.