I have been trying to get browserify-incremental to work in browserify-rails by making it switch between the browserify and browserifyinc based on an option.

With my current configuration, this command runs:

/myapp/node_modules/.bin/browserifyinc -d --extension=.jsx --fast --cachefile="/myapp/tmp/browserify-rails/browserifyinc-cache.json" -o "/myapp/tmp/browserify-rails/output20150106-13751-ay9tra" -

Where --extension=.jsx and --fast are added by my Rails application's config. The main JavaScript file comes from stdin, and the command is run in the correct directory so that all requires are tracked properly.

This is the corresponding browserify command, when the switch for browserifyinc is disabled:

/myapp/node_modules/.bin/browserify -d --extension=.jsx --fast -o "/myapp/tmp/browserify-rails/output20150106-14638-165eoyg" -

Everything seems the same, however the output of browserifyinc fails with an error when it runs in my browser:

Uncaught ReferenceError: global is not defined (modernizr.js:4)

The error seems weird, it comes from a part of the code that seems completely unrelated (I use modernizr with browserify-shim).

However, when inspecting the output of the two commands with a piece of code as simple as console.log("Test"), I get radically different outputs:

When run with browserifyinc (the above command plus < testfile.js), I get:

(function e(t,n,r){function s(o,u){if(!n[o]){if(!t[o]){var a=typeof require=="function"&&require;if(!u&&a)return a(o,!0);if(i)return i(o,!0);var f=new Error("Cannot find module '"+o+"'");throw f.code="MODULE_NOT_FOUND",f}var l=n[o]={exports:{}};t[o][0].call(l.exports,function(e){var n=t[o][1][e];return s(n?n:e)},l,l.exports,e,t,n,r)}return n[o].exports}var i=typeof require=="function"&&require;for(var o=0;o<r.length;o++)s(r[o]);return s})({"/Users/gabriele/Work/Impraise/impraise-web/app/assets/javascripts/_stream_0.js":[function(require,module,exports){
console.log("Test");

},{}]},{},["/Users/gabriele/Work/Impraise/impraise-web/app/assets/javascripts/_stream_0.js"])

Running the above browserify command, though, I get a radically different and longer output (it seems to contain Buffer, due to the --fast option, but I suspect the differences don't stop there): Gist of both files.

Lastly, the browserify node_module that is installed in my Rails app which browserify-rails uses is version 6.3.4. browserify-incremental depends on ^6.1.0, and going into its node_modules I don't see a browserify directory, so it seems to be relying on the same one the application itself uses if I'm not mistaken.

Could you help me shed some light on this issue? I will be experimenting more options, but if you know anything I think it would help me a lot.

Although I tried removing the file and it didn't make any difference on browserify's output, here are the contents of my package.json. The only thing that might have some bearing (since the error I mentioned above seems to have some correlation with browserify-shim) is the "browserify" key, which specifies 6to5ify and browserify-shim as transforms. Maybe they are not being picked up.

{
  "devDependencies": {
    "6to5ify": "*",
    "browserify": "~> 6.3",
    "browserify-incremental": "^1.0.0",
    "browserify-shim": "^3.8.0",
    "eventemitter2": "^0.4.14",
    "fuse.js": "~> 1.2",
    "humps": "https://codeload.github.com/impraise/humps/tar.gz/master",
    "jquery": "~> 2.1",
    "lodash": ">= 2.4.1",
    "moment": "^2.8.4",
    "react": "^0.12.1",
    "superagent": "~> 0.21",
    "underscore.string": "^2.4.0"
  },
  "license": "MIT",
  "engines": {
    "node": ">= 0.11"
  },
  "browserify": {
    "transform": [
      "6to5ify",
      "browserify-shim"
    ]
  },
  "browserify-shim": {
    "stripe": "global:Stripe",
    "__mixpanel": "global:mixpanel",
    "__youtube": "global:YT",
    "jquery-ujs": {
      "depends": [
        "jquery:jQuery"
      ]
    },
    "jquery.payment": {
      "depends": [
        "jquery:jQuery"
      ]
    },
    "modernizr": {
      "exports": "Modernizr"
    },
    "uservoice": {
      "exports": "UserVoice"
    }
  },
  "browser": {
    "jquery-ujs": "./bower_components/jquery-ujs/src/rails.js",
    "jquery.payment": "./bower_components/jquery.payment/lib/jquery.payment.js",
    "modernizr": "./vendor/assets/javascripts/modernizr.js",
    "uservoice": "./vendor/assets/javascripts/uservoice.js"
  },
  "dependencies": {
    "async": "^0.9.0",
    "cookies-js": "^1.1.0",
    "pluralize": "^1.1.0"
  }

For a minimal example, put in file a.js:

module.exports = 'hi';

And in file b.js:

module.exports = require('a');

And compile with browserifyinc --require ./a.js:a --require ./b.js:b. The first time this works fine; the second (after filling the cache) it results in Error: ENOENT, open 'a'. This is because the cache contains

      "deps": {
        "a": "a"
      }

We should not try to open that, since it is a module name and not a file path.

• Note: The module.exports setting is irrelevant, but useful if you want to see that this is a meaningful thing to do. In file c.js:

console.log(require('a'))
console.log(require('b'))

And then compile with browserifyinc --external a --external b c.js. When run after the first bundle, this will print 'hi' twice.

The thing is that I use typescript sources that get compiled into new js files with new modified times even though the sources themselves do not change, which means that the incremental build always think they changed even when they actually didn't.

Could there be an option to switch from mtimes to md5/sha1 hashes?

Actually it could even be both for speed reasons:
if mtime did not change, file did not change
else if hash did not change file did not change and update mtime
else file did change and update mtime and hash

Today I had the joy of experimencing this "impossible" error:

https://www.youtube.com/watch?v=9UObTLLYhmo

Deleting my build cache file (buildCache.json) solved the problem. I have now disabled browserify-incremental, and I haven't had the problem since.

bar.js

var _ = require('lodash');
console.log(_.VERSION);

$ npm i [email protected] browserify browserify-incremental
...
$ ./node_modules/.bin/browserifyinc bar.js -o bundle.js -v
412214 bytes written to bundle.js (0.51 seconds)
$ node bundle.js 
3.10.1
$ rm -r node_modules/
$ npm i [email protected] browserify browserify-incremental
$ ./node_modules/.bin/browserifyinc bar.js -o bundle.js -v
412214 bytes written to bundle.js (0.03 seconds)
$ node bundle.js 
3.10.1
$ node bar.js
3.10.0

Issue doesn't happen when going to a newer version of a dependency.

browserifyinc -p bundle-collapser/plugin foo.js

Doesn't have same behavior as

browserify -p bundle-collaper/plugin foo.js

I.e., the plugin doesn't seem to be applied.

Vanilla browserify allows you to do b.bundle(cb) and get the bundle passed into cb, but with browserify-incremental the callback will only be called if .pipe(someStream) is included.

This code works:

var fs = require('fs');
var browserifyInc = require('browserify-incremental');

var b = browserifyInc({
  entries: ['entry.js'],
  cache: {},
  packageCache: {},
  fullPaths: true,
  cacheFile: 'cache.json'
});

b.on('log', console.log.bind(console));
bundle();

function bundle() {
  b.bundle().pipe(fs.createWriteStream('output.js'));
}

but if bundle is changed to b.bundle(), the cache file is never written and therefore incremental builds do not work.

Is this a bug or am I missing some valid reason this has to happen? If it's expected behavior it should probably be documented.

So there is a murky backwater in node such that you can set NODE_PATH in the environment to multiple directories. Say you had a file system structure like this:

$ tree
.
├── a
│   └── b
│       └── trolls
│           └── something.js
└── no
    └── trolls
        └── other.js

Then you could do NODE_PATH=a/b/trolls:no/trolls and these things would work:

require('something.js');
require('other.js');

This is stinky. But it works. Obviously, collisions are imminent. It's a bad idea all around. But... occasionally, it is useful.

Would browserify-incremental do the right thing and use the paths to the files so that if you have a collision on the require name, the actual key in the cache file does not collide? I'll figure out how to try this soon but I wanted to ask in case you already knew.

Just by adding a single . to the front of the filename, we gain

• a cleaner project directory
• ability to do simple code search with grep 'code' * */* without getting (spam) results from browserify-cache.json

Why not?

I'm using the envify transform and uglify to remove dead code.

.transform('envify', {
    global: true,
    NODE_ENV: process.argv.indexOf('--production') !== -1 ? 'production' : 'development'
  })

When the NODE_ENV property is changed between runs, browserify-incremental still uses its cached version of files which produces a different output to what is expected.

Is this fixable in browserify-incremental? Otherwise, before I run browserify-incremental I need to delete the cache file when NODE_ENV changes between runs and that means storing state somewhere. 😞

I'm trying to use browserify-incremental with gulp but it won't work. When using browserify it works fine but when I change to browserify-incremental i get the following stacktrace:

I tried using different node versions from 0.10 to 0.12.

RangeError: Maximum call stack size exceeded
    at /Users/alexander/dev/my-project/node_modules/browserify-incremental/node_modules/through2/through2.js:88:16
    at Function.<anonymous> (/Users/alexander/dev/my-project/node_modules/browserify-incremental/node_modules/through2/through2.js:46:12)
    at Browserify.b.bundle (/Users/alexander/dev/my-project/node_modules/browserify-incremental/node_modules/browserify-cache-api/index.js:65:32)
    at Browserify.rebundle (/Users/alexander/dev/my-project/gulp/javascripts.js:44:15)
    at Browserify.emit (events.js:107:17)
    at /Users/alexander/dev/my-project/node_modules/browserify-incremental/node_modules/browserify-cache-api/index.js:124:11
    at /Users/alexander/dev/my-project/node_modules/browserify-incremental/node_modules/browserify-cache-api/index.js:321:5
    at /Users/alexander/dev/my-project/node_modules/browserify-incremental/node_modules/browserify-cache-api/node_modules/async/lib/async.js:180:24
    at Object.async.eachLimit (/Users/alexander/dev/my-project/node_modules/browserify-incremental/node_modules/browserify-cache-api/node_modules/async/lib/async.js:171:12)
    at invalidateModifiedFiles (/Users/alexander/dev/my-project/node_modules/browserify-incremental/node_modules/browserify-cache-api/index.js:306:9)

My code looks like this:

createBundle: function(bundle, config) {
    config = merge({
      production: false,
      watch: true
    }, config);

    var extension = config.production ? '.min.js' : '.js';
    var outputFile = bundle.output + extension;

    var bundler = browserifyInc(bundle.input, {
        extensions: ['.js', '.jsx'],
        transform: [reactify]
      });

    if(config.watch) {
      bundler = watchify(bundler);
    }

    bundler.transform('brfs');

    var rebundle = function() {
      var startTime = new Date().getTime();
      bundler.bundle()
        .on('error', gutil.log.bind(gutil, 'Browserify Error'))
        .pipe(source(outputFile))
          .pipe(buffer())
          .pipe(gulpif(!config.production, sourcemaps.init({loadMaps: true}))) // loads map from browserify file
          .pipe(gulpif(config.production, uglify())) // Uglify if production flag is set
          .pipe(gulpif(!config.production, sourcemaps.write('./'))) // write .map file unless production
        .on('end', function() {
          var time = (new Date().getTime() - startTime) / 1000;
          console.log("Finished browserify on file " + outputFile.cyan + " in " + (time + 's').magenta);
        })
        .pipe(gulp.dest(bundle.dest))
        .pipe(gulpif(!config.production, connect.reload())); // Trigger livereload change
    }
    // Rebuild file on update
    bundler.on('update', rebundle);
    // Initial build
    rebundle();
  }

It looks like browserify-inc is not very good at dealing with .json files. On subsequent build of the .json files, browserify-inc will always append module.exports in front of the previous source.

As seen in the below source for "/mnt/myapp/app/assets/javascripts/app/a.json":
has source "module.exports=module.exports=module.exports=module.exports={b:2}\n"

This only happen if the file required contains the whole json object raw without getting set to module.exports manually.

Please see the cache file in gist
https://gist.github.com/sleungcy/c8b2b27c22df334c41eb

capaj/require-globify lets you write code like

// index.js
const templates = require('./templates/*', { mode: 'hash' });

which at some point expands into

var templates = {
    'a': require('./template/a.html'),
    'b': require('./template/b.html')
};

When used with browserify-incremental index.js is not rebuilt after contents of the template directory change (like adding or removing a file).

My build script looks like this

// gulpfile.js
var b = browserify(_.defaults({
    entries: entries,
    debug: true
}, browserifyInc.args));

b.transform('require-globify');

browserifyInc(b, {
    cacheFile: './.browserify-incremental-cache.json'
});

Maybe there is a way to make browserify-incremental kick in after require-globify transform?

When used with lessify

Changes to less files included in an @import statement doesn't cause the output to be updated. You have to touch the main less file to trigger an update.

Example:

# main.less
@import "foo.less";

# foo.less
body { background: blue }

# main.coffee
require('main.less')

When a cache exists, chaning foo.less doesn't cause the resulting CSS in main.coffee to change. I have to run touch main.less to cause an update.

Can be reproduced using this test repo I made. The error is Uncaught TypeError: Cannot read property 'version' of undefined, which is being thrown by the elliptic.js module of Browserify's crypto implementation. This only happens after the cache begins being used, which is incidentally after running browserify-incremental three times (see #8).

It's worth noting that the Watchify library has encountered the same problem (browserify/watchify#130, indutny/elliptic#30).

This is a weird thing I'm seeing: if I run browserifyinc, pointing it at a cache file that doesn't exist, my build takes 28 seconds (and it creates the cache file), So far, expected.

But if I re-run it, it takes 28 seconds again. Only on the third run does it pick up the cache file, and I get the nice 2.25 second built time that I want.

This isn't really a problem, since I haven't seen a reason (apart from testing) to blow out my cache, but it's... odd.

We've been using browserify-incremental where I work for a while and I've found that it doesn't work well with watchify. I know there's some overlap there in what this and watchify are supposed to accomplish, but even with the faster compilation between runs, it's nice to have a watch mode that watches the right files and triggers a bundle when they change.

From playing around with watchify, it looks like the issue is that browserify-incremental doesn't emit file events for cached files. I wrote a hackish plugin to change that:

var BrowserifyCache = require('browserify-cache-api');

module.exports = function incrementalWatchPlugin(b) {
    var hasReadCache = false;
    b.on('bundle', function() {
        var cache;
        if (!hasReadCache) {
            cache = BrowserifyCache.getCache(b);
            Object.keys(cache.dependentFiles).forEach(function(file) {
                b.emit('file', file);
            });
            Object.keys(cache.modules).forEach(function(file) {
                b.emit('file', file);
            });
        }
        hasReadCache = true;
    });
};

This seems to work, but it's not super efficient. Watchify still has a cache it doesn't need.

Do you have any thoughts on this? Is there a better way to have a working watch mode? Is there a reason browserify-incremental doesn't emit file events?

browserify-incremental is awesome, by the way. It's the main reason we're still using Browserify; otherwise it'd be too slow for us.

In line 94 of README.md

 browserifyInc(b {cacheFile: './browserify-cache.json'})

Is there a missing comma or b is a function?

I would like to generate bundles without the fullPaths option, because I use browserifyinc to generate deployment builds and these paths conflict with other developers' builds. However, it seems that this option is a requirement for browserify-incremental in order to cache the files. Is there a way to avoid that problem?

It seems that watchify suffered from the same problem but it has been solved: see browserify/watchify#78

Probably this is specific to my configuration, but is nothing different from a standard build task with gulp.

var b = browserify(bundleConfig);
browserifyInc(b, {cacheFile: './tmp/cache/browserify/browserifyinc-cache.json'});

at the end of the build, there's a cache file created, but between each build, when starts browserify, it doesn't use the file and recreates every time.

Is there anything extra that I need to do to get cacheFile working?

Thanks!

For some reason the cache file is never saved why I try using this.

Versions:

    "babelify": "^7.2.0",
    "browserify": "^12.0.1",
    "browserify-incremental": "^3.0.1",

Node: v0.12.0 (I think)

I'm using this inside of a cordova js hook.

Relevant parts of the code:

var browserify = require('browserify-incremental'),
    b = browserify({cacheFile: path.join(ctx.opts.projectRoot, 'browserify-cache.json')})
        .transform(require('babelify'))
        .require(filePath, {expose: 'app'});

return Q.ninvoke(b, 'bundle')
    .then(function(buf) {
    });

I dug into the code and fiddled around with some console logging.

In this code bundle gets called but end is never called.

  b.on('bundle', function(bundleStream) {
    // store on completion
    bundleStream.on('end', function() {
      storeCache(b, cacheFile);
    });
  });

In this code, if I kill the proxyEvent and pipe and just set outputStream = bundleStream the end event is emitted and the cache file is saved.

      var bundleStream = prevBundle.call(b, cb);
      proxyEvent(bundleStream, outputStream, 'file');
      proxyEvent(bundleStream, outputStream, 'package');
      proxyEvent(bundleStream, outputStream, 'transform');
      proxyEvent(bundleStream, outputStream, 'error');
      bundleStream.pipe(outputStream);

So for some reason the end event isn't being proxied/piped. I tried various things like adding a proxyEvent for it or calling .end() on outputStream in an .on handler for bundleStream. But none of them seemed to work, so I can't figure out what exactly is wrong.

Note that I'm not sure my change would be a proper fix anyways. The cache file was saved and read but I never got any notable speed-up to my compilation of babel+react so I don't know if the caching wasn't working or I need a different browserify cache module.

Hey,

If I modify the contents of the main bundle entry re-using the same bundler instance, output is not updated accordingly. It seems that the cache is not properly invalidated. However sometimes it does work. Could it be related to mtime resolution not being precise enough? I'm on OS X.

Here is a gist showing the issue: https://gist.github.com/ngryman/52ca69f809d4c38f4cdb4db710de3592

git clone https://gist.github.com/52ca69f809d4c38f4cdb4db710de3592.git
cd 52ca69f809d4c38f4cdb4db710de3592
npm install
node index.js # re-use the same bundler instance, fails
node index.js --force # force re-create a new bundler, succeeds

/ref: ngryman/gulp-bro#4

I used the following command:

browserifyinc -t brfs src/js/app.js -o build/out.js

The brfs transform doesn't get applied.

I doubt this is something that is fixable easily, and probably not within the purview of browserify-incremental. I also don't have a test case (I know, this is a cardinal sin)...

But I have a project using the following structure:

- views
  - view1
    - i18n.js (content A)
    - index.js (does not require lib1)
  - view2
    - i18n.js (content A)
    - index.js (does require lib1)
- node_modules
  - lib1
    - locales
      - index.js (content A)
    - index.js

Files marked as content A have the same file contents, but are at different paths.

and the build script is basically:

$ browserifyinc views/view1/index.js -o build/view1.bundle & browserifyinc views/view2/index.js -o build/view2.bundle

Notice how the browserifyinc commands are running in parallel-ish. When I execute the view1.bundle, I get a very strange error:

TypeError: undefined is not an object (evaluating 'arguments[4]["./node_modules/lib1/locales/index.js"][0]')

Doing some digging it appears that it may be related to browserify's deduping of modules with identical contents: https://twitter.com/KoryNunn/status/435198784146927616.

What appears to be happening is that browserify-incremental (or browserify?) is deduping the module contents to a module that doesn't end up existing in the final bundle! view1 never requires lib1, but has a file with the same contents as a file within lib1.

I'm not sure if the concurrency is the issue, or simply the similar file contents with different entry points. Browserify may, for example, be assuming that whatever is in the cache is valid for inclusion in the bundle.

Again, this is probably a weird edge case (or maybe not? perhaps it's all shared caches?), and I don't have a test case, and I don't expect a magical fix. But I wanted to share some info in the event that someone else finds this same problem. Also if there is anything that jogs your mind as to the real problem here, I would be very grateful!

Thanks for listening.

Splitting this issue report out from #14 (comment).

$ npm install browserify browserify-incremental
$ echo "require('./b.js')" > a.js
$ echo "console.log('hi')" > b.js
$ ./node_modules/.bin/browserifyinc a.js -o app.js
$ rm b.js
$ ./node_modules/.bin/browserifyinc a.js -o app.js
$ node app.js
hi

I have 3 javascript compilation gulp tasks. First time I used single cache file, and compilation was very fast. But now compilation is falls dow. When I separate cache to 3 files compilation became very slow.

I'm not sure why this restriction is in place, since the debug information is being correctly written to stderr.

It's a simple change, but apparently I can't create a PR for it as master doesn't include any recent tag-only releases.

https://github.com/ianthehenry/browserify-incremental/commit/3eff0c4460522d73ea6b0a1c529677ff632712d6

I was running into some bugs around Browserify's deduplication that were fixed by upgrading to 8.1.3.

Now I see that 0.9 just came out, which I haven't tested yet.

What's the long term plan for browserify-incremental track browserify versions?

Specifying a wildcard version on browserify might sort of make sense, so that users of browserify-incremental have full control over the version they get (with documented "known to work" versions, perhaps). But it's also potentially risky, as it requires reading a little bit of docs. Thoughts?

I just installed [email protected] using npm install browserify-incremental and running it throws an error about a missing usage.txt file:

$./node_modules/browserify-incremental/bin/cmd.js 
events.js:85
      throw er; // Unhandled 'error' event
            ^
Error: ENOENT, open 'node_modules/browserify-incremental/bin/usage.txt'
    at Error (native)

How is this different than watchify / browserify combo? Maybe updating the README to explain the differences?

It would be nice to pass in the command line the file(s) which have updated to avoid any cache checks at all, for faster bundling.

browserifyinc build/main.js -o public/js/main.js --reload build/f1.js build/f2.js

I can't reproduce (and I couldn't understand the generated source also), but apparently there is an issue with dependencies that change versions and/or deleted dependencies. Maybe browserify-incremental keeps using the same old versions. Is it possible that this is happening?

I'm getting an error while using browserify-incremental with gulp:

[gulp] Using gulpfile ~/Public/advertentietool/gulpfile.js
[gulp] Starting 'watch'...
[gulp] Finished 'watch' after 813 ms
[16:19:16] Live reload server listening on: 35729
[16:19:18] calendar.js was reloaded.
[gulp] Starting 'javascripts'...

/home/leon/Public/advertentietool/node_modules/browserify/index.js:569
        throw new Error(
              ^
Error: bundle() no longer accepts option arguments.
Move all option arguments to the browserify() constructor.
    at Browserify.bundle (/home/leon/Public/advertentietool/node_modules/browserify/index.js:569:15)
    at Browserify.b.bundle (/home/leon/Public/advertentietool/node_modules/browserify-incremental/node_modules/browserify-cache-api/index.js:57:21)
    at Browserify.b.bundle (/home/leon/Public/advertentietool/node_modules/browserify-incremental/index.js:40:21)
    at /home/leon/Public/advertentietool/gulpfile.js:66:10
    at Transform.write [as _transform] (/home/leon/Public/advertentietool/node_modules/vinyl-transform/index.js:14:18)
    at Transform._read (/home/leon/Public/advertentietool/node_modules/vinyl-transform/node_modules/through2/node_modules/readable-stream/lib/_stream_transform.js:184:10)
    at Transform._write (/home/leon/Public/advertentietool/node_modules/vinyl-transform/node_modules/through2/node_modules/readable-stream/lib/_stream_transform.js:172:12)
    at doWrite (/home/leon/Public/advertentietool/node_modules/vinyl-transform/node_modules/through2/node_modules/readable-stream/lib/_stream_writable.js:237:10)
    at writeOrBuffer (/home/leon/Public/advertentietool/node_modules/vinyl-transform/node_modules/through2/node_modules/readable-stream/lib/_stream_writable.js:227:5)
    at Transform.Writable.write (/home/leon/Public/advertentietool/node_modules/vinyl-transform/node_modules/through2/node_modules/readable-stream/lib/_stream_writable.js:194:11)

This is how I'm using browserify-incremental:

      var args = {
          paths: [paths.src.javascripts]
      };

      var b = browserify(filename, args);
      var bi = browserifyInc(b);
      return bi
        // Configure transforms in package.json?
        .bundle();

I'm getting an ENOENT: can't open... error when I try to use --require in conjunction with a custom NODE_PATH (Rails app, can't get around the path issue.) The cachefile gets written initially, but the next time it's read it throws an error trying to read whatever file is passed to --require.

Here's a repo: https://github.com/joshfrench/browserifyinc-demo

To reproduce:

1. git clone && npm install
2. npm run build three times.

I suspect #8 is making it so you have to run the build an extra time up front, but once the cache actually gets read the error will occur.

I've create a PR to grunt-browserify to add support for browserify-incremental to that plugin. :)