broccolijs / broccoli-persistent-filter Goto Github PK

View Code? Open in Web Editor NEW

12.0 12.0 33.0 767 KB

License: MIT License

JavaScript 57.51% TypeScript 42.49%

broccoli-persistent-filter's People

Contributors

Stargazers

Watchers

broccoli-persistent-filter's Issues

sometimes unlink in cache fails...

Logging may no longer work

DEBUG=broccoli* ember sa no longer yields broccoli-persistent-filter logging.

Parallel builds with similar apps and deps can cause crashes

With @rwjblue's help, I discovered that building two of my apps in parallel can cause a crash related to broccoli-persistent-filter. Here is an example stack:

Build failed.
The Broccoli Plugin: [BroccoliMergeTrees: TreeMerger (appAndDependencies)] failed with:
Error: unexpected end of file
    at Zlib._handle.onerror (zlib.js:370:17)

The broccoli plugin was instantiated at:
    at BroccoliMergeTrees.Plugin (/mnt/jenkins-workspaces/batterii-horizons-deploy/src/buttercup-epcot/apps/bc-admin/node_modules/broccoli-plugin/index.js:7:31)
    at new BroccoliMergeTrees (/mnt/jenkins-workspaces/batterii-horizons-deploy/src/buttercup-epcot/apps/bc-admin/node_modules/ember-cli/node_modules/broccoli-merge-trees/index.js:16:10)
    at Function.BroccoliMergeTrees [as _upstreamMergeTrees] (/mnt/jenkins-workspaces/batterii-horizons-deploy/src/buttercup-epcot/apps/bc-admin/node_modules/ember-cli/node_modules/broccoli-merge-trees/index.js:10:53)
    at mergeTrees (/mnt/jenkins-workspaces/batterii-horizons-deploy/src/buttercup-epcot/apps/bc-admin/node_modules/ember-cli/lib/broccoli/merge-trees.js:85:33)
    at EmberApp._mergeTrees (/mnt/jenkins-workspaces/batterii-horizons-deploy/src/buttercup-epcot/apps/bc-admin/node_modules/ember-cli/lib/broccoli/ember-app.js:1815:12)
    at EmberApp.appAndDependencies (/mnt/jenkins-workspaces/batterii-horizons-deploy/src/buttercup-epcot/apps/bc-admin/node_modules/ember-cli/lib/broccoli/ember-app.js:1135:17)
    at EmberApp.javascript (/mnt/jenkins-workspaces/batterii-horizons-deploy/src/buttercup-epcot/apps/bc-admin/node_modules/ember-cli/lib/broccoli/ember-app.js:1235:30)
    at EmberApp.toArray (/mnt/jenkins-workspaces/batterii-horizons-deploy/src/buttercup-epcot/apps/bc-admin/node_modules/ember-cli/lib/broccoli/ember-app.js:1672:12)
    at EmberApp.toTree (/mnt/jenkins-workspaces/batterii-horizons-deploy/src/buttercup-epcot/apps/bc-admin/node_modules/ember-cli/lib/broccoli/ember-app.js:1694:38)
    at module.exports (/mnt/jenkins-workspaces/batterii-horizons-deploy/src/buttercup-epcot/apps/bc-admin/ember-cli-build.js:97:14)

We determined the cause was related to stefanpenner/async-disk-caches use of the system temp directory. The two applications are separate apps (in separate directories), but they are similar in size and almost identical in dependencies. The builds kick off at the same time in parallel. Both builds end up using the same system tmp cache and one clobbers the other.

A workaround is to set an env variable that causes it to not use the system cache:

BROCCOLI_PERSISTENT_FILTER_CACHE_ROOT=./cache ember build -prod

readme

Add tests to confirm file contents when `persist: true` is set.

Worker is terminated

We're having the following build error since yesterday. I'm not able to reproduce locally yet but it fails on our CI when running ember test. I've tried many different things but the error persists. This is blocking our builds so any help would be much appreciated.

Note that this time it mentioned 'ember-data-copyable' but sometimes it mentions a different dependency so not sure if that's relevant.

Build Error (broccoli-persistent-filter:Babel > [Babel: ember-data-copyable]) in ember-data-copyable/-private/symbols.js

Worker is terminated

EEXIST: file already exists, symlink

working on a fix ...

EEXIST: file already exists, symlink '/Users/stefanpenner/src/y/dsp/tests/index.html' -> '/Users/stefanpenner/src/y/dsp/tmp/babel-output_path-gVwcluk
V.tmp/app/tests/index.html'
Error: EEXIST: file already exists, symlink '/Users/stefanpenner/src/y/dsp/tests/index.html' -> '/Users/stefanpenner/src/y/dsp/tmp/babel-output_path-
gVwclukV.tmp/app/tests/index.html'
    at Error (native)
    at Object.fs.symlinkSync (fs.js:899:18)
    at symlink (/Users/stefanpenner/src/y/dsp/node_modules/ember-cli-babel/node_modules/broccoli-babel-transpiler/node_modules/broccoli-persistent-fi
lter/node_modules/symlink-or-copy/index.js:82:14)
    at symlinkOrCopySync (/Users/stefanpenner/src/y/dsp/node_modules/ember-cli-babel/node_modules/broccoli-babel-transpiler/node_modules/broccoli-per
sistent-filter/node_modules/symlink-or-copy/index.js:58:5)
    at Babel.<anonymous> (/Users/stefanpenner/src/y/dsp/node_modules/ember-cli-babel/node_modules/broccoli-babel-transpiler/node_modules/broccoli-per
sistent-filter/index.js:92:16)
    at /Users/stefanpenner/src/y/dsp/node_modules/ember-cli-babel/node_modules/broccoli-babel-transpiler/node_modules/broccoli-persistent-filter/node
_modules/promise-map-series/index.js:11:14
    at lib$rsvp$$internal$$tryCatch (/Users/stefanpenner/src/y/dsp/node_modules/ember-cli-babel/node_modules/broccoli-babel-transpiler/node_modules/b
roccoli-persistent-filter/node_modules/rsvp/dist/rsvp.js:493:16)
    at lib$rsvp$$internal$$invokeCallback (/Users/stefanpenner/src/y/dsp/node_modules/ember-cli-babel/node_modules/broccoli-babel-transpiler/node_mod
ules/broccoli-persistent-filter/node_modules/rsvp/dist/rsvp.js:505:17)
    at /Users/stefanpenner/src/y/dsp/node_modules/ember-cli-babel/node_modules/broccoli-babel-transpiler/node_modules/broccoli-persistent-filter/node
_modules/rsvp/dist/rsvp.js:1001:13
    at lib$rsvp$asap$$flush (/Users/stefanpenner/src/y/dsp/node_modules/ember-cli-babel/node_modules/broccoli-babel-transpiler/node_modules/broccoli-
persistent-filter/node_modules/rsvp/dist/rsvp.js:1198:9)

Consider dropping support for node v0.12

It's making updating dependencies impossible/hard. There's a pretty good chance it doesn't work out of the box on v0.12 anyway, and Appveyor doesn't run with it (only 4, 6, and 7 right now). I think we should be testing against 4, 6, 8, and 9. Possibly even just 6, 8, and 9.

make it easy to step out of the persistent cache strategy.

Unexpected token on spread operator after update to ember-cli 2.14

I am working on a bigger app that has been running smoothly for more than a year. Currently we have ember-cli 2.12 as dependency, everything works ok.

As soon as I switch to ember-cli 2.14 from 2.12, I am getting this error:

The Broccoli Plugin: [broccoli-persistent-filter:Babel > [Babel: lintTree(app)]: Babel: lintTree(app)] failed with:
SyntaxError: XXX/tests/.../XXX.js: Unexpected token (66:6)
  64 |     const maxDistance = Math.max(...data.map(hole => hole.distance || 0)) || 1;
  65 |     return data.map(hole => ({
> 66 |       ...hole,
     |       ^
  67 |       y: sizes.width / 2 / maxDistance * hole.distance,
  68 |       d: this.shapePath('', sizes.hole),
  69 |     }));
    at Parser.pp$5.raise (/usr/local/src/dev/ui/node_modules/babylon/lib/index.js:4452:13)
    at Parser.pp.unexpected (/usr/local/src/dev/ui/node_modules/babylon/lib/index.js:1761:8)
    at Parser.pp$3.parseIdentifier (/usr/local/src/dev/ui/node_modules/babylon/lib/index.js:4330:10)
    at Parser.pp$3.parsePropertyName (/usr/local/src/dev/ui/node_modules/babylon/lib/index.js:4154:96)
    at Parser.pp$3.parseObj (/usr/local/src/dev/ui/node_modules/babylon/lib/index.js:4043:12)
    at Parser.pp$3.parseExprAtom (/usr/local/src/dev/ui/node_modules/babylon/lib/index.js:3717:19)
    at Parser.pp$3.parseExprSubscripts (/usr/local/src/dev/ui/node_modules/babylon/lib/index.js:3492:19)
    at Parser.pp$3.parseMaybeUnary (/usr/local/src/dev/ui/node_modules/babylon/lib/index.js:3472:19)
    at Parser.pp$3.parseExprOps (/usr/local/src/dev/ui/node_modules/babylon/lib/index.js:3402:19)
    at Parser.pp$3.parseMaybeConditional (/usr/local/src/dev/ui/node_modules/babylon/lib/index.js:3379:19)
    at Parser.pp$3.parseMaybeAssign (/usr/local/src/dev/ui/node_modules/babylon/lib/index.js:3342:19)
    at Parser.pp$3.parseParenAndDistinguishExpression (/usr/local/src/dev/ui/node_modules/babylon/lib/index.js:3826:26)
    at Parser.pp$3.parseExprAtom (/usr/local/src/dev/ui/node_modules/babylon/lib/index.js:3707:19)
    at Parser.pp$3.parseExprSubscripts (/usr/local/src/dev/ui/node_modules/babylon/lib/index.js:3492:19)
    at Parser.pp$3.parseMaybeUnary (/usr/local/src/dev/ui/node_modules/babylon/lib/index.js:3472:19)
    at Parser.pp$3.parseExprOps (/usr/local/src/dev/ui/node_modules/babylon/lib/index.js:3402:19)
    at Parser.pp$3.parseMaybeConditional (/usr/local/src/dev/ui/node_modules/babylon/lib/index.js:3379:19)

The broccoli plugin was instantiated at: 
    at Babel.Plugin (/usr/local/src/dev/ui/node_modules/broccoli-plugin/index.js:7:31)
    at Babel.Filter [as constructor] (/usr/local/src/dev/ui/node_modules/broccoli-persistent-filter/index.js:62:10)
    at new Babel (/usr/local/src/dev/ui/node_modules/ember-cli/node_modules/broccoli-babel-transpiler/index.js:35:10)
    at processModulesOnly (/usr/local/src/dev/ui/node_modules/ember-cli/lib/broccoli/babel-process-modules-only.js:28:10)
    at EmberApp.lintTestTrees (/usr/local/src/dev/ui/node_modules/ember-cli/lib/broccoli/ember-app.js:1195:17)
    at EmberApp.appTests (/usr/local/src/dev/ui/node_modules/ember-cli/lib/broccoli/ember-app.js:1162:28)
    at EmberApp.test (/usr/local/src/dev/ui/node_modules/ember-cli/lib/broccoli/ember-app.js:1148:28)
    at EmberApp.toArray (/usr/local/src/dev/ui/node_modules/ember-cli/lib/broccoli/ember-app.js:1679:63)
    at EmberApp.toTree (/usr/local/src/dev/ui/node_modules/ember-cli/lib/broccoli/ember-app.js:1694:38)
    at module.exports (/usr/local/src/dev/ui/ember-cli-build.js:26:14)
    at Builder.setupBroccoliBuilder (/usr/local/src/dev/ui/node_modules/ember-cli/lib/models/builder.js:56:19)
    at new Builder (/usr/local/src/dev/ui/node_modules/ember-cli/lib/models/builder.js:30:10)
    at ServeTask.run (/usr/local/src/dev/ui/node_modules/ember-cli/lib/tasks/serve.js:24:55)
    at Promise.resolve.then (/usr/local/src/dev/ui/node_modules/ember-cli/lib/models/command.js:243:46)
    at tryCatch (/usr/local/src/dev/ui/node_modules/rsvp/dist/rsvp.js:539:12)
    at invokeCallback (/usr/local/src/dev/ui/node_modules/rsvp/dist/rsvp.js:554:13)

Any ideas what could be the problem? Thanks!

(I have babel-plugin-transform-object-rest-spread and babel-plugin-transform-es2015-destructuring installed as dependencies)

Prevent inconsistent ember-cli builds with different extensions

Maybe this is correct place for this issues collection, if not, feel free to close it or redirect me to proper repo.

ember-cli/ember-ajax#428
typed-ember/ember-cli-typescript#724
emberjs/ember-cli-babel#240

broccoli-persistent-filter overwrites source files

I discovered that under certain conditions, when the input node contains two conflicting files like file.js and file.coffee, broccoli-persistent-filter will overwrite files in the source directory. I'm not sure if the culprit is broccoli-persistent-filter or fs-tree-diff, so instead of writing a test case I just created a sample repo for now. To reproduce this issue, run:

git clone https://github.com/joliss/broccoli-persistent-filter-issue
cd broccoli-persistent-filter-issue
npm install
./node_modules/.bin/broccoli serve

Then in a new terminal, run

echo 's = "CoffeeScript source"' > src/file.coffee
sleep 1
git diff src/file.js

src/file.js has been overwritten with the CoffeeScript output.

Build Error (OneShot)

This is a new one to me, and I don't even know if this is the right repo to report this issue on?

 Build Error (OneShot)

components/collapsible/index.ts: /home/runner/work/emberclear/emberclear/client/web/emberclear/components/collapsible/index.ts: unable to use this non-serializable babel config in this node process
        in /tmp/broccoli-3020EI1K66U07kNx/out-02-funnel
        at broccoli-persistent-filter:Babel > [Babel: @emberclear/ui] (Babel: @emberclear/ui)

from ci log: https://github.com/NullVoxPopuli/emberclear/pull/922/checks?check_run_id=746347841

on this branch: NullVoxPopuli/emberclear#922
(using v4-alpha ember-cli-typescript and embroider 0.19.0)

maybe cc @chriskrycho and @ef4?
maaaybe related? emberjs/ember-test-waiters#158

An in-range update of walk-sync is breaking the build 🚨

Version 0.3.2 of walk-sync just got published.

Branch	Build failing 🚨
Dependency	walk-sync
Current Version	0.3.1
Type	dependency

This version is covered by your current version range and after updating it in your project the build failed.

walk-sync is a direct dependency of this project this is very likely breaking your project right now. If other packages depend on you it’s very likely also breaking them.
I recommend you give this issue a very high priority. I’m sure you can resolve this 💪

Status Details

✅ continuous-integration/travis-ci/push The Travis CI build passed Details
❌ continuous-integration/appveyor/branch AppVeyor build failed Details

Commits

The new version differs by 5 commits.

16b1ec5 release v0.3.2 🎉
eec7313 commit yarn.lock
6d377b5 Merge pull request #31 from riegelTech/feature/add_includeBasePath_option
50a21b5 Use path separators to be more compliant with win paths
5a4fb73 Add 'includeBasePath' to options so files pathes are prefixed with baseDir path

See the full diff

Not sure how things should work exactly?

There is a collection of frequently asked questions and of course you may always ask my humans.

Your Greenkeeper Bot 🌴

persistent restore, always re-writes itself...

This is slow.

An in-range update of mocha is breaking the build 🚨

The devDependency mocha was updated from `6.0.2` to `6.1.0`.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

mocha is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details

❌ continuous-integration/travis-ci/push: The Travis CI build failed (Details).
✅ continuous-integration/appveyor/branch: AppVeyor build succeeded (Details).

Release Notes for v6.1.0

6.1.0 / 2019-04-07

🔒 Security Fixes

#3845: Update dependency "js-yaml" to v3.13.0 per npm security advisory (@plroebuck)

🎉 Enhancements

#3766: Make reporter constructor support optional options parameter (@plroebuck)
#3760: Add support for config files with .jsonc extension (@sstephant)

📠 Deprecations

These are soft-deprecated, and will emit a warning upon use. Support will be removed in (likely) the next major version of Mocha:

#3719: Deprecate this.skip() for "after all" hooks (@juergba)

🐛 Fixes

#3829: Use cwd-relative pathname to load config file (@plroebuck)
#3745: Fix async calls of this.skip() in "before each" hooks (@juergba)
#3669: Enable --allow-uncaught for uncaught exceptions thrown inside hooks (@givanse)

and some regressions:

#3848: Fix Suite cloning by copying root property (@fatso83)
#3816: Guard against undefined timeout option (@boneskull)
#3814: Update "yargs" in order to avoid deprecation message (@boneskull)
#3788: Fix support for multiple node flags (@aginzberg)

📖 Documentation

mochajs/mocha-examples: New repository of working examples of common configurations using mocha (@craigtaub)
#3850: Remove pound icon showing on header hover on docs (@jd2rogers2)
#3812: Add autoprefixer to documentation page CSS (@Munter)
#3811: Update doc examples "tests.html" (@DavidLi119)
#3807: Mocha website HTML tweaks (@plroebuck)
#3793: Update config file example ".mocharc.yml" (@cspotcode)

🔩 Other

#3830: Replace dependency "findup-sync" with "find-up" for faster startup (@cspotcode)
#3799: Update devDependencies to fix many npm vulnerabilities (@XhmikosR)

Commits

The new version differs by 28 commits.

f4fc95a Release v6.1.0
bd29dbd update CHANGELOG for v6.1.0 [ci skip]
aaf2b72 Use cwd-relative pathname to load config file (#3829)
b079d24 upgrade deps as per npm audit fix; closes #3854
e87c689 Deprecate this.skip() for "after all" hooks (#3719)
81cfa90 Copy Suite property "root" when cloning; closes #3847 (#3848)
8aa2fc4 Fix issue 3714, hide pound icon showing on hover header on docs page (#3850)
586bf78 Update JS-YAML to address security issue (#3845)
d1024a3 Update doc examples "tests.html" (#3811)
1d570e0 Delete "/docs/example/chai.js"
ade8b90 runner.js: "self.test" undefined in Browser (#3835)
0098147 Replace findup-sync with find-up for faster startup (#3830)
d5ba121 Remove "package" flag from sample config file because it can only be passes as CLI arg (#3793)
a3089ad update package-lock
75430ec Upgrade yargs-parser dependency to avoid loading 2 copies of yargs

There are 28 commits in total.

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

Your Greenkeeper Bot 🌴

Node 10 doesn't support recursive directory operations.

So the 3.0 release has a bug in node 10 that evidently isn't covered by the test cases.

This project is using @types/node@^13 which didn't flag this as an issue. (I recommend we downgrade to ^10).

According to the node.js docs, recursive rmdir was added in 12.10.0.

The API is used here: https://github.com/broccolijs/broccoli-persistent-filter/blob/master/index.ts#L228-L229

Maybe the right fix is for the fs-merger to shim the node-12 api when in node 10?

cc: @SparshithNR @stefanpenner

broccoli-persistent-filter - childprocess fail

Build Canceled: Broccoli Builder ran into an error with broccoli-persistent-filter:Babel > [Babel: ember-lodash] plugin. 💥
Worker terminated unexpectedly
Error: Worker terminated unexpectedly
at ChildProcess. (/opt/home/jenkins/workspace/_CORE-2815_wrong-question-for-US/site/src/ember/node_modules/workerpool/lib/WorkerHandler.js:181:17)
at emitTwo (events.js:106:13)
at ChildProcess.emit (events.js:191:7)
at Process.ChildProcess._handle.onexit (internal/child_process.js:215:12)
:site:emberTest FAILED

Rejecting / Excluding Files | broccoli-funnel

Now that we have Dependency Tracking #168, this plugin becomes much more interesting for bundler-liker build scenarios, where n files get merged into 1 or fewer than n files.

It would be nice, if broccoli-persistent-filter could add a hook to reject a file from the build (similar to getDestFilePath), removing the need for a subsequent broccoli-funnel.

Add mechanism to clear persistent cache.

It would be nice to have an environment variable that can be used to blow all the caches and force a fresh full build. Because async-disk-cache doesn't nest under a single base dir in the os.tmpDir() (I plan to submit a PR over there to do this), we cannot simply do rm -rf /tmp/some/path/here (even if we knew the path).

I would like to do something like:

CLEAR_PERSISTENT_CACHE=true ember build

And have that do something like the following just before the first boot (before this):

  this.processor.clear();

I'm happy to PR this, but wanted to throw it out there before I implemented to make sure there were no major objections...

Allow tree to implement custom `cacheKey`

I could be wrong (close if I am), but it seems like the individual tree can no longer specify its own cacheKey. Instead it always calls hashForDep(ctx.baseDir()) (here), but sometimes you need to make a custom cacheKey that is not related to package.json (for example: bower dependencies).

`dependencyInvalidation` and `concurrency` should not be required

Shall we mark these two options as optional as well?

broccoli-persistent-filter/index.ts

Lines 134 to 135 in 8e93916

 dependencyInvalidation: boolean; 

 concurrency: number;

I'm writing a subclass based on the broccoli-persistent-filter in TypeScript, by now, calling super(node, options) in the constructor will cause an unfixable type error:

IMO, the dependencyInvalidation, and the concurrency options should not be required because the base class will set a default value for both of them when calling the super(node, option).

Currently, I'm using a workaround below:

interface Options {
  name?: string;
}

interface SuperOptions extends Options {
  dependencyInvalidation: boolean;
  concurrency: number;
}

class MyFilter extends PersistentFilter {
  constructor(node: InputNode, options: Options) {
    options.name = options.name || 'my-filter';
    super(node, options as SuperOptions);
  }
}

tests

`async-disk-cache` 1.2.1 breaks Node 0.12 compatibility

Some addons (like ember-cli-deploy-build and torii) are depending on broccoli-persistent-filter and are still supporting Node 0.12 for some reason.

Is it a good idea to release a version of broccoli-persistent-filter which point to async-disk-cache <=1.2.0? Or should those addon developers drop 0.12 support instead?

helpers / docs for safe & stable build.

readme
inline docs

This project appears to introduce 965 files via itself and its deps

own files we may not want:

deps we may want to investigate:

test errors w/ hash-for-dep

[Without using new hash-for-dep], I get the following errors on test (below).

They can be repaired by replacing

F.prototype.baseDir = function() {
  return '../';

With

var path = require('path');

F.prototype.baseDir = function() {
  return path.join(__dirname, '../');

I'll submit a patch in a second... however I have no idea what changed/is different about my env (macosx 10.11.3 node 0.12.4 ... npm 3.6.0) to cause the failures.

  1) Filter persistent cache initializes cache:
     Cannot find module '/Volumes/Macintosh_HD/Users/shauncutts/src/broccoli-persistent-filter/node_modules//package.json'
  Error: Cannot find module 'node_modules//package.json'
      at pkg (node_modules/hash-for-dep/lib/pkg.js:19:20)
      at statPathsFor (node_modules/hash-for-dep/lib/stat-paths-for.js:14:20)
      at hashForDep (node_modules/hash-for-dep/index.js:15:21)
      at Filter.cacheKey (index.js:117:10)
      at Object.module.exports.cacheKey (lib/strategies/persistent.js:22:16)
      at Object.module.exports.init (lib/strategies/persistent.js:12:50)
      at Processor.init (lib/processor.js:16:18)
      at Filter (index.js:56:18)
      at new F (test/test.js:541:14)
      at Context.<anonymous> (test/test.js:564:15)

  2) Filter persistent cache initializes cache using ENV variable if present:
     Cannot find module '/Volumes/Macintosh_HD/Users/shauncutts/src/broccoli-persistent-filter/node_modules//package.json'
  Error: Cannot find module 'node_modules//package.json'
      at pkg (node_modules/hash-for-dep/lib/pkg.js:19:20)
      at statPathsFor (node_modules/hash-for-dep/lib/stat-paths-for.js:14:20)
      at hashForDep (node_modules/hash-for-dep/index.js:15:21)
      at Filter.cacheKey (index.js:117:10)
      at Object.module.exports.cacheKey (lib/strategies/persistent.js:22:16)
      at Object.module.exports.init (lib/strategies/persistent.js:12:50)
      at Processor.init (lib/processor.js:16:18)
      at Filter (index.js:56:18)
      at new F (test/test.js:541:14)
      at Context.<anonymous> (test/test.js:576:15)

  3) Filter persistent cache `cacheKeyProcessString` return correct first level file cache:
     Cannot find module '/Volumes/Macintosh_HD/Users/shauncutts/src/broccoli-persistent-filter/node_modules//package.json'
  Error: Cannot find module 'node_modules//package.json'
      at pkg (node_modules/hash-for-dep/lib/pkg.js:19:20)
      at statPathsFor (node_modules/hash-for-dep/lib/stat-paths-for.js:14:20)
      at hashForDep (node_modules/hash-for-dep/index.js:15:21)
      at Filter.cacheKey (index.js:117:10)
      at Object.module.exports.cacheKey (lib/strategies/persistent.js:22:16)
      at Object.module.exports.init (lib/strategies/persistent.js:12:50)
      at Processor.init (lib/processor.js:16:18)
      at Filter (index.js:56:18)
      at new F (test/test.js:541:14)
      at Context.<anonymous> (test/test.js:600:15)

  4) Filter persistent cache properly reads the file tree:
     Cannot find module '/Volumes/Macintosh_HD/Users/shauncutts/src/broccoli-persistent-filter/node_modules//package.json'
  Error: Cannot find module 'node_modules//package.json'
      at pkg (node_modules/hash-for-dep/lib/pkg.js:19:20)
      at statPathsFor (node_modules/hash-for-dep/lib/stat-paths-for.js:14:20)
      at hashForDep (node_modules/hash-for-dep/index.js:15:21)
      at ReplaceFilter.Filter.cacheKey (index.js:117:10)
      at Object.module.exports.cacheKey (lib/strategies/persistent.js:22:16)
      at Object.module.exports.init (lib/strategies/persistent.js:12:50)
      at Processor.init (lib/processor.js:16:18)
      at ReplaceFilter.Filter (index.js:56:18)
      at new ReplaceFilter (test/helpers/replacer.js:15:10)
      at ReplaceFilter (test/helpers/replacer.js:12:12)
      at node_modules/broccoli-test-helpers/index.js:63:29
      at lib$rsvp$$internal$$initializePromise (node_modules/rsvp/dist/rsvp.js:541:9)
      at new lib$rsvp$promise$$Promise (node_modules/rsvp/dist/rsvp.js:757:9)
      at node_modules/broccoli-test-helpers/index.js:62:12
      at Context.<anonymous> (test/test.js:611:14)

Version 10 of node.js has been released

Version 10 of Node.js (code name Dubnium) has been released! 🎊

To see what happens to your code in Node.js 10, Greenkeeper has created a branch with the following changes:

Added the new Node.js version to your .travis.yml

If you’re interested in upgrading this repo to Node.js 10, you can open a PR with these changes. Please note that this issue is just intended as a friendly reminder and the PR as a possible starting point for getting your code running on Node.js 10.

More information on this issue

Greenkeeper has checked the engines key in any package.json file, the .nvmrc file, and the .travis.yml file, if present.

engines was only updated if it defined a single version, not a range.
.nvmrc was updated to Node.js 10
.travis.yml was only changed if there was a root-level node_js that didn’t already include Node.js 10, such as node or lts/*. In this case, the new version was appended to the list. We didn’t touch job or matrix configurations because these tend to be quite specific and complex, and it’s difficult to infer what the intentions were.

For many simpler .travis.yml configurations, this PR should suffice as-is, but depending on what you’re doing it may require additional work or may not be applicable at all. We’re also aware that you may have good reasons to not update to Node.js 10, which is why this was sent as an issue and not a pull request. Feel free to delete it without comment, I’m a humble robot and won’t feel rejected 🤖

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

Your Greenkeeper Bot 🌴

Documentation for BROCCOLI_PERSISTENT_FILTER_CACHE_ROOT

Can we add BROCCOLI_PERSISTENT_FILTER_CACHE_ROOT to the README?

It would be easier to find a solution when stefanpenner/async-disk-cache#35 happens.
Unfortunately, I didn't see anything in the ember documentation which could point to that issue and indicate how to not have that issue happen.

add concept for writing multiple files for one input

basically to support writing .map files

It is possible today, but with hacks. It would be nicer if it was less hacky.

Add typescript types

Upgrade promise-map-series to async/await version once ready

joliss/promise-map-series#9

outputEncoding = null does not work anymore

ENOENT: no such file or directory

Following the recent updates, rebuilds now seem to be broken in my ember-cli app after a nom:

Output after changing a file (full path abbreviated to … for anonymity):

file changed helpers/in-array.js
ENOENT: no such file or directory, unlink …/Dashboard/tmp/output_path-WDavmjzf.tmp/helpers/in-array.js'
Error: ENOENT: no such file or directory, unlink '…/Dashboard/tmp/output_path-WDavmjzf.tmp/helpers/in-array.js'
    at Error (native)
    at Object.fs.unlinkSync (fs.js:937:18)
    at null.<anonymous> (…/Dashboard/node_modules/ember-suave/node_modules/broccoli-jscs/node_modules/broccoli-persistent-filter/index.js:85:12)
    at …/Dashboard/node_modules/ember-suave/node_modules/broccoli-jscs/node_modules/broccoli-persistent-filter/node_modules/promise-map-series/index.js:11:14
    at lib$rsvp$$internal$$tryCatch (…/Dashboard/node_modules/ember-suave/node_modules/broccoli-jscs/node_modules/broccoli-persistent-filter/node_modules/rsvp/dist/rsvp.js:493:16)
    at lib$rsvp$$internal$$invokeCallback (…/Dashboard/node_modules/ember-suave/node_modules/broccoli-jscs/node_modules/broccoli-persistent-filter/node_modules/rsvp/dist/rsvp.js:505:17)
    at …/Dashboard/node_modules/ember-suave/node_modules/broccoli-jscs/node_modules/broccoli-persistent-filter/node_modules/rsvp/dist/rsvp.js:1001:13
    at lib$rsvp$asap$$flush (…/Dashboard/node_modules/ember-suave/node_modules/broccoli-jscs/node_modules/broccoli-persistent-filter/node_modules/rsvp/dist/rsvp.js:1198:9)
    at doNTCallback0 (node.js:407:9)
    at process._tickCallback (node.js:336:13)

Mac OS X 10.11.1 with Ember CLI 1.13.8.

issue with renames

investigating..

Changelog is outdated

@stefanpenner what changed since v1.2.11?

Unable to recover from initial errors

see ember-cli/ember-cli#6264

Document limitations in Readme

The readme mentions that the persistent cache does not work "on all systems":

It is recommended that persistent re-builds is opt-in by the consumer as it does not currently work on all systems.

As tooling author, I'm trying to decide (a) whether I need to expose this option to my consumers, and (b) if so, when to recommend they disable the persistent cache.

A quick note about what systems have trouble would be helpful here. Doesn't need to be super detailed if that information doesn't exist - even just a "flakey on Windows" would work great.

consider `create` "deopting" to unlink + link

more: ember-cli/ember-cli#5336 (comment)

idea:

Filter.prototype._handleFile = function(relativePath, srcDir, destDir, entry, outputPath, isChange, instrumentation) {
  if (this.canProcessFile(relativePath)) {
    instrumentation.processed++;
    return this.processAndCacheFile(srcDir, destDir, entry, isChange, instrumentation);
  } else {
    instrumentation.linked++;
    if (isChange) {
      fs.unlinkSync(outputPath);
    }
    var srcPath = srcDir + '/' + relativePath;

    // this try/catch exists, so we can do the less expensive more common operation first: merely copy to the target, but if we bet wrong, we can fallback to the remove then the copy. Such "deopt"....
    try {
      symlinkOrCopySync(srcPath, outputPath);
    } catch (e) {
       if (typeof e === 'object' && e !== null && e.name === 'EEXIST') {
         fs.unlinkSync(outputPath); // we guessed wrong, output needs to be unlinked first
         // https://github.com/ember-cli/ember-cli/issues/5336
         symlinkOrCopySync(srcPath, outputPath);
       } else {
         throw e;
       }
     }
  }
};

This would preserve the existing "optimistic" performance semantics, gambling that it is more common to no need to replace on create. But there are legitimate cases where this is required, so we should actually handle it.

Typescript issues with 3.0 release.

Typescript is picking up the .ts files in the package instead of the .d.ts files because they're all in the same directory together. This causes the packages that don't have types to become errors in projects that have noImplicitAny set to true (The rules in a .d.ts file are different with respect to noImplicitAny - it's not enforced there).

I know you don't like it, but it seems like the official recommendation from the typescript maintainers is that either we don't ship .ts files or we ship the compiled output from a different directory than the sources.

The problem with not shipping the .ts files is that sourcemaps won't work right especially with definitionSourcemaps enabled -- even if definition sourcemaps supported inline source maps, it wouldn't work correctly in vscode.

So the options appear to be:

1a. Keep the current directory layout; disable sourcemaps; exclude *.ts from the package.json.
1b. Same as 1a but with inline sourcemaps + sources inline.
2. Change the directory layout; keep sourcemaps enabled and keep *.ts in the package.json.
3. Get official types into 7 packages that broccoli-persistent-filter uses or get official @types packages for them.

My preference is for option 2. Thoughts?

cacheKeyProcessString does not invalidate across rebuilds.

The cacheKeyProcessString() method allows an addon to provide invalidation on a per-file basis, but this invalidation only occurs when performing the initial build against the persistent cache.

The logic for cache invalidation should not depend on which cache is being accessed.

I propose that the cache key value is added to the fs tree entries so that if the key changes, fs-tree-diff considers this to be a "change" for the file forcing it to be reprocessed (probably due to a dependency change that affects the output).

Also sufficient for my needs:

If reading the files in to call cacheKeyProcessString is too onerous, adding a new cache key method that doesn't receive the contents of the file but is called for every update is fine.
Adding an option that when true opts in to the more aggressive cache invalidation.
Invalidating a file during rebuilds as well as from the persistent cache based on an ability to specify the files that a file depends on.

docs

Use glob expression as file dependency

I'm using broccoli-babel-transpiler with a bunch of custom Babel plugins. Among other things, these include a wildcard module resolver to allow for things such as import "tests/**/*.spec" - in other words, my source file depends on a glob expression, not a single file. When a new test is added to the project, or when a test file is removed, the source file containing the wildcard import (presumably /alltests.js or similar) needs to be rebuilt.

Can I achieve this with the current dependencyInvalidation mechanism?

On first reading of the code, I don't believe the current implementation can do this, but a fix should be fairly simple. Storing glob-like dependencies in a separate inverse-map and matching changes against each registered pattern after the regular dependent lookup should be sufficient.

Is this something you would be interested in?

Crypto dependency warning

warning ember-cli-babel > broccoli-babel-transpiler > broccoli-persistent-filter > [email protected]: 
This package is no longer supported. It's now a built-in Node module. If you've depended on crypto, 
you should switch to the one that's built-in.

v1.0.5 seems to return `undefined` for every file.

ember new foo
ember build
cat dist/assets/foo.js
~~~~ snip ~~~~

define('foo/router', function () {

        'use strict';

        undefined

});
define('foo/app', function () {

        'use strict';

        undefined

});

move to broccoli org?

@rwjblue / @SparshithNR / @Turbo87 / @oligriffiths any thoughts/ concerns?

An in-range update of symlink-or-copy is breaking the build 🚨

Version 1.2.0 of symlink-or-copy was just published.

Branch	Build failing 🚨
Dependency	symlink-or-copy
Current Version	1.1.8
Type	dependency

This version is covered by your current version range and after updating it in your project the build failed.

symlink-or-copy is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Status Details

✅ continuous-integration/appveyor/branch AppVeyor build succeeded Details
❌ continuous-integration/travis-ci/push The Travis CI build failed Details

Commits

The new version differs by 7 commits.

dcd548f release v1.2.0 🎉
cc7cff2 Make tests idempotent
d208e8a Merge pull request #36 from garethbosch/master
1afc3e2 Removed uneeded process require that was failing for old versions of node.
0a1849e fix #35
8725dab Merge pull request #32 from bfred-it/patch-1
aa624a1 asynchronoukks -> asynchronous

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

Your Greenkeeper Bot 🌴

I would like https://github.com/stefanpenner/broccoli-persistent-filter/blob/master/lib/strategies/persistent.js#L15-L17 to become:

    this._persistentCache = new AsyncDiskCache(ctx.constructor._persistentCacheKey, {
      compression: 'deflate',
      location: process.env['BROCCOLI_PERSISTENT_FILTER_CACHE_ROOT']
    });

The above would pass a location if that environment variable was set, and otherwise it would use the default (since https://github.com/stefanpenner/async-disk-cache/blob/master/index.js#L81 uses a falsey check on options.location).

Then we could use a path that is in the CI services (both Travis and Circle support these options) cached directory listing, and get much faster CI builds...

broccolijs / broccoli-persistent-filter Goto Github PK

broccoli-persistent-filter's People

Contributors

Stargazers

Watchers

Forkers

broccoli-persistent-filter's Issues

Version 0.3.2 of walk-sync just got published.

The devDependency mocha was updated from 6.0.2 to 6.1.0.

6.1.0 / 2019-04-07

🔒 Security Fixes

🎉 Enhancements

📠 Deprecations

🐛 Fixes

📖 Documentation

🔩 Other

own files we may not want:

deps we may want to investigate:

Version 10 of Node.js (code name Dubnium) has been released! 🎊

Version 1.2.0 of symlink-or-copy was just published.

Recommend Projects

Recommend Topics

Recommend Org

The devDependency mocha was updated from `6.0.2` to `6.1.0`.