aspect-build / rules_js Goto Github PK

View Code? Open in Web Editor NEW

281.0 9.0 96.0 12.31 MB

High-performance Bazel rules for running Node.js tools and building JavaScript projects

Home Page: https://docs.aspect.build/rules/aspect_rules_js

License: Apache License 2.0

Starlark 65.75% JavaScript 19.25% Shell 6.57% Batchfile 0.03% TypeScript 8.34% HTML 0.04% Less 0.03%

bazel bazel-rules

rules_js's Introduction

Bazel rules for JavaScript

This ruleset is a high-performance and npm-compatible Bazel integration for JavaScript.

Lazy: only fetches/installs npm packages needed for the requested build/test targets.
Correct: works seamlessly with node.js module resolution. For example there are no pathMapping issues with TypeScript rootDirs.
Fast: Bazel's sandbox only sees npm packages as directories, not individual files.
Supports npm "workspaces": nested npm packages in a monorepo.

Many companies are successfully building with rules_js. If you're getting value from the project, please let us know! Just comment on our Adoption Discussion.

https://blog.aspect.dev/rulesjs-npm-benchmarks shows benchmarks for fetching, installing, and linking packages under rules_js as well as typical alternatives like npm and yarn.

Google does not fund development of rules_js. If your company benefits, please consider donating to continue development and maintenance work: https://opencollective.com/aspect-build/projects/rules_js

rules_js is just a part of what Aspect provides:

Need help?
- Community support is available on the #javascript channel on Bazel Slack
- This ruleset has commercial support provided by https://aspect.dev.
See our other Bazel rules, especially those built for rules_js:
- rules_ts - Bazel rules for TypeScript
- rules_swc - Bazel rules for swc
- rules_jest - Bazel rules to run tests using Jest
- rules_esbuild - Bazel rules for esbuild JS bundler
- rules_webpack - Bazel rules for Webpack
- rules_rollup - Bazel rules for Rollup - a JavaScript bundler
- rules_jasmine - Bazel rules to run tests using Jasmine
- rules_terser - Bazel rules for Terser - a JavaScript minifier
- rules_cypress - Bazel rules to run tests using Cypress

Bazel compatibility

The ruleset is known to work with:

Bazel 7 using WORKSPACE and Bzlmod (tested on CI)
Bazel 6 using WORKSPACE and Bzlmod (tested on CI)
Bazel 5 using WORKSPACE (no longer tested on CI)

Note

Remote Execution (RBE) requires at least Bazel 6.0.

Known issues

ESM imports escape the runfiles tree and the sandbox due to #362

Installation

From the release you wish to use: https://github.com/aspect-build/rules_js/releases copy the WORKSPACE snippet into your WORKSPACE file.

To use a commit rather than a release, you can point at any SHA of the repo.

For example to use commit abc123:

Replace url = "https://github.com/aspect-build/rules_js/releases/download/v0.1.0/rules_js-v0.1.0.tar.gz" with a GitHub-provided source archive like url = "https://github.com/aspect-build/rules_js/archive/abc123.tar.gz"
Replace strip_prefix = "rules_js-0.1.0" with strip_prefix = "rules_js-abc123"
Update the sha256. The easiest way to do this is to comment out the line, then Bazel will print a message with the correct value.

Note that GitHub source archives don't have a strong guarantee on the sha256 stability, see https://github.blog/2023-02-21-update-on-the-future-stability-of-source-code-archives-and-hashes

Usage

See the documentation in the docs folder.

Examples

Basic usage examples can be found under the examples folder.

Note that the examples also rely on code in the /WORKSPACE file in the root of this repo.

The e2e folder also has a few useful examples such as js_image_layer for containerizing a js_binary and js_run_devserver, a generic rule for running a devserver in watch mode with ibazel.

Larger examples can be found in our bazel-examples repository including:

Relationship to rules_nodejs

rules_js is an alternative to the build_bazel_rules_nodejs Bazel module and accompanying npm packages hosted in https://github.com/bazelbuild/rules_nodejs, which is now unmaintained. All users are recommended to use rules_js instead.

rules_js replaces some parts of bazelbuild/rules_nodejs and re-uses other parts:

Layer	Legacy	Modern
Custom rules	`npm:@bazel/typescript`, etc.	`aspect_rules_ts`, etc.
Package manager and Basic rules	`build_bazel_rules_nodejs`	`aspect_rules_js`
Toolchain and core providers	`rules_nodejs`	`rules_nodejs`

The common layer here is the rules_nodejs Bazel module, documented as the "core" in https://bazelbuild.github.io/rules_nodejs/:

It is currently useful for Bazel Rules developers who want to make their own JavaScript support.

That's what rules_js does! It's a completely different approach to making JS tooling work under Bazel.

First, there's dependency management.

build_bazel_rules_nodejs uses existing package managers by calling npm install or yarn install on a whole package.json.
rules_js uses Bazel's downloader to fetch only the packages needed for the requested targets, then mimics pnpm to lay out a node_modules tree.

Then, there's how a Node.js tool can be executed:

build_bazel_rules_nodejs follows the Bazel idiom: sources in one folder, outputs in another.
rules_js follows the npm idiom: sources and outputs together in a common folder.

There are trade-offs involved here, but we think the rules_js approach is superior for all users, especially those at large scale. Read below for more in-depth discussion of the design differences and trade-offs you should be aware of. Also see the slides for our Bazel eXchange talk

Design

The authors of rules_js spent four years writing and re-writing build_bazel_rules_nodejs. We learned a lot from that project, as well as from discussions with Rush maintainer @octogonz.

There are two core problems:

How do you install third-party dependencies?
How does a running Node.js program resolve those dependencies?

And there's a fundamental trade-off: make it fast and deterministic, or support 100% of existing use cases.

Over the years we tried a number of solutions and each end of the trade-off spectrum.

Installing third-party libraries

Downloading packages should be Bazel's job. It has a full featured remote downloader, with a content-address-cached (confusingly called the "repository cache"). We now mirror pnpm's lock file into starlark code, then use only Bazel repository rules to perform fetches and translate the dependency graph into Bazel's representation.

For historical context, we started thinking about this in February 2021 in a (now outdated) design doc and have been working through the details since then.

Running Node.js programs

Fundamentally, Bazel operates out of a different filesystem layout than Node. Bazel keeps outputs in a distinct tree outside of the sources.

Our first attempt was based on what Yarn PnP and Google-internal Node.js rules do: monkey-patch the implementation of require in NodeJS itself, so that every resolution can be aware of the source/output tree difference. The main downside to this is compatibility: many packages on npm make their own assumptions about how to resolve dependencies without asking the require implementation, and you can't patch them all. Unlike Google, most of us don't want to re-write all the npm packages we use to be compatible.

Our second attempt was essentially to run npm link before running a program, using a runtime linker. This was largely successful at papering over the filesystem layout differences without disrupting execution of programs. However, it required a lot of workarounds anytime a JS tool wanted to be aware of the input and output locations on disk. For example, many tools like react-scripts (the build system used by Create React App aka. CRA) insist on writing their outputs relative to the working directory. Such programs were forced to be run with Bazel's output folder as the working directory, and their sources copied to that location.

rules_js takes a better approach, where we follow that react-scripts-prompted workaround to the extreme. We always run JS tools with the working directory in Bazel's output tree. We can use a pnpm-style layout tool to create a node_modules under bazel-out, and all resolutions naturally work.

This third approach has trade-offs.

The benefit is that very intractable problems like TypeScript's rootDirs just go away. In that example, we filed microsoft/TypeScript#37378 but it probably won't be solved, so many users trip over issues like this and this. Now this just works, plus results like sourcemaps look like users expect: just like they would if the tool had written outputs in the source tree.
The downside is that Bazel rules/macro authors (even genrule authors) must re-path inputs and outputs to account for the working directory under bazel-out, and must ensure that sources are copied there first. This forces users to pass a BAZEL_BINDIR in the environment of every node action. bazelbuild/bazel#15470 suggests a way to improve that, avoiding that imposition on users.

rules_js's People

Stargazers

Watchers

Forkers

thesayyn jbedard aptenodytes-forsteri stoiky gregmagolan kzadorozhny apesternikov alex-torok jfirebaugh kormide duarten mrmeku jialipassion dgp1130 tetsuok longlho vpanta sourcegraph sbussard pedrobarco dpoluyanov rcarcasses matthewjh sallustfire devversion mbrukman rc-glean druglike reddaly lavatoaster bazaglia vaticle dastbe kon72 yoks joac kreempuff gonzojive jjjhhhlll vincer2040 ewianda realtimetodie vors ocrolus aghassi zemnmez gzm0 mattrobmattrob robin-pham troycarlson nosamad julidekadam ddavisso4 jorunfa keegancsmith betaboon taozhou-glean pqn data-enabler tarungunampalli gregjacobs sfc-gh-ptabor snowflake-labs bcmyers archen kylepl vkbansal-rubrik jgao54 twheys lqc phillcunnington vfomin-sc coreyjboyle mvgijssel raopg edbaunton supercairos duhruh dzbarsky hjellek davidzbarsky-at laurencetews mishazharov-forked jackvincentnz shinypb oikarinen ckilpatrick-figma silic0ns0ldier mattem sinimawath mai93 jacobshirley louwers pat-trunk-io hunshcn klandergren

rules_js's Issues

Support ngcc running on 3p packages that don't ship Angular ivy instructions

Global postinstall in translate_pnpm_lock

Gazelle extension for TypeScript

Set appropriate cross-compiling env vars in postinstall action

Similar to how electron documents things

No code coverage when testing with jest

I am trying to add code coverage reports to opened PRs in my projects CI pipelines (using rules_js), but I am just getting empty reports.

I stumbled across this example, which is working for me. However, it is using the old ts_project rule from bazel. I forked and ported it to use rules_js (as that is what I am using), and the resulting setup is again getting empty code coverage reports. Is this a problem with rules_js, am I doing something wrong, or is it something else entirely?

Here is a version where code coverage is working (using old ts_project from bazel): https://github.com/GV94/bazel_jest_test_coverage.
Here is the same code ported to use rules_js (empty code coverage reports): GV94/bazel_jest_test_coverage#1.

Fix translate_pnpm_lock unit tests.

We had translate_npm_lock unit tests which are obsolete now have to be updated for translate_pnpm_lock.

Better error messaging when using execpath and other make expansions

#108 is an example of user error which is understandable, since rules_js sets the working directory differently from typical Bazel actions. We could:

detect use of $(execpath) etc. without ../ and warn ahead-of-time
same but only print after the action fails?
provide different expansion helpers?
modify behavior of them?

Example of building a Vue app

Run npm init vue@latest following https://vuejs.org/guide/quick-start.html#local and then show how to run the existing build with Bazel using rules_js.

Example of "differential loading"

Translate this example https://github.com/bazelbuild/rules_nodejs/tree/stable/examples/webapp to rules_js
which shows how to create an ESM and ES5 bundle and serve the right one to modern browsers.

Expose translate_package_lock as a module_extension

Per
bazelbuild/bazel#14445 (comment)

This would give end-users a way to change to bzlmod with sth like this in their MODULE.bazel

bazel_dep(name="aspect_rules_js", version="x")
# needed for transitive visibility for the extensions to use
bazel_dep(name = "rules_nodejs", version="y")

package_lock = use_extension("@aspect_rules_js//js:extensions.bzl", "package_lock")
package_lock.translate(name = "my_npm", package_lock_json = "//:package-lock.json")
use_repo(package_lock, "my_npm")
# if needed for visibility
use_repo(package_lock, "npm__swc_cli-0.5.1")

then in BUILD.bazel

...
deps = ["@my_npm//@swc/cli"],
# OR
deps = ["@npm__swc_cli-0.5.1"],

The package_lock extension would both create the npm_import rules for every package, and also the my_npm aliases repository (but not with a repositories.bzl helper inside, as bzlmod gives no way to call that helper since extensions run in parallel)

Better error messages for sad paths

https://github.com/aspect-build/rules_js/pull/89/files#r867406427

Ensure transition to pnpm doesn't re-lock versions

If a user runs pnpm install to get a new pnpm lockfile, it will change all their transitive versions, which isn't desirable. The existing npm or yarn lockfile needs to be used as an input.

Code highlight for npm/js_package dependencies

While migrating from rules_nodejs I came across a limitation that affects the developer experience a lot. The previous yarn install rule I used can create and track the node_modules folder as a managed directory on Bazel.

I understand rules_js has another approach, but having the node_modules folder allowed all my external npm dependencies and local libraries (js_library) to be available to IDEs/text editors like VSCode. Right now, all the dependencies are seen as missing since there's no way for them to be found, removing access to definitions, syntax highlight, linters, etc. while coding.

Are there any alternatives or recommended approaches?

Fix test in example for peer deps

#89 (comment)

Add tooling/test to keep .bazelignore up-to-date with where pnpm lock file knows there are node_modules

#195 (comment)

bug: `github.com` packages not supported

We have a transitive dependency on aframe which requires other packages via a github reference. In our pnpm-lock.json that looks like:

  /aframe/1.3.0:
    resolution: {integrity: sha512-f6OQaDf49SzdSxVskJPvPWldOwHpYMsGttmQHhU6FRiASsnyULKG1nqZom9pDuS3fFYEzQVqRMakQ2AhXeLkFg==}
    engines: {node: '>= 4.6.0', npm: '>= 2.15.9'}
    dependencies:
      custom-event-polyfill: 1.0.7
      debug: github.com/ngokevin/debug/ef5f8e66d49ce8bc64c6f282c15f8b7164409e3a
      deep-assign: 2.0.0
      document-register-element: github.com/dmarcos/document-register-element/8ccc532b7f3744be954574caf3072a5fd260ca90
      ...

(caveat: we don't use pnpm ourselves, but this comes from pnpm import on our Node package-lock.json)

With [email protected], bazel sync gets the following error:

Repository rule npm_translate_lock defined at:
  /private/var/tmp/_bazel_vasilios/96c9699731d9b99e83af4f7d36096194/external/aspect_rules_js/npm/npm_import.bzl:32:37: in <toplevel>
ERROR: An error occurred during the fetch of repository 'npm':
   Traceback (most recent call last):
        File "/private/var/tmp/_bazel_vasilios/96c9699731d9b99e83af4f7d36096194/external/aspect_rules_js/npm/private/npm_translate_lock.bzl", line 319, column 33, in _impl
                lockfile = _process_lockfile(rctx)
        File "/private/var/tmp/_bazel_vasilios/96c9699731d9b99e83af4f7d36096194/external/aspect_rules_js/npm/private/npm_translate_lock.bzl", line 145, column 43, in _process_lockfile
                return translate_to_transitive_closure(lockfile, rctx.attr.prod, rctx.attr.dev, rctx.attr.no_optional)
        File "/private/var/tmp/_bazel_vasilios/96c9699731d9b99e83af4f7d36096194/external/aspect_rules_js/npm/private/transitive_closure.bzl", line 86, column 17, in translate_to_transitive_closure
                fail("unsupported package path " + package_path)
Error in fail: unsupported package path github.com/dmarcos/document-register-element/8ccc532b7f3744be954574caf3072a5fd260ca90

This error seems to actually occur when processing the list of packages themselves, which expects first that the package path starts with /, then that it has a resolution.integrity field (these packages have a resolution.tarball field instead).

Presumably, it needs to recognize general paths for packages, and pull them down via the tarball link.

Migration documentation

Question about generated link_js_packages and link_* functions

I've been starting to play with rules_js, and am super excited about where things are going! I've been wondering about what purpose the generated link_js_packages and link_PKG_NAME functions are for.

I've poked around in the internals and see that they generate the targets for running lifecycle scripts and generating the pnpm-style symlinked node_modules directory. What I'm not sure about is why the decision was made to have those targets be generated by a macro function in package where the pnpm lockfile is versus having them exist in each external repository's generated BUILD file.

I have a feeling that there is some limitation or feature that gets solved by doing this, but it isn't super clear what that is. Is it something about locating the node_modules directory to the correct location in the runfiles?

When would you only call one of the link_PKG_NAME functions versus the whole link_js_packages function? Do you have a usecase in mind for only needing to link a subset of the modules?

Apologies if this is already answered somewhere in the docs, but I'm curious and figured it would be good to have this issue open to track adding a note about this in the docs.

Minify our post-install program

I removed minification because it was getting bad character encoding, spewing garbage and making a terminal state corrupt.

This is the commit to revert 4794f1f

[email protected] is the package to add a unit test - it calls node-gyp.

Probably there's some file encoding problem in ncc, or maybe we should run terser on the file outside of ncc to get more control.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Awaiting Schedule

These updates are awaiting their schedule. Click on a checkbox to get an update now.

chore(deps): update rollup (@rollup/plugin-commonjs, @rollup/plugin-json, @rollup/plugin-node-resolve, @rollup/plugin-terser, rollup)
chore(deps): update dependency webpack-bundle-analyzer to v4.10.2

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

chore(deps): update @kubernetes/client-node digest to 93a0070
chore(deps): update @types (@types/archiver, @types/google-protobuf, @types/node, @types/tar-stream)
chore(deps): update patch updates (bazel, bufferutil, platforms, puppeteer, tslib)
chore(deps): update bazel (bazel_skylib, bazel_skylib_gazelle_plugin, io_bazel_rules_go)
chore(deps): update dependency aspect_rules_lint to v0.21.0
chore(deps): update webfactory/ssh-agent action to v0.9.0
fix(deps): update dependency acorn to v8.11.3
fix(deps): update dependency chalk-alt to v5.3.0
Click on this checkbox to rebase all open PRs at once

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

Detected dependencies

bazel

WORKSPACE

rules_nodejs v6.1.0

WORKSPACE.bzlmod

com_grail_bazel_toolchain c65ef7a45907016a754e5bf5bfabac76eb702fd3

js/dev_repositories.bzl

io_bazel_rules_go v0.46.0

bazel_skylib 1.5.0

bazel_skylib_gazelle_plugin 1.5.0

io_bazel_stardoc 0.6.2

buildifier_prebuilt 6.4.0

aspect_rules_lint v0.7.0

com_grail_bazel_toolchain c65ef7a45907016a754e5bf5bfabac76eb702fd3

js/repositories.bzl

bazel_skylib 1.4.1

rules_nodejs 5.8.4

aspect_bazel_lib v1.42.3

bazel_features v0.1.0

bazel-module

MODULE.bazel

aspect_bazel_lib 1.42.3

bazel_features 1.9.0

bazel_skylib 1.5.0

platforms 0.0.5

aspect_rules_lint 0.12.0

bazel_skylib_gazelle_plugin 1.5.0

buildifier_prebuilt 6.4.0

gazelle 0.36.0

rules_go 0.46.0

stardoc 0.6.2

bazelisk

.bazelversion

bazel 7.1.1

github-actions

.github/workflows/ci.yaml

actions/checkout v4

actions/checkout v4

webfactory/ssh-agent v0.8.0

actions/cache v4

.github/workflows/new_issue.yaml

actions/github-script v6

.github/workflows/release.yml

actions/checkout v4

actions/cache v4

softprops/action-gh-release v1

npm

js/private/coverage/bundle/package.json

c8 7.13.0

@rollup/plugin-commonjs 23.0.2

@rollup/plugin-json 5.0.1

@rollup/plugin-node-resolve 15.0.1

rollup 3.2.5

js/private/image/package.json

tar-stream 3.0.0

rollup 2.79.1

@rollup/plugin-commonjs 23.0.0

@rollup/plugin-node-resolve 15.0.0

@rollup/plugin-typescript 10.0.1

@types/node 18.11.11

@types/tar-stream 2.2.2

@types/archiver 5.3.1

typescript 4.9.5

tslib 2.6.0

js/private/test/image/package.json

acorn 8.8.2

js/private/test/js_run_devserver/package.json

@types/node 16.18.11

jasmine 5.1.0

js/private/worker/src/package.json

google-protobuf 3.21.2

rollup 2.79.1

@rollup/plugin-commonjs 23.0.0

@rollup/plugin-node-resolve 15.0.0

@rollup/plugin-json 5.0.0

@types/google-protobuf 3.15.6

@rollup/plugin-typescript 10.0.1

@rollup/plugin-terser 0.2.0

@types/node 18.11.18

typescript 4.9.5

tslib 2.6.0

abortcontroller-polyfill 1.7.5

npm/private/lifecycle/package.json

@pnpm/lifecycle 14.1.7

@pnpm/logger 5.0.0

@rollup/plugin-commonjs ^24.1.0-0

@rollup/plugin-json ^6.0.0

@rollup/plugin-node-resolve ^15.0.1

@rollup/plugin-replace ^5.0.2

npm/private/test/npm_package/package.json

chalk 5.0.1

chalk-alt 5.1.1

npm/private/test/npm_package_publish/package.json

npm/private/test/package.json

@figma/nodegit ^0.28.0-figma.2

@kubernetes/client-node fc681991e61c6808dd26012a2331f83671a11218

@plotly/regl 2.1.2

bufferutil 4.0.7

debug 9742c5f383a6f8046241920156236ade8ec30d53

esbuild 0.14.38

hot-shots 10.0.0

json-stable-stringify 1.0.1

node-gyp 9.3.0

plotly.js 2.25.2

protoc-gen-grpc be5580b06348d3eb9b4610a4a94065154a0df41f

puppeteer 19.11.0

regl 2.1.2

segfault-handler 1.3.0

semver-first-satisfied 1.1.0

syncpack c245af8ea73ce3345d92bbda6c684092a841e262

typescript *

webpack-bundle-analyzer 4.5.0

npm/private/test/vendored/is-odd/package.json

is-number ^6.0.0

gulp-format-md ^1.0.0

mocha ^3.5.3

node >=4

npm/private/test/vendored/semver-max/package.json

semver ^5.0.1

is-odd 3.0.1

tape ^4.0.1

package.json

@types/node 16.18.11

chalk 5.1.1

inline-fixtures 1.1.0

jsonpath-plus 7.2.0

typescript 4.9.5

Check this box to trigger a request for Renovate to run again on this repository

js_binary does not work on Windows

I'm trying to get rules_swc working on Windows, and have reached the point where it fails inside js_binary:

    INFO: Invocation ID: d3e53c76-a957-42a2-b151-46337966704c
ERROR: C:/users/mhill/_bazel_mhill/bqmuzrid/external/aspect_rules_swc/swc/BUILD.bazel:56:10: in js_binary rule @aspect_rules_swc//swc:cli:
Traceback (most recent call last):
        File "C:/users/mhill/_bazel_mhill/bqmuzrid/external/aspect_rules_js/js/private/js_binary.bzl", line 210, column 32, in _js_binary_impl
                launcher = _create_launcher(ctx)
        File "C:/users/mhill/_bazel_mhill/bqmuzrid/external/aspect_rules_js/js/private/js_binary.bzl", line 193, column 70, in _create_launcher
                launcher = create_windows_native_launcher_script(ctx, ctx.outputs.launcher_sh) if ctx.attr.is_windows else bash_launcher
Error: No attribute 'launcher_sh' in outputs. Make sure you declared a rule output with this name.
ERROR: C:/users/mhill/_bazel_mhill/bqmuzrid/external/aspect_rules_swc/swc/BUILD.bazel:56:10: Analysis of target '@aspect_rules_swc//swc:cli' failed

Sure enough, there is no launcher_sh attribute:

https://github.com/aspect-build/rules_js/blob/main/js/private/js_binary.bzl

bug: `file:` package protocol not supported

Admittedly, this might be more WAI then a bug, and we can likely work around it, but filing as a bug in case y'all decide it's a necessary feature.

We have local packages which we reference via a file: protocol. When we have them in our pnpm-lock.yaml and execute a bazel sync with [email protected], we get the following error:

ERROR: An error occurred during the fetch of repository 'npm':
   Traceback (most recent call last):
        File "/private/var/tmp/_bazel_vasilios/96c9699731d9b99e83af4f7d36096194/external/aspect_rules_js/npm/private/npm_translate_lock.bzl", line 319, column 33, in _impl
                lockfile = _process_lockfile(rctx)
        File "/private/var/tmp/_bazel_vasilios/96c9699731d9b99e83af4f7d36096194/external/aspect_rules_js/npm/private/npm_translate_lock.bzl", line 145, column 43, in _process_lockfile
                return translate_to_transitive_closure(lockfile, rctx.attr.prod, rctx.attr.dev, rctx.attr.no_optional)
        File "/private/var/tmp/_bazel_vasilios/96c9699731d9b99e83af4f7d36096194/external/aspect_rules_js/npm/private/transitive_closure.bzl", line 86, column 17, in translate_to_transitive_closure
                fail("unsupported package path " + package_path)
Error in fail: unsupported package path file:packages-dist/admin-bunsen

Similar to #166, the error here is in recognition of the listed packages, which in this case begin with file: and have a resolution.directory entry. We can/will prune these from the pnpm-lock.yaml, as we'd prefer local packages to reference each other via local bazel targets. But it is possibly worth an explicit comment that local packages are not allowed, or some feature allowing them to be ignored when processing pnpm-lock.yaml?

Example of qwik.js

https://www.youtube.com/watch?v=0dC11DMR3fU

Example of building a React app

Start from latest create-react-app and then wrap the supplied build system with Bazel using rules_js.

Demonstrate how to build a container

Should be able to build a docker container with the right nodejs binary, a layer for npm deps (a pnpm store?) and the application code

Help a user who forgot to update their pnpm-lock.yaml

We could have some repository rule to re-run pnpm install --lockfile-only and verify that the users file is up-to-date.

Otherwise you might add a first-party dep to your package.json and then Bazel doesn't see that change.

js rules should be hermetic and not follow symlinks out of the sandbox

I noticed some of my uses of js_binary were accidentally escaping the sandbox through __dirname. For consistency with nodejs_binary and per previous discussions in rules_nodejs (e.g. bazelbuild/rules_nodejs#109), --preserve-symlinks should be the default.

support pnpm workspaces

This should also provide a gazelle extension to mirror the data from the tree of package.json files.

Homepage at docs.aspect.build/rules_js

Dynamic linker

It should be possible for an action that has user-provided deps on an npm package to have the binary resolve those with normal node resolve

For example, a typescript type-check action in rules_js currently fails because it can't resolve @types/node.

generated bin doesn't work in subpackage

third_party/npm/BUILD

load("@npm//npm-check:package_json.bzl", "bin")

exports_files(["package.json"])

bin.npm_check_binary(
    name = "check",
    data = [
        "//third_party/npm:package.json",
    ],
    args = [
        "--no-color",
        "--no-emoji",
        "--save-exact",
        "--skip-unused",
        "third_party/npm",
    ],
)

Then you get a bad error message about "must be a treeartifact" but actually the label does'nt exist at all

alexeagle@system76-pc:~/Projects/net_$ bazel build third_party/npm:all
ERROR: /home/alexeagle/Projects/net_/third_party/npm/BUILD:5:21: in directory_path rule //third_party/npm:check__entry_point: 
Traceback (most recent call last):
	File "/home/alexeagle/.cache/bazel/_bazel_alexeagle/b027b6729610d26a644911de4dac9747/external/aspect_bazel_lib/lib/private/directory_path.bzl", line 17, column 13, in _directory_path
		fail("directory attribute must be created with Bazel declare_directory (TreeArtifact)")
Error in fail: directory attribute must be created with Bazel declare_directory (TreeArtifact)
ERROR: Analysis of target '//third_party/npm:check__entry_point' failed; build aborted: Analysis of target '//third_party/npm:check__entry_point' failed
FAILED: Build did NOT complete successfully (6 packages loaded, 8 targets configured)
    currently loading: third_party/python ... (4 packages)
    Fetching @go_sdk; fetching

alexeagle@system76-pc:~/Projects/net_$ bazel query --output=build //third_party/npm:check__entry_point
# /home/alexeagle/Projects/net_/third_party/npm/BUILD:5:21
directory_path(
  name = "check__entry_point",
  generator_name = "check",
  generator_function = "npm_check_binary",
  generator_location = "third_party/npm/BUILD:5:21",
  directory = "//third_party/npm:jsp__npm-check__5.9.2__dir",
  path = "bin/cli.js",
)

# Rule check__entry_point instantiated at (most recent call last):
#   /home/alexeagle/Projects/net_/third_party/npm/BUILD:5:21                                                                       in <toplevel>
#   /home/alexeagle/.cache/bazel/_bazel_alexeagle/b027b6729610d26a644911de4dac9747/external/npm__npm-check__5.9.2/package_json.bzl:36:20 in npm_check_binary
# Rule directory_path defined at (most recent call last):
#   /home/alexeagle/.cache/bazel/_bazel_alexeagle/b027b6729610d26a644911de4dac9747/external/aspect_bazel_lib/lib/private/directory_path.bzl:20:22 in <toplevel>

alexeagle@system76-pc:~/Projects/net_$ bazel query --output=build //third_party/npm:jsp__npm-check__5.9.2__dir
[no result]

Generated bin should have the package in data by default

Users have to manually repeat the package in data right now. For example, the following example fails with this confusing error

==================== Test output for //example:lint:
node:internal/modules/cjs/loader:936
  throw err;
  ^

Error: Cannot find module 'v8-compile-cache'
Require stack:
- /home/alexeagle/.cache/bazel/_bazel_alexeagle/581b2ac03dd093577e8a6ba6b6509be5/execroot/aspect_rules_js/bazel-out/k8-fastbuild/bin/example/node_modules/.aspect_rules_js/[email protected]/node_modules/eslint/bin/eslint.js
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:933:15)
    at Function.Module._load (node:internal/modules/cjs/loader:778:27)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at require (node:internal/modules/cjs/helpers:94:18)
    at Object.<anonymous> (/home/alexeagle/.cache/bazel/_bazel_alexeagle/581b2ac03dd093577e8a6ba6b6509be5/execroot/aspect_rules_js/bazel-out/k8-fastbuild/bin/example/node_modules/.aspect_rules_js/[email protected]/node_modules/eslint/bin/eslint.js:13:1)
    at Module._compile (node:internal/modules/cjs/loader:1101:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:79:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/home/alexeagle/.cache/bazel/_bazel_alexeagle/581b2ac03dd093577e8a6ba6b6509be5/execroot/aspect_rules_js/bazel-out/k8-fastbuild/bin/example/node_modules/.aspect_rules_js/[email protected]/node_modules/eslint/bin/eslint.js'
  ]
}

unless you uncomment the data line

load("@npm//:defs.bzl", "link_js_packages")
load("@npm//eslint:package_json.bzl", "bin")

link_js_packages()

bin.eslint_test(
    name = "lint",
#     data = ["@npm//eslint"],
)

(I found it especially confusing because the error wasn't about locating the eslint entry point, rather one of its unlinked dependencies)

Example of building a next.js app

Start from a copy of https://github.com/vercel/next.js/tree/v12.1.6/examples/blog-starter-typescript and get the existing build system supplied by next to work when wrapped by Bazel.

Shorthand for all packages in an npm namespace (e.g. `@npm//@types`)

With rules_nodejs, I was sometimes using deps = ["@npm//@types", ...] in ts_project as a convenient way to depend on all @types packages in the repository. Are there any plans to support that in rules_js as well?

bug: aliased transitive dependency names do not work

We have a dependency on plotly.js, which uses an aliased version of one of its dependencies:

https://github.com/plotly/plotly.js/blob/e0a5f13a4c9b8e2f74204499168d12b2d0bab285/package.json#L110

When adding a dependency like this, the pnpm lockfile has a version marker that matches the pnpm name:

  /plotly.js/2.12.1:
    resolution: {integrity: sha512-XbZ3w3jSl+ihZPMXWwblmjinqbzdd+vT+3XKZpyffxmlMmTp/3vY1ewpQDuNyufUoiT0o0ekCqa80Qlzi7jwlA==}
    dependencies:
      ...
      probe-image-size: 7.2.3
      regl: /@plotly/regl/2.1.2
      regl-error2d: 2.0.12

The issue presents itself first as an error with iterating through the deps in transitive_dependencies.bzl:

ERROR: An error occurred during the fetch of repository 'npm':
   Traceback (most recent call last):
        File "/home/alex.torok/.cache/bazel/_bazel_alex.torok/080e6abb1a3d3304a3cf19fc0f4ef9c1/external/aspect_rules_js/npm/private/npm_translate_lock.bzl", line 319, column 33, in _impl
                lockfile = _process_lockfile(rctx)
        File "/home/alex.torok/.cache/bazel/_bazel_alex.torok/080e6abb1a3d3304a3cf19fc0f4ef9c1/external/aspect_rules_js/npm/private/npm_translate_lock.bzl", line 145, column 43, in _process_lockfile
                return translate_to_transitive_closure(lockfile, rctx.attr.prod, rctx.attr.dev, rctx.attr.no_optional)
        File "/home/alex.torok/.cache/bazel/_bazel_alex.torok/080e6abb1a3d3304a3cf19fc0f4ef9c1/external/aspect_rules_js/npm/private/transitive_closure.bzl", line 114, column 34, in translate_to_transitive_closure
                gather_transitive_closure(
        File "/home/alex.torok/.cache/bazel/_bazel_alex.torok/080e6abb1a3d3304a3cf19fc0f4ef9c1/external/aspect_rules_js/npm/private/transitive_closure.bzl", line 34, column 36, in gather_transitive_closure
                package_info = packages[utils.pnpm_name(name, version)]
Error: key "regl//@plotly/regl/2.1.2" not found in dictionary

I thought an easy fix could be to handle the version number differently if we detect a full path:

@@ -27,6 +29,9 @@ def _strip_peer_dep_version(version):
diff --git a/npm/private/utils.bzl b/npm/private/utils.bzl
index b05d467..2289ac3 100644
--- a/npm/private/utils.bzl
+++ b/npm/private/utils.bzl
 def _pnpm_name(name, version):
     "Make a name/version pnpm-style name for a package name and version"
+    if version.startswith("/"):
+        # Strip leading '/'
+        return version[1:]
     return "%s/%s" % (name, version)

This seems to address the issue seen when iterating over the transitive deps, but then I hit an issue with the link package names not being correct:

ERROR: /home/alex.torok/code/av/BUILD.bazel:8:22: //:store_link__regl___@plotly_regl_2.1.2__pkg: missing input file '//:store_link__regl___@plotly_regl_2.1.2__pkg'
ERROR: /home/alex.torok/code/av/BUILD.bazel:8:22: 1 input file(s) do not exist

Looking in my external repo directory, it appears that the plotly.js does not gain a dep on the correct link target:

$ rg store_link__regl___@ npm*__links/
npm__react-plotly.js__2.5.1__plotly.js_2.12.1_react_16.14.0__links/defs.bzl
85:        "store_link__regl___@plotly_regl_2.1.2__pkg",
379:        "store_link__regl___@plotly_regl_2.1.2__pkg",

npm__plotly.js__2.12.1__links/defs.bzl
78:        "store_link__regl___@plotly_regl_2.1.2__pkg",
366:        "store_link__regl___@plotly_regl_2.1.2__pkg",
648:        "store_link__regl___@plotly_regl_2.1.2__ref",

I got to this point and feel like the fix for this is over my head, so I'm hoping someone with more familiarity of the whole data flow from lockfile parsing to generated link targets can help out.

Add .publish run target to npm_package

Should be able to build a package and publish it to some registry

Support bzlmod in rules_js family of repos

Tracking issue:

Example of building Angular with architect

Make a new fresh Angular 14 app using ng new and then wrap the architect-cli to build under Bazel.

Example of protocol buffers and gRPC

Lots of potential integrations that could work:

https://github.com/thesayyn/protoc-gen-ts#with-bazel might be a place to start :)
we could translate https://github.com/bazelbuild/rules_nodejs/tree/stable/examples/protobufjs
see what https://github.com/stackb/rules_proto would look like
check in with buf.build

translate_package_lock should accept postinstall attribute dict

Wrong version of types_node

in rules_jest

bazel query --output=build @jest//:jsp__at_types_graceful-fs__4.1.5__pkg
# /home/alexeagle/.cache/bazel/_bazel_alexeagle/10a4f667f42bb2eb78f9c5135e392727/external/jest/BUILD.bazel:5:17
link_js_package_store_internal(
  name = "jsp__at_types_graceful-fs__4.1.5__pkg",
  generator_name = "jsp__at_types_graceful-fs__4.1.5__pkg",
  generator_function = "link_js_packages",
  generator_location = "/home/alexeagle/.cache/bazel/_bazel_alexeagle/10a4f667f42bb2eb78f9c5135e392727/external/jest/BUILD.bazel:5:17",
  src = "@npm__at_types_graceful-fs__4.1.5//:jsp_source_directory",
  deps = ["@jest//:jsp__at_types_node__17.0.35__ref"],
  package = "@types/graceful-fs",
  version = "4.1.5",
)

but in a client project it somehow gets a wrong version of @types/node

bazel query --output=build @jest//:jsp__at_types_graceful-fs__4.1.5__pkg
# /home/alexeagle/.cache/bazel/_bazel_alexeagle/b027b6729610d26a644911de4dac9747/external/jest/BUILD.bazel:5:17
link_js_package_store_internal(
  name = "jsp__at_types_graceful-fs__4.1.5__pkg",
  generator_name = "jsp__at_types_graceful-fs__4.1.5__pkg",
  generator_function = "link_js_packages",
  generator_location = "/home/alexeagle/.cache/bazel/_bazel_alexeagle/b027b6729610d26a644911de4dac9747/external/jest/BUILD.bazel:5:17",
  src = "@npm__at_types_graceful-fs__4.1.5//:jsp_source_directory",
  deps = ["@jest//:jsp__at_types_node__16.4.1__ref"],
  package = "@types/graceful-fs",
  version = "4.1.5",
)

and so it fails

ERROR: /home/alexeagle/.cache/bazel/_bazel_alexeagle/b027b6729610d26a644911de4dac9747/external/jest/BUILD.bazel:5:17: in deps attribute of link_js_package_store_internal rule @jest//:jsp__at_types_graceful-fs__4.1.5__pkg: target '@jest//:jsp__at_types_node__16.4.1__ref' does not exist. Since this rule was created by the macro 'link_js_packages', the error might have been caused by the macro implementation

I don't see how a version can be different since they use the same jest/private/v28.1.0/repositories.bzl

Npm alias protocol does not resolve

As per https://yarnpkg.com/features/protocols a package name may have an alias to another package in the form of:

"react-intl-cdo": "npm:[email protected]",

This in pnpm-lock.yaml transforms to react-intl-cdo: /react-intl/[email protected][email protected] and will fail inside js/private/translate_pnpm_lock.js with the error below because the pnpmName will be react-intl-cdo//react-intl/[email protected][email protected]

(node:54092) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'dependencies' of undefined
    at gatherTransitiveClosure 
(/Users/mstoichi/code/github.com/rules_js/js/private/translate_pnpm_lock.js:52:34)
    at main (/Users/mstoichi/code/github.com/rules_js/js/private/translate_pnpm_lock.js:188:9)

npm package remark failed to fetch

repro: https://github.com/aspect-build/bazel-examples/pull/new/next

% bazel test ...
INFO: Repository npm__stringify-entities__4.0.2 instantiated at:
  /home/alexeagle/Projects/bazel-examples/next.js/WORKSPACE.bazel:30:17: in <toplevel>
  /home/alexeagle/.cache/bazel/_bazel_alexeagle/1112fa9a04c6878cb3f2bf123baf106a/external/npm/repositories.bzl:3979:15: in npm_repositories
  /home/alexeagle/.cache/bazel/_bazel_alexeagle/1112fa9a04c6878cb3f2bf123baf106a/external/aspect_rules_js/js/npm_import.bzl:150:16: in npm_import
Repository rule npm_import defined at:
  /home/alexeagle/.cache/bazel/_bazel_alexeagle/1112fa9a04c6878cb3f2bf123baf106a/external/aspect_rules_js/js/private/npm_import.bzl:493:29: in <toplevel>
ERROR: An error occurred during the fetch of repository 'npm__stringify-entities__4.0.2':
   Traceback (most recent call last):
        File "/home/alexeagle/.cache/bazel/_bazel_alexeagle/1112fa9a04c6878cb3f2bf123baf106a/external/aspect_rules_js/js/private/npm_import.bzl", line 274, column 27, in _impl
                pkg_json = json.decode(rctx.read(pkg_json_path))
Error in decode: at offset 2312, object has duplicate key: "typeCoverage"
ERROR: /home/alexeagle/Projects/bazel-examples/next.js/WORKSPACE.bazel:30:17: fetching npm_import rule //external:npm__stringify-entities__4.0.2: Traceback (most recent call last):
        File "/home/alexeagle/.cache/bazel/_bazel_alexeagle/1112fa9a04c6878cb3f2bf123baf106a/external/aspect_rules_js/js/private/npm_import.bzl", line 274, column 27, in _impl
                pkg_json = json.decode(rctx.read(pkg_json_path))
Error in decode: at offset 2312, object has duplicate key: "typeCoverage"

ERROR: /home/alexeagle/Projects/bazel-examples/next.js/BUILD.bazel:4:17: //:store_link__stringify-entities__4.0.2__pkg depends on @npm__stringify-entities__4.0.2//:jsp_source_directory in repository @npm__stringify-entities__4.0.2 which failed to fetch. no such package '@npm__stringify-entities__4.0.2//': at offset 2312, object has duplicate key: "typeCoverage"
ERROR: Analysis of target '//lib:lib' failed; build aborted: 
INFO: Elapsed time: 0.347s
INFO: 0 processes.
ERROR: Couldn't start the build. Unable to run tests
INFO: Build Event Protocol files produced successfully.
FAILED: Build did NOT complete successfully (7 packages loaded, 21 targets configured)

Allow for translate_package_lock to store result in the repo

Rather than dynamically create @some_external_repo//:repositories.bzl for the user to then load() and call from their WORKSPACE, we could act like rules_go's update-repos command.

We would write the npm_import calls into the WORKSPACE or some designated helper macro.

This has a couple advantages:

user can hand-modify the result
WORKSPACE install is simpler so if you're a ruleset you might want to make your setup harder in exchange for making users setup easier
Workaround for bazelbuild/bazel#14445 (not sure yet if that's a bug/missing feature in bzlmod)

npm package lifecycle [postinstall]

Some packages rely on running some code after the fetch.

https://github.com/npm/run-script might be a place to start. pnpm may also have some useful bits for this (in the case where packages live apart from each other)

rules_rollup

npm packages Docker integration

I'm wondering what are the current limitations to build a Docker image out of targets created using rules_js. I'm able to create a tar from the ts_project rule or swc from rules_swc, but I'm struggling with bundling the npm packages.

I've tried to use the container_layer rule to create a npm_deps layer that is copied to the /node_modules of my image:

container_layer(
    name = "npm_deps",
    data_path = "node_modules",
    files = [
        "@npm//axios",
        "@npm//@types/node",
    ],
)

The problem is, indirect dependencies end up exclusively inside .aspect_rules_js (follow-redirects is an indirect dependency of axios):

/node_modules # tree -L 2 -a
.
├── .aspect_rules_js
│   ├── @[email protected]
│   ├── [email protected]
│   └── [email protected]
├── @types
│   └── node
└── axios
    ├── CHANGELOG.md
    ├── LICENSE
    ├── README.md
    ├── SECURITY.md
    ├── UPGRADE_GUIDE.md
    ├── dist
    ├── index.d.ts
    ├── index.js
    ├── lib
    ├── package.json
    ├── tsconfig.json
    └── tslint.json

In the following example, lodash.camelcase is an indirect dependency of an external npm package. When running my Docker container, it fails with:

node:internal/modules/cjs/loader:936
  throw err;
  ^

Error: Cannot find module 'lodash.camelcase'

That makes sense since one of the npm packages tried to import it but it isn't linked (but is available inside .aspect_rules_js). Is there a way to get the indirect dependencies also linked to the node_modules?

As a side note, symlinks seem to become the whole directory copied, so there are three axios and two follow-redirects (I'm limiting the tree to 2 subdirectories, but inside the node_modules of packages in .aspect_rules_js the duplication happens).

If that app is a good reference then it should be using rules_js.
Also it's a chance to help out buildbuddy and find more bugs before our 1.0.0 launch