Giter Club home page Giter Club logo

resolve-dependencies's Introduction

Build Status Downloads Version License

Install: npm i nexe -g

Nexe is a command-line utility that compiles your Node.js application into a single executable file.

Motivation and Features

  • Self contained applications
  • Ability to run multiple applications with different node.js runtimes.
  • Distribute binaries without needing node / npm.
  • Idempotent builds
  • Start and deploy faster.
  • Lockdown specific application versions, and easily rollback.
  • Flexible build pipeline
  • Cross platform builds

Usage

  • Application entrypoint:

    nexe my-app.js

  • stdin interface

    rollup -c | nexe --resource "./public/**/*" -o my-app.exe

For more CLI options see: nexe --help

Examples

  • nexe server.js -r "public/**/*.html"
  • nexe --build
  • nexe -t x86-8.0.0

Resources

Additional files or resources can be added to the binary by passing -r "glob/pattern/**/*". These included files can be read in the application by using fs.readFile or fs.readFileSync.

Compiling the nexe Executable

By default nexe will attempt to download a pre-built executable. These are listed on the releases page. The exact version you want may be unavailable or you may want to customize what is built. See nexe --help for a list of options available when passing the --build option. You will also need to ensure your environment is setup to build node. Note: the python binary in your path should be an acceptable version of python 3; you can create a symlink or use the --python parameter (e.g. nexe --build --python=$(which python3)).

Linux and macOS

Prerequisites & details

Windows

The fastest and most reliable way to get started is simply to run the commands below. If you'd rather read the details or perform a manual install of the prerequisites, you can find that here.

The instructions below are the fastest and most reliable method. Run the following sets of commands with PowerShell (running as Administrator).

Install all required build tools (and dependencies):

Set-ExecutionPolicy Unrestricted -Force
iex ((New-Object System.Net.WebClient).DownloadString('https://boxstarter.org/bootstrapper.ps1'))
get-boxstarter -Force
Install-BoxstarterPackage https://raw.githubusercontent.com/nodejs/node/master/tools/bootstrap/windows_boxstarter -DisableReboots

Set config:

npm config set msvs_version 2019
npm config set python python3.8

Where 2019 is the version of Visual Studio you have (if you have it).

Notes:

  • The above works and has been tested with node.js 14.5.4 and 15.8.0
  • Python 3 and Python 2 can coexist and nexe will still work, considering the set config area above
  • Don't use npm install windows-build-tools unless you're having some type of issue, because the above commands configures and installs the latest/preferred too.

Node.js API

Example

const { compile } = require('nexe')

compile({
  input: './my-app.js',
  build: true, //required to use patches
  patches: [
    async (compiler, next) => {
      await compiler.setFileContentsAsync(
        'lib/new-native-module.js',
        'module.exports = 42'
      )
      return next()
    }
  ]
}).then(() => {
  console.log('success')
})

NexeOptions

options: object

  • input: string

    • Input bundle file path
    • default: stdin or the current directory's main file (package.json)
  • output: string

    • Output executable file path
    • default: same as name with an OS specific extension.
  • target: string | object

    • An object or string describing platform-arch-version. e.g. 'windows-ia32-10.13.0'
      • each segment is optional, and will be merged with the current environment

      • Examples: (full list)

        • 'win32-x86-10.13.0
        • { platform: 'alpine' }
        • darwin-10.13.0
        • linux-x64
        • macos-10.13.0

        See test/target.spec.ts

    • If the build flag is set, the platform portion of the target is ignored.
    • default: process
  • bundle: string | boolean

    • If a string is provided it must be a valid relative module path and should provide an export with the following signature:
    export function createBundle (options: NexeOptions): Promise<string>
    • default: true
  • name: string

    • Module friendly name of the application
    • default: basename of the input file, or nexe_${Date.now()}
  • cwd: string

    • Directory nexe will operate on as though it is the cwd
    • default: process.cwd()
  • mangle: boolean

    • If set to false, nexe will not include the virtual filesystem (your application and resources) on the output.
    • This will cause the output to error as an "Invalid Binary" unless a userland patch alters the contents of lib/_third_party_main.js in the nodejs source.
    • default: true
  • build: boolean

    • Build node from source, passing this flag tells nexe to download and build from source. Subsequently using this flag will cause nexe to use the previously built binary. To rebuild, first add --clean
  • remote: string

    • Provide a custom remote location for fetching pre-built nexe binaries from. This can either be an HTTP or HTTPS URL.
    • default: null
  • asset: string

    • Provide a pre-built nexe binary asset, this is a file path is resolved relative to cwd.
  • python: string

    • On Linux this is the path pointing to your python3 executable
    • On Windows this is the directory where python can be accessed
    • default: null
  • flags: string[]

    • Array of node runtime flags to build node with.
    • Example: ['--expose-gc']
    • default: []
  • configure: string[]

    • Array of arguments for the node build configure step
    • Example: ['--with-dtrace', '--dest-cpu=x64']
    • default: []
  • make: string[]

    • Array of arguments for the node build make step
    • default: []
  • vcBuild: string[]

    • Options for windows build
    • default: ['nosign', 'release']
  • snapshot: string

    • path to a file to be used as the warmup snapshot for the build
    • default: null
  • resources: string[]

    • Array of globs with files to include in the build
    • Example: ['./public/**/*']
    • default: []
  • temp: string

    • Path to use for storing nexe's build files
    • Override in the env with NEXE_TEMP
    • default: ~/.nexe
  • ico: string

    • Path to a user provided icon to be used (Windows only). Requires --build to be set.
  • rc: object

    • Settings for patching the node.rc configuration file (Windows only).
    • Example (keys may vary depending on the version. Reference the file linked above):
        {
          CompanyName: "ACME Corp",
          PRODUCTVERSION: "17,3,0,0",
          FILEVERSION: "1,2,3,4"
          ...
        }
    • default: {}
  • clean: boolean

    • If included, nexe will remove temporary files for the accompanying configuration and exit
  • enableNodeCli: boolean

    • Enable the original Node CLI (will prevent application cli from working).
    • Node CLI arguments passed via the NODE_OPTIONS environment variable will still be processed. NODE_OPTIONS support can be disabled with the --without-node-options configure flag.
    • default: false
  • fakeArgv: boolean

    • fake the entry point file name (process.argv[1]). If nexe was used with stdin this will be '[stdin]'.
  • ghToken: string

    • Provide a Github Token for accessing nexe releases
    • This is usually needed in CI environments
    • default: process.env.GITHUB_TOKEN
  • sourceUrl: string

    • Provide an alternate url for the node source code
    • Note: temporary files will still be created for this under the specified version
  • loglevel: string

    • Set the loglevel, info, silent, or verbose
    • default: 'info'
  • patches: NexePatch[]

    • Userland patches for patching or modifying node source
    • default: []
  • plugins: NexePatch[]

    • Userland plugins for modifying nexe executable behavior
    • default: []

NexePatch: (compiler: NexeCompiler, next: () => Promise<void>) => Promise<void>

Patches and Plugins are just a middleware functions that take two arguments, the compiler, and next. The compiler is described below, and next ensures that the pipeline continues. Its invocation should always be awaited or returned to ensure correct behavior. Patches also require that --build be set, while plugins do not.

For examples, see the built in patches: src/patches.

NexeCompiler

  • setFileContentsAsync(filename: string, contents: string): Promise<void>
    • Quickly set a file's contents within the downloaded Node.js source.
  • replaceInFileAsync(filename: string, ...replaceArgs): Promise<void>
    • Quickly perform a replace in a file within the downloaded Node.js source. The rest arguments are passed along to String.prototype.replace
  • readFileAsync(filename: string): Promise<NexeFile>
    • Access (or create) a file within the downloaded Node.js source.
  • addResource(filename: string, contents: Buffer): Promise<void>
    • Add a resource to the nexe bundle
  • files: NexeFile[]
    • The cache of the currently read, modified, or created files within the downloaded Node.js source.

NexeFile

  • contents: string
  • absPath: string
  • filename: string

Any modifications made to NexeFile#contents will be maintained in the cache without the need to explicitly write them back out, e.g. using NexeCompiler#setFileContentsAsync.

Native Modules

In order to use native modules, the native binaries must be shipped alongside the binary generated by nexe.

Troubleshooting

Error: Entry file "" not found! means you need to provide nexe with input. Either use -i or pipe data to it.

Error: https://github.com/nexe/nexe/releases/download/v3.3.3/windows-x64-15.8.0 is not available, create it using the --build flag or similar message means that it either:

  • You are having networking issues such as the download being blocked
  • You should specify the target so nexe knows what version of the executable to use.
    • See the releases page to find the executable's version number
    • Example
      • nexe -i "app.js" -r "public/**/*.html" -o "dist/myApp.exe" -t x64-14.15.3
      • where -i specifies the input, -r specifies resources to embed, -o specifies the output, -t specifies the target.
    • Alternatively you can compile the executable yourself, see that section for details

Contributing

Building

$ git clone [email protected]:nexe/nexe.git
$ cd nexe
$ npm i && npm run build

Testing

$ npm test

resolve-dependencies's People

Contributors

calebboyd avatar msftenhanceprovenance avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

resolve-dependencies's Issues

TypeError [ERR_INVALID_ARG_VALUE]: The argument 'path' must be a string or Uint8Array without null bytes

- Starting...
ℹ nexe 4.0.0-beta.15
✔ nexe 4.0.0-beta.15
- Finished in 1.976s
✔ Finished in 1.976s
TypeError [ERR_INVALID_ARG_VALUE]: The argument 'path' must be a string or Uint8Array without null bytes. Received '/zigbee2mqtt/node_modules/es5-ext/string/\x00#/contains/index.js'
    at Object.openSync (fs.js:468:10)
    at readFileSync (fs.js:377:35)
    at Object.load (/usr/lib/node_modules/nexe/node_modules/resolve-dependencies/lib/node-loader.js:111:25)
    at Loader.load (/usr/lib/node_modules/nexe/node_modules/resolve-dependencies/lib/loader.js:45:29)
    at /usr/lib/node_modules/nexe/node_modules/resolve-dependencies/lib/loader.js:79:48
    at Array.map (<anonymous>)
    at Loader.load (/usr/lib/node_modules/nexe/node_modules/resolve-dependencies/lib/loader.js:78:14)
    at /usr/lib/node_modules/nexe/node_modules/resolve-dependencies/lib/loader.js:79:48
    at Array.map (<anonymous>)
    at Loader.load (/usr/lib/node_modules/nexe/node_modules/resolve-dependencies/lib/loader.js:78:14)

When I look in the /zigbee2mqtt/node_modules/es5-ext/string/ directory, there is a directory named # which seems to get encoded somewhere and it breaks.

# ls -l /zigbee2mqtt/node_modules/es5-ext/string/
total 32
drwxr-xr-x    9 root     root          4096 Jan  2 18:23 #
-rw-r--r--    1 root     root           708 Oct 26  1985 format-method.js
drwxr-xr-x    2 root     root          4096 Jan  2 18:23 from-code-point
-rw-r--r--    1 root     root           287 Oct 26  1985 index.js
-rw-r--r--    1 root     root           291 Oct 26  1985 is-string.js
-rw-r--r--    1 root     root           205 Oct 26  1985 random-uniq.js
-rw-r--r--    1 root     root          1064 Oct 26  1985 random.js
drwxr-xr-x    2 root     root          4096 Jan  2 18:23 raw

The issue happened with the latest Nexe beta but the issue seems to be in this package. If it isn't the correct place to raise this issue, let me know :)

Regression between resolve-dependencies version 2.2.3 and 3.0.2

Summary

Resolve dependencies returns different results between 2.2.3 and 3.0.2. This results in apps that compile and run correctly with Nexe 3.3.2 failing with Nexe 4.0.0-beta-1 and later.

To reproduce

  1. git clone https://source.small-tech.org/site.js/app.git
  2. npm i [email protected]
  3. Run the test-dependencies.js script at the end of this issue
  4. npm i [email protected]
  5. Change the output file in the script to 3.0.2.json
  6. diff 2.2.3.json 3.0.2.json | grep -C5 serve

The diff shows that the entry for the serve file – which is what results in the module not found error in nexe/nexe#755 – has different data between 2.2.3 and 3.0.2 (it is null in the latter):

27528,27541c32094
<           "./serve": {
<             "deps": {
<               "send": {
<                 "$ref": "$[\"/home/aral/small-tech/site.js/app/index.js\"][\"deps\"][\"express\"][\"deps\"][\"./lib/express\"][\"deps\"][\"./application\"][\"deps\"][\"./utils\"][\"deps\"][\"send\"]"
<               },
<               "url": null
<             },
<             "absPath": "/home/aral/small-tech/site.js/app/node_modules/@small-tech/instant/lib/serve.js",
<             "contents": null,
<             "variableImports": false,
<             "belongsTo": {
<               "$ref": "$[\"/home/aral/small-tech/site.js/app/index.js\"][\"deps\"][\"@small-tech/instant\"]"
<             }
<           },
---
>           "./serve": null,

@calebboyd I hope this gives you something concrete to go on. I’m sorry I’m not of more use as I’m not familiar with the innards of Nexe and it’s not a trivial piece of code to get your head around :)

Script

const fs = require('fs')
const { resolve } = require('resolve-dependencies')

;(async () => {
  const { entries, files } = await resolve({ entries: './index.js', loadContent: false })
  console.log(`${Object.keys(files).length} files returned`)
  fs.writeFileSync('2.2.3.json', JSON.stringify(JSON.decycle(entries)), 'utf-8')
})()

////

/*
    cycle.js
    2018-05-15
    Public Domain.
    NO WARRANTY EXPRESSED OR IMPLIED. USE AT YOUR OWN RISK.
    This code should be minified before deployment.
    See http://javascript.crockford.com/jsmin.html
    USE YOUR OWN COPY. IT IS EXTREMELY UNWISE TO LOAD CODE FROM SERVERS YOU DO
    NOT CONTROL.
*/

// The file uses the WeakMap feature of ES6.

/*jslint eval */

/*property
    $ref, decycle, forEach, get, indexOf, isArray, keys, length, push,
    retrocycle, set, stringify, test
*/

if (typeof JSON.decycle !== "function") {
  JSON.decycle = function decycle(object, replacer) {
      "use strict";

// Make a deep copy of an object or array, assuring that there is at most
// one instance of each object or array in the resulting structure. The
// duplicate references (which might be forming cycles) are replaced with
// an object of the form

//      {"$ref": PATH}

// where the PATH is a JSONPath string that locates the first occurance.

// So,

//      var a = [];
//      a[0] = a;
//      return JSON.stringify(JSON.decycle(a));

// produces the string '[{"$ref":"$"}]'.

// If a replacer function is provided, then it will be called for each value.
// A replacer function receives a value and returns a replacement value.

// JSONPath is used to locate the unique object. $ indicates the top level of
// the object or array. [NUMBER] or [STRING] indicates a child element or
// property.

      var objects = new WeakMap();     // object to path mappings

      return (function derez(value, path) {

// The derez function recurses through the object, producing the deep copy.

          var old_path;   // The path of an earlier occurance of value
          var nu;         // The new object or array

// If a replacer function was provided, then call it to get a replacement value.

          if (replacer !== undefined) {
              value = replacer(value);
          }

// typeof null === "object", so go on if this value is really an object but not
// one of the weird builtin objects.

          if (
              typeof value === "object"
              && value !== null
              && !(value instanceof Boolean)
              && !(value instanceof Date)
              && !(value instanceof Number)
              && !(value instanceof RegExp)
              && !(value instanceof String)
          ) {

// If the value is an object or array, look to see if we have already
// encountered it. If so, return a {"$ref":PATH} object. This uses an
// ES6 WeakMap.

              old_path = objects.get(value);
              if (old_path !== undefined) {
                  return {$ref: old_path};
              }

// Otherwise, accumulate the unique value and its path.

              objects.set(value, path);

// If it is an array, replicate the array.

              if (Array.isArray(value)) {
                  nu = [];
                  value.forEach(function (element, i) {
                      nu[i] = derez(element, path + "[" + i + "]");
                  });
              } else {

// If it is an object, replicate the object.

                  nu = {};
                  Object.keys(value).forEach(function (name) {
                      nu[name] = derez(
                          value[name],
                          path + "[" + JSON.stringify(name) + "]"
                      );
                  });
              }
              return nu;
          }
          return value;
      }(object, "$"));
  };
}


if (typeof JSON.retrocycle !== "function") {
  JSON.retrocycle = function retrocycle($) {
      "use strict";

// Restore an object that was reduced by decycle. Members whose values are
// objects of the form
//      {$ref: PATH}
// are replaced with references to the value found by the PATH. This will
// restore cycles. The object will be mutated.

// The eval function is used to locate the values described by a PATH. The
// root object is kept in a $ variable. A regular expression is used to
// assure that the PATH is extremely well formed. The regexp contains nested
// * quantifiers. That has been known to have extremely bad performance
// problems on some browsers for very long strings. A PATH is expected to be
// reasonably short. A PATH is allowed to belong to a very restricted subset of
// Goessner's JSONPath.

// So,
//      var s = '[{"$ref":"$"}]';
//      return JSON.retrocycle(JSON.parse(s));
// produces an array containing a single element which is the array itself.

      var px = /^\$(?:\[(?:\d+|"(?:[^\\"\u0000-\u001f]|\\(?:[\\"\/bfnrt]|u[0-9a-zA-Z]{4}))*")\])*$/;

      (function rez(value) {

// The rez function walks recursively through the object looking for $ref
// properties. When it finds one that has a value that is a path, then it
// replaces the $ref object with a reference to the value that is found by
// the path.

          if (value && typeof value === "object") {
              if (Array.isArray(value)) {
                  value.forEach(function (element, i) {
                      if (typeof element === "object" && element !== null) {
                          var path = element.$ref;
                          if (typeof path === "string" && px.test(path)) {
                              value[i] = eval(path);
                          } else {
                              rez(element);
                          }
                      }
                  });
              } else {
                  Object.keys(value).forEach(function (name) {
                      var item = value[name];
                      if (typeof item === "object" && item !== null) {
                          var path = item.$ref;
                          if (typeof path === "string" && px.test(path)) {
                              value[name] = eval(path);
                          } else {
                              rez(item);
                          }
                      }
                  });
              }
          }
      }($));
      return $;
  };
}

Use BFS

Recursive synchronous DFS in 6.0 is potentially bad for performance / stack size.
If we use BFS we can use threads for the broadening tree.

Include pkg config assets

Right now we just expand (capture additional package files) when non-static dependencies are detected.

} else if (
options.expand === 'variable' &&
file.variableImports &&
options.context &&
options.context.moduleRoot &&
!options.context.expanded
) {
await expand(file, fileDir, options.context.moduleRoot, options.context.globs)

But we can include the pkg scripts and assets by default if present.

Option for limiting the number of workers?

Hey there, thanks for creating this project, it's really cool and extremely helpful.

We're trying to use nexe to bundle some of our tools and we're running into an issue when using it inside a container: the container runs out of memory due to the number of processes spawned (the host machine has 72 cores).

As you can see in nodejs/node#28765 and nodejs/node#28855, node's os.cpus doesn't really query the correct location on Linux and therefore can't get the number of available CPUs, so even though we're actually running the container with only 2 cores, nexe/resolve-dependencies still assumes 72 cores are available.

Would you be open to introducing an option in nexe and resolve-dependencies to limit the number of children spawned by the dependency resolution process? Or perhaps use a namespace-aware library for querying the number of available CPUs?

Thanks!

Native Modules go unresolved

This package is unable to resolve native dependencies when they exist in a parent node_modules folder.
Currently, I have to make a copy of them using something like find-up in a postinstall script to properly run my executable in my consumer app.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.