Giter Club home page Giter Club logo

wds's Introduction

wds

A reloading dev server for server side TypeScript projects. Compiles TypeScript real fast, on demand, using require.extensions. Similar to and inspired by ts-node-dev.

wds stands for Whirlwind (or web) Development Server.

Examples

After installing wds, you can use it like you might use the node command line program:

# run one script with wds compiling TS to JS
wds some-script.ts

# run one server with wds `watch` mode, re-running the server on any file changes
wds --watch some-server.ts

# run one script with node command line arguments that you'd normally pass to `node`
wds --inspect some-test.test.ts

Features

  • Builds and runs TypeScript really fast using swc
  • Incrementally rebuilds only what has changed in --watch mode, restarting the process on file changes
  • Execute commands on demand with the --commands mode
  • Plays nice with node.js command line flags like --inspect or --prof
  • Supports node.js ipc channels between the process starting wds and the node.js process started by wds.
  • Produces sourcemaps which Just Work™️ by default for debugging with many editors (VSCode, IntelliJ, etc)
  • Monorepo aware, allowing for different configuration per package and only compiling what is actually required from the monorepo context

Motivation

You deserve to get stuff done. You deserve a fast iteration loop. If you're writing TypeScript for node, you still deserve to have a fast iteration loop, but with big codebases, tsc can get quite slow. Instead, you can use a fast TS => JS transpiler like swc to quickly reload your runtime code and get to the point where you know if your code is working as fast as possible. This means a small sacrifice: tsc no longer typechecks your code as you run it, and so you must supplement with typechecking in your editor or in CI.

This tool prioritizes rebooting a node.js TypeScript project as fast as possible. This means it doesn't typecheck. Type checking gets prohibitively slow at scale, so we recommend using this separate typechecker approach that still gives you valuable feedback out of band. That way, you don't have to wait for it to see if your change actually worked. We usually don't run anything other than VSCode's TypeScript integration locally, and then run a full tsc --noEmit in CI.

Usage

Options:
      --help       Show help                                           [boolean]
      --version    Show version number                                 [boolean]
  -c, --commands   Trigger commands by watching for them on stdin. Prevents
                   stdin from being forwarded to the process. Only command right
                   now is `rs` to restart the server. [boolean] [default: false]
  -w, --watch      Trigger restarts by watching for changes to required files
                                                      [boolean] [default: false]
  -s, --supervise  Supervise and restart the process when it exits indefinitely
                                                      [boolean] [default: false]

Configuration

Configuration for wds is done by adding a wds.js file to your pacakge root, and optionally a .swcrc file if using swc as your compiler backend.

An wds.js file needs to export an object like so:

module.exports = {
  // which file extensions to build, defaults to .js, .jsx, .ts, .tsx extensions
  extensions: [".tsx", ".ts", ".mdx"],

  // file paths to explicitly not transform for speed, defaults to [], plus whatever the compiler backend excludes by default, which is `node_modules` for swc
  ignore: ["spec/integration/**/node_modules", "spec/**/*.spec.ts", "cypress/", "public/"],
};

When using swc (the default)

swc is the fastest TypeScript compiler we've found and is the default compiler wds uses. wds sets up a default swc config suitable for compiling to JS for running in Node:

{
  "jsc": {
    "parser": {
      "syntax": "typescript",
      "decorators": true,
      "dynamicImport": true
    },
    "target": "es2020"
  },
  "module": {
    "type": "commonjs",
    // turn on lazy imports for maximum reboot performance
    "lazy": true
  }
}

Note: the above config is different than the default swc config. It's been honed to give maximum performance for server start time, but can be adjusted by creating your own .swcrc file.

Configuring swc's compiler options with with wds can be done using the wds.js file. Create a file named wds.js in the root of your repository with content like this:

// wds.js
module.exports = {
  swc: {
    env: {
      targets: {
        node: 12,
      },
    },
  },
};

You can also use swc's built in configuration mechanism which is an .swcrc file. Using an .swcrc file is useful in order to share swc configuration between wds and other tools that might use swc under the hood as well, like @swc/jest. To stop using wds's default config and use the config from a .swcrc file, you must configure wds to do so using wds.js like so:

// in wds.js
module.exports = {
  swc: ".swcrc",
};

And then, you can use swc's standard syntax for the .swcrc file

// in .swcrc, these are the defaults wds uses
{
  "jsc": {
    "parser": {
      "syntax": "typescript",
      "decorators": true,
      "dynamicImport": true
    },
    "target": "es2022"
  },
  "module": {
    "type": "commonjs",
    // turn on lazy imports for maximum reboot performance
    "lazy": true
  }
}

Refer to the SWC docs for more info.

Comparison to ts-node-dev

ts-node-dev (and ts-node) accomplish a similar feat but are often 5-10x slower than wds in big projects. They are loaded with features and will keep up with new TypeScript features much better as they use the mainline TypeScript compiler sources, and we think they make lots of sense! Because they use TypeScript proper for compilation though, even with --transpile-only, they are destined to be slower than swc. wds is for the times where you care a lot more about performance and are ok with the tradeoffs swc makes, like not supporting const enum and being a touch behind on supporting new TypeScript releases.

wds's People

Contributors

airhorns avatar angelini avatar calebboyd avatar dependabot[bot] avatar depfu[bot] avatar elyse0 avatar gmalette avatar hyrious avatar jasong689 avatar kira-bruneau avatar pistachiobaby avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

wds's Issues

Performance and profiling discussion

TL; DR:

  • Booting esbuild-dev currently costs ~130 ms only for itself.
  • Requiring validate-boot.ts and not executing its code takes ~1800ms.
    • File System access from the child process appears to be negligible.
    • esbuild takes ~255 ms.
    • module._compile takes ~1100 ms.
    • There appears to be a ~370 ms overhead in SyncWorker.
    • There appears to be high variability in how long it takes to schedule a reload once a file changes.

Suggestions:

  • Invest in reducing the number of files to _compile.
  • Look into bootsnap-like caching of compiled files/bytecode if possible.
  • Understand if there's actual overhead in SyncWorker, and why.
  • Investigate reload scheduling variability.

Understanding esbuild-dev's performance

Part 1: Big Blocks

At the moment I'm pretty clueless about the performance of esbuild-dev, how long I should expect anything to take, and where the time is spent.

To start, I'm interested in the following parts:

  • Boot of NodeJS, doing nothing. This is a fixed cost which we cannot avoid paying.
  • Boot of esbuild-dev. This could be optimized, if needs be.
  • Time between save and restart under --watch --supervise (small and large).
  • Time for restart after package.json / yarn.lock changes.

These tests were measured on a:

MacBook Pro 14, 2021
32 GB Ram
10 Cores (8 performance/2 efficiency)

Vanilla Node boot

To test this, I launched node processes with a file that only logs, and measure the time taken. This gives us the expected time required to boot a node process.

node src/noop.js #{run_number}

p50 = 30ms
p99 = 37ms

image

esbuild-dev boot

Similar to the previous exercise, but running esbuild-dev:

esbuild-dev src/noop.js #{run_number}

p50 = 138ms
p99 = 160ms

We do expect esbuild-dev to startup a file more slowly than node because there's two node process being started: one for esbuild-dev and one executed by it for the desired process (src/noop.js), and so our expectations are at least 60ms, but we can see it's a bit slower than that here.

image

esbuild-dev supervise a small file

This test measures how long esbuild-dev takes to react and restart a process when changing a single small file.

esbuild-dev --supervise src/noop.js #{run_number}

In this test we measure two metrics:

  1. Delay between touch and the time esbuild-dev schedules the file to be reloaded
  2. The time it takes between the scheduling and for the file to have been fully reloaded

The delay at #1 (between touch and scheduling) is susceptible to high variability, and I'm not sure what causes it. It's possible that the fs.utimesSync is not flushed right away.

Schedule
p50 = 114ms
p99 = 117ms

Reload
p50 = 117ms
p99 = 127ms

In this test we can see that the Reload time is very much in line with the esbuild-dev boot minus booting the initial node process.

image

esbuild-dev supervise a larger codebase (with cache)

This test is is similar to the previous one but uses a larger codebase, in this case a modified version of Gadget's api/tools/validate-boot.ts file, to avoid running any code.

In this first test we measured with full use of the OS's file cache.

Schedule
p50 =117 ms
p99 = 124 ms

Reload
p50 = 699 ms
p99 = 799 ms

image

esbuild-dev supervise a larger codebase (with cache)

This test is is similar to the previous one but uses a larger codebase, in this case Gadget's api/tools/validate-boot.ts file.

This is probably the most common scenario.

In this first test we measured with full use of the OS's file cache.

Schedule
p50 =117 ms
p99 = 124 ms

Reload
p50 = 699 ms
p99 = 799 ms

esbuild-dev supervise a larger codebase (without cache)

This test is is similar to the previous one purges the file cache between each run. This scenario isn't excepted to be common, but it's still interesting.

Interestingly, we can see that the cache is resposible for a speedup of ~32%.

Schedule
p50 = 113 ms
p99 = 117 ms

Reload
p50 = 1,018 ms
p99 = 1,168 ms

image

Part 2: Where Time is Spent

image

At first glance, it looks like module._compile is taking a long time.

In fact, using the same modified validate-boot.ts file, which takes approx 1800 ms to load it, approx 1100 ms are attributable to module._compile. Of the remainder, 255 ms are attributable to esbuild.build. File system access is negligible (1.7ms).

image

From what I've been able to measure*, 369 ms were spent as SyncWorker communication overhead/delay.

  • I have low relative confidence because of how log messages are passed from the worker to the main thread, apparently?

Non-compilable files

Any way to support the swc --copy-files CLI arg somehow? Right now, non-compilable files like JSON, for example, do not trigger the event while watching, and also do not update after a file changes.

Depfu Error: No dependency files found

Hello,

We've tried to activate or update your repository on Depfu and couldn't find any supported dependency files. If we were to guess, we would say that this is not actually a project Depfu supports and has probably been activated by error.

Monorepos

Please note that Depfu currently only searches for your dependency files in the root folder. We do support monorepos and non-root files, but don't auto-detect them. If that's the case with this repo, please send us a quick email with the folder you want Depfu to work on and we'll set it up right away!

How to deactivate the project

  • Go to the Settings page of either your own account or the organization you've used
  • Go to "Installed Integrations"
  • Click the "Configure" button on the Depfu integration
  • Remove this repo (gadget-inc/wds) from the list of accessible repos.

Please note that using the "All Repositories" setting doesn't make a lot of sense with Depfu.

If you think that this is a mistake

Please let us know by sending an email to [email protected].


This is an automated issue by Depfu. You're getting it because someone configured Depfu to automatically update dependencies on this project.

Removed files aren't purged from the current build

When deleting a branch, we throw this currently:

 > error: Could not read from file: /Users/airhorns/Code/gadget/packages/app-sandbox/index.ts

1 error
(node:59865) UnhandledPromiseRejectionWarning: Error: Build failed with 1 error:
error: Could not read from file: /Users/airhorns/Code/gadget/packages/app-sandbox/index.ts
    at failureErrorWithLog (/Users/airhorns/Code/gadget/node_modules/esbuild-dev/node_modules/esbuild/lib/main.js:947:15)
    at buildResponseToResult (/Users/airhorns/Code/gadget/node_modules/esbuild-dev/node_modules/esbuild/lib/main.js:762:32)
    at /Users/airhorns/Code/gadget/node_modules/esbuild-dev/node_modules/esbuild/lib/main.js:775:21
    at handleIncomingPacket (/Users/airhorns/Code/gadget/node_modules/esbuild-dev/node_modules/esbuild/lib/main.js:562:9)
    at Socket.readFromStdout (/Users/airhorns/Code/gadget/node_modules/esbuild-dev/node_modules/esbuild/lib/main.js:478:7)
    at Socket.emit (events.js:314:20)
    at addChunk (_stream_readable.js:303:12)
    at readableAddChunk (_stream_readable.js:279:9)
    at Socket.Readable.push (_stream_readable.js:218:10)
    at Pipe.onStreamRead (internal/stream_base_commons.js:188:23)
(node:59865) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 104)

Should either:

  • invalidate the file map and ask the tsconfig for the new files
  • change strategies away from using the tsconfig to enumerate all the files

Immediate internal error: timed out communication with wds sync worker

I'd like to give more info, but this is about all I have:

Running node 16.14.2

❯ node_modules/.bin/wds ./src/index.ts
Error: [wds] Internal error: timed out communicating with wds sync worker thread, likely an wds bug
    at SyncWorker.call (/Users/bart.riepe/Projects/horizon-core/node_modules/wds/src/SyncWorker.ts:90:13)
    at compile (/Users/bart.riepe/Projects/horizon-core/node_modules/wds/src/child-process-registration.ts:47:31)
    at Object.require.extensions.<computed> [as .ts] (/Users/bart.riepe/Projects/horizon-core/node_modules/wds/src/child-process-registration.ts:65:32)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12)
    at node:internal/main/run_main_module:17:47

Can't assume tsconfig.json is at the same location with package.json

I'm making an electron app, the project structure is like:

/package.json
	/src
		/main
			index.ts
			tsconfig.json 	// for node.js env
		/render
			tsconfig.json 	// for browser env

https://github.com/gadget-inc/esbuild-dev/blob/a77d709db01507aa9961c29a4b99e941c4571d26/src/Compiler.ts#L31-L35

When I run esbuild-dev ./path/to/main/index.ts, it says:

Error: couldnt find tsconfig.json near ./path/to/main/index.ts

Also, provide an option to disable type checking entirely (because vscode does it for us)?

SyntaxError: Unexpected token '.' when running esbuild-dev

Hello! I'm looking for a ts-node-dev replacement and I decided to test this package.

After installing it, if I try to run our Node + Typescript API with it, I'm getting the following error:

yarn run v1.19.0
$ esbuild-dev packages/admin-api/src/server.ts
C:\XXX\XXX\XXX\XXX\node_modules\esbuild-dev\pkg\index.js:91
            project.watcher?.add(filename);
                            ^

SyntaxError: Unexpected token '.'
    at wrapSafe (internal/modules/cjs/loader.js:915:16)
    at Module._compile (internal/modules/cjs/loader.js:963:27)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
    at Module.load (internal/modules/cjs/loader.js:863:32)
    at Function.Module._load (internal/modules/cjs/loader.js:708:14)
    at Module.require (internal/modules/cjs/loader.js:887:19)
    at require (internal/modules/cjs/helpers.js:74:18)
    at Object.<anonymous> (C:\XXX\XXX\XXX\XXX\node_modules\esbuild-dev\pkg\esbuild-dev.bin.js:4:14)
    at Module._compile (internal/modules/cjs/loader.js:999:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

The project is a yarn wokrspaces monorepo and I'm on a Windows 10 machine with Node v.12.22.1

Getting process exit code 1 when changing files

"C:\Program Files\nodejs\npm.cmd" run service-dev-watch

> [email protected] service-dev-watch
> wds --inspect=9230 ./src/index.ts

Debugger listening on ws://127.0.0.1:9230/f2cb6f48-1b77-4214-bf73-9f447f5b8098
For help, see: https://nodejs.org/en/docs/inspector
Running!!!
Service listening on port: 8080
Debugger attached.
[wds pid=23084] \src\index.ts changed, restarting ...

Process finished with exit code 1

Hi there, just started using this project today- but noticed the "hot reload" doesn't work for me. This includes simple updates like just changing the text inside a console.log(). I know it has nothing to do with the --inspect=9230 as I tried without it and it still threw the error.

Example of how to start

What I'm really looking for is a thing that says either:

Run this in the command line:

wds ./src/your/file.ts

Or add to your package.json:

{
  "scripts": {
    "start": "wds ./src/your/file.ts"
  }
}

Cache resulting require paths to speed up module resolution

Like bootsnap for ruby or https://github.com/bahmutov/cache-require-paths, we can make module name => file resolution go faster by injecting the map ahead of time into the managed process. Not exactly sure the fastest way to do this, but both TypeScript and esbuild know how to do this resolution ahead of time, so we may be able to do it one pass all ahead of time.

Alternatively and less desirably we could just cache the resolved path on the first run and re-use it when in --watch, but that doesn't improve first boot time which I'd like to do.

Debugger advice

Hi!

I switched a project to using esbuild-dev over ts-node-dev to detect some esbuild specific issues earlier in the dev cycle. It's really helped with that so thanks so much for putting this together!

In vscode, with ts-node-dev we used the node debugger. With esbuild-dev, the debugger is attached but line numbers don't match and it goes to the compiled JS not to the source files.
I'm guessing this is because source maps are not made by default.
Do you have any advice on how I can enable source maps through esbuild-dev? Or other work arounds?

Thanks,

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.