Webpack build hangs with parallel: true about terser-webpack-plugin HOT 71 CLOSED

I've just tried to reproduce vuejs/vue-cli#3327 with parallel: true on WSL (1803) again and couldn't get into the lock situation anymore. Looks like the issue was fixed in WSL.

from terser-webpack-plugin.

Okey, we disable parallels on WSL and for 2.0 we migrate on jest-worker (they work with WSL) #89

from terser-webpack-plugin.

This thread caught my eye as I was searching for possible reasons we are seeing CI builds occasionally time out (webpack hanging for 45+ minutes). We are not running WSL, but this in particular stood out to me:

if you watch Task Manager you'll notice that eventually the node processes will just cease utilizing CPU resource with the build reporting 92% chunk asset optimization TerserPlugin (as pictured below)

I've seen our build hang for a while running locally on my Mac, and it was always at 92%. I haven't been able to get any more details out of our CI build hangs yet, but I'm going to test the parallel setting and see if it makes any difference.

I should note that this is an intermittent issue for us; I'm not sure if other people are seeing this consistently or not.

from terser-webpack-plugin.

I am on 1803. Surprisingly it worked now for my repo now on WSL😃.Previously it wasn't working, had to manually disable parallel

Edit: let me try the repro test
Edit 2: Doesn't work on the #21 (comment)

Very interesting because it is working on my local repo that once had this problem, but still not working on the reproducible repo. Could it be a dependency problem ?

from terser-webpack-plugin.

I'm sorry. I gave up on trying to use Windows for development because problems like this was the norm and not an exception so now I only have Linux to test on.

from terser-webpack-plugin.

@Pajn Thanks for issue, we have some problem on WSL for uglify plugin webpack-contrib/uglifyjs-webpack-plugin#302, seems problem in worker-farm, thanks for reproducible test repo, i am not familiar with WSL, can you provide instruction how i can setup this env?

from terser-webpack-plugin.

To set up WSL you need Windows 10, then you should be able to follow this guide https://docs.microsoft.com/en-us/windows/wsl/install-win10

After that you can basically treat is as a normal Ubuntu Linux so install node by following this https://nodejs.org/en/download/package-manager/#debian-and-ubuntu-based-linux-distributions
Git can be installed via sudo apt install git

The only consideration with a case like this is that Linux files are not visible to Windows, so to be able to open files in a graphical editor you need to save them under /mnt/c

from terser-webpack-plugin.

@Pajn oh, i don't have windows 10 right now, can debug this only on next week, feel free to debug problem self. Here default arguments to work-farm https://github.com/webpack-contrib/terser-webpack-plugin/blob/master/src/TaskRunner.js#L36, maybe we need improve this check process.platform === 'win32' and pass maxConcurrentCallsPerWorker: 1 on WSL

from terser-webpack-plugin.

You are right that worker-farm is somehow related.
This is what I get from pstree in a stuck build

  │   │   ├─node /usr/share/yarn/bin/yarn.js build
  │   │   │   ├─sh -c tscomp build
  │   │   │   │   └─node /c/Users/Rasmus/Development/RAXA/packages/web/node_modules/.bin/tscomp build
  │   │   │   │       ├─node /c/Users/Rasmus/Development/RAXA/packages/web/node_modules/tscomp/scripts/build.js
  │   │   │   │       │   ├─node /c/Users/Rasmus/Development/RAXA/packages/web/node_modules/tscomp/node_modules/worker-farm/lib/child/index.js /usr/bin/node /c/Users/Rasmus/Development/RAXA/packages/web/node_modules/tscomp/scripts/build.js
  │   │   │   │       │   │   └─5*[{node}]
  │   │   │   │       │   ├─node /c/Users/Rasmus/Development/RAXA/packages/web/node_modules/tscomp/node_modules/worker-farm/lib/child/index.js /usr/bin/node /c/Users/Rasmus/Development/RAXA/packages/web/node_modules/tscomp/scripts/build.js
  │   │   │   │       │   │   └─5*[{node}]
  │   │   │   │       │   └─9*[{node}]
  │   │   │   │       └─5*[{node}]
  │   │   │   └─9*[{node}]

Changing that condition so that maxConcurrentCallsPerWorker: 1 is passed (process.platform is linux) did not help unfortunately.

I tried running the basic example from the worker-farm readme and that worked without hanging so I guess there some problem with file locking.

I'll continue the debugging tomorrow.

from terser-webpack-plugin.

@Pajn thanks for helping! How many cpu you are have? Based on output worker-farm is create child processes but don't get response.

from terser-webpack-plugin.

I have four, however the above pstree output is from a run where I limited the number of workers to two, just to see if it did help.

I modified the code so that it would still use worker-farm with only one worker, and then it did finish again. I also tried to attach strace to see if I got some hint of the problem but did not find anything useful in the output. If you are interested I can of course share those outputs.

from terser-webpack-plugin.

@Pajn Feel free to publish outputs

from terser-webpack-plugin.

Guys i try to reproducible problem but i can't, maybe can create really minimum reproducible test repo with really minimum setup?

from terser-webpack-plugin.

@evilebottnawi you should be able to recreate this by compiling the an Angular demo app (any should do) with aot and optimization flags turned on.

I get this issue where the TerserPlugin seems to hang with a lot of node processes showing in Task Manager using 0% CPU resource.

from terser-webpack-plugin.

@charsleysa can you provide instruction for minimum reproducible test repo?

from terser-webpack-plugin.

@evilebottnawi it seems to be something to do with having a project sufficiently large enough producing source maps on a production build.

The following instructions assume you have already setup WSL with Node 10.x (node inside WSL) using standard setup instructions. The below instructions are executed inside bash.

1. clone this repo https://github.com/formio/angular-demo
2. cd into the cloned repo directory
3. install the angular cli npm install -g @angular/cli
4. install the repo requirements npm install
5. run a standard build just to check that everything is working ng build
6. run a production build with source maps turned on ng build --configuration=production --project=demo --source-map=true
7. if you watch Task Manager you'll notice that eventually the node processes will just cease utilizing CPU resource with the build reporting 92% chunk asset optimization TerserPlugin (as pictured below)

from terser-webpack-plugin.

@charsleysa thanks for instructions, anyway feel free to investigate problem(s), i think problem inside worker-farm, maybe you can find problem faster than i can, because i need time to setup WSL

from terser-webpack-plugin.

WSL is becoming more and more popular. I'm seeing the same problem as well and can confirm that parallel: false works. If #6 is applied to make parallel the default, then this needs to be fixed first.

from terser-webpack-plugin.

@andreialecu can you investigate why it is happens? Don't have WSL around and don't have windows machine near, maybe you can find problem

from terser-webpack-plugin.

I've been digging at it and from my initial findings it seems like this callback is never ran:

https://github.com/webpack-contrib/terser-webpack-plugin/blob/master/src/TaskRunner.js#L74

However, the task itself and the code in minify.js is called, and is returning properly. I'm trying to investigate why it never gets there now. I suspected it would be a problem in worker-farm but the examples in the worker-farm repo and all of its tests pass properly.

from terser-webpack-plugin.

@andreialecu yep, problem in worker-farm, maybe we need to setup more options (https://github.com/webpack-contrib/terser-webpack-plugin/blob/master/src/TaskRunner.js#L34) for WSL

from terser-webpack-plugin.

What is process.platform on WSL?

from terser-webpack-plugin.

But like I said, worker-farm's entire test suite passes properly.

from terser-webpack-plugin.

Further progress on debugging this, it seems like the forked process' on('message') handler in worker-farm is never called when running with this plugin on WSL:

https://github.com/rvagg/node-worker-farm/blob/master/lib/farm.js#L145

However, when running worker-farm outside of this scenario, it is called. I verified using this: https://github.com/rvagg/node-worker-farm/tree/master/examples/basic

What could be different?

from terser-webpack-plugin.

@andreialecu only setup this options https://github.com/webpack-contrib/terser-webpack-plugin/blob/master/src/TaskRunner.js#L34 (as i written above)

from terser-webpack-plugin.

I think I found the root cause.

It is related to the length of the result being passed from the worker process back to the farm via process.send(). If I make it send a simple 'hello' instead of the entire minified script result it seems that all callbacks are called.

Making this callback with a shorter string results in all callbacks working:
https://github.com/webpack-contrib/terser-webpack-plugin/blob/master/src/worker.js#L17

I'm trying to reproduce it here https://github.com/rvagg/node-worker-farm/blob/master/examples/basic/index.js by changing the code to send a 10MB result and it works perfectly:

module.exports = function (inp, callback) {
  const msg = 'a'.repeat(10 * 1000 * 1000);
  callback(null, inp + ' BAR (' + process.pid + ')' + msg)
}

So I'm not sure what is going on.

from terser-webpack-plugin.

@Pajn Maybe some characters break this 😕

from terser-webpack-plugin.

@evilebottnawi Well, I just tested that by dumping the objects returned by the minify() worker functions to local .json files then trying to read each of them in simple workers in the basic example above, and then pass them on to the main process. Everything worked.

I'm stumped.

Perhaps someone can take over. Running out of time assigned to this right now.

FWIW: as a quick test, I tried calling back with the first 100000 characters of JSON.stringify(minify(...)) from the minify worker and those went through, but the first 1000000 characters made it hang.

from terser-webpack-plugin.

weird 😕

from terser-webpack-plugin.

A possible workaround for this would be to save all minified scripts to temp files from the worker and call back with the path to the temp files, then load them back in the master. This would probably fix it. How would you feel about this?

from terser-webpack-plugin.

@andreialecu as workaround sounds good, but will be great find real problem why it is doesn't work as expected, maybe bug in WSL and we should report about this bug. Because on macos/linux/windows without WSL it is works as expected. So i think problem in WSL. If problem won't fix we implement workaround. Write files in temp directory decrease perf and it is not good.

from terser-webpack-plugin.

When this works through a workaround, is the temp directory in /tmp or is it under /mnt/c/ ? I'm wondering if this is a file system related bug.

from terser-webpack-plugin.

On WSL:

$ node -pe 'os.tmpdir()'
/tmp

from terser-webpack-plugin.

I wonder if this fails under /mnt/c/Users/[windowsuser]/temp - which is (usually) NTFS.

from terser-webpack-plugin.

@joewood maybe somebody has image for virtualbox with installed WSL?

from terser-webpack-plugin.

I can try later today on my regular machine as I'm seeing this issue too. I suspect this is a WSL bug, the idea of trying on the different disk formats is to compare on the WSL formatted disk vs. regular NTFS (which has some differences).

from terser-webpack-plugin.

The problem is with interprocess communication not with anything file system related. /tmp isn't used. It was just a suggestion I made to use as a workaround so it is just hypothetical at this point.

I could not reproduce the error outside of webpack so I'm still not sure what the root cause is.

from terser-webpack-plugin.

Take a look here for Windows VMs: https://developer.microsoft.com/en-us/windows/downloads/virtual-machines

from terser-webpack-plugin.

@andreialecu already work on this, i will put information here

from terser-webpack-plugin.

Tested on latest version and all works fine, i think problem was fixed inside WSL

from terser-webpack-plugin.

Try to reproduce with #21 (comment)

from terser-webpack-plugin.

Parallels option works fine on latest WSL version

from terser-webpack-plugin.

@evilebottnawi what Windows version are you testing on? 1903?

from terser-webpack-plugin.

@andreialecu virtual machine from windows, how i can get version of windows?

from terser-webpack-plugin.

I am trying to reproduce but can't, all work fine

from terser-webpack-plugin.

Run winver in a command prompt or from run. Should say version 1809 or 1903.

from terser-webpack-plugin.

@andreialecu Version 1803 (OS Build 17134.706)

from terser-webpack-plugin.

@andreialecu maybe you can create minimum reproducible test repo?

from terser-webpack-plugin.

I'm also on 18031809, which is the currently widely available Windows version. 1903 is supposed to be released this month, and it has some WSL fixes, so maybe it might impact this issue. I haven't tested 1903 yet.

They also announced WSL2.0 which is a complete architecture change from WSL1.0 and whatever is causing this issue should be fixed by that, because it uses a real Linux kernel among other things. https://devblogs.microsoft.com/commandline/announcing-wsl-2/

That being said, I was just able to reproduce it on 18031809 using #21 (comment)

One thing to check (just suggesting it, since it looks like you're not a Windows guy :) - are you sure you're running under WSL and not the regular Command Prompt?

EDIT: Actually I'm on 1809, 1803 is an older release.

from terser-webpack-plugin.

Attaching repro proof:

from terser-webpack-plugin.

@andreialecu yes

node -e 'console.log(require("os").release())'
4.4.0-17134-Microsoft

from terser-webpack-plugin.

Investigate deeply, thanks for fast reply

from terser-webpack-plugin.

@evilebottnawi actually, minor correction on my previous comment. I'm on 1809, not 1803. You're one version behind. If you definitely cannot reproduce it, that may point out why.

from terser-webpack-plugin.

@andreialecu reproduce problem ⭐ WIP on fix

from terser-webpack-plugin.

Repo #21 (comment) is very hard 😞 i was spent around hour to search configuration and no luck, in future all big repositories will be ignored, it is hard to debug and it spends a lot of time

from terser-webpack-plugin.

I think #21 (comment) webpack config is defined/ constructed dynamically on @angular/cli which is another huge project.

from terser-webpack-plugin.

@endiliey @andreialecu looks it is not freeze, try to wait 5-25 minutes (depends on you hardware), problem what source map generation takes a lot of time

from terser-webpack-plugin.

In webpack@5 we update https://github.com/mozilla/source-map to latest version (WASM) and it should solve problem with perf, but here we can't do this 😞

from terser-webpack-plugin.

Tried on many repos and all works fine, in some repos i have waited around 20 minutes due it is really big application. Just ensure it is not problem in plugin with WSL, just disable source map and try again

from terser-webpack-plugin.

Did you see 0% cpu for node processes for a very long time while you waited, and it eventually finished? Did you try with parallel: false?

A problem still exists on WSL that is not present for native Windows or native Linux builds. If you go through my comments above you can use a debugger to attach and break on something inside the terser plugin code even if it is running via ng build.

Anyway, it's probably not worth working too much on this because I assume that it will automagically fix itself on WSL 2.0, and in the mean time parallel: false works without huge performance penalties.

from terser-webpack-plugin.

I have been running it for 2 hours but it still stucks in 92%. Also 0% node process

from terser-webpack-plugin.

@endiliey @andreialecu it is very strange, 20 min for #21 (comment) and all were minified, maybe you can create other minimum reproducible test repo?

from terser-webpack-plugin.

I am on SSD, maybe WSL has timeouts and in some cases it is failed 😕

from terser-webpack-plugin.

I can disable parallels on WSL by default, but i want to ensure it is real problem in WSL, not in webpack or other ecosystem

from terser-webpack-plugin.

Somebody can test master version on WSL?

from terser-webpack-plugin.

Can't setup azure pipelines for WSL, if anyone had experience please put here 😄

from terser-webpack-plugin.

/cc @Pajn can you test terser-webpack-plugin@2 on WSL? Thanks

from terser-webpack-plugin.

/cc @andreialecu friendly ping, you are still on WSL?

from terser-webpack-plugin.

I am closing issue, if somebody will get an error, just open a new problem with a minimal example, thanks

from terser-webpack-plugin.

I spent like an hour trying to figure out why Terser refused to run in parallel no matter what I did only to realize that parallel is disabled if you are on WSL. I'm currently on Windows build 19041 (Slow ring insider preview) using WSL2 and it works perfectly with all cores.

from terser-webpack-plugin.

Upgraded my terser plugin from 1.3.5 to 2.3.5 and the issue is resolved, can use parallel now.

from terser-webpack-plugin.