npm install task (not msbuild) fails with EPERM on hosted build about azure-pipelines-tasks HOT 102 CLOSED

Based on this thread, I may have found a workaround: first install npm@3, then run that using a Command Line task to do the actual npm install:

(npm task) -> npm install npm@3
(Command Line task) -> $(Build.SourcesDirectory)\[your project]\node_modules\.bin\npm.cmd

My npm install tasks finished successfully without having to restart it multiple times for the first time in months!

from azure-pipelines-tasks.

the front page of the agent repo talks about the npm issue
https://github.com/Microsoft/vso-agent

from azure-pipelines-tasks.

Front page talks about installing vsoagent, but I have that issue on hosted pool/agent while running npm install task during build?

from azure-pipelines-tasks.

Ahhh. interesting. Are you installing from npm globally? (-g)

from azure-pipelines-tasks.

I don't think so. I just add Npm Install task to my build and setup working directory to folder where is my package.json

from azure-pipelines-tasks.

I have the same problem with 'Npm install' build step on Visual Studio Online using hosted pool agent to build.

from azure-pipelines-tasks.

OK - we will investigate today

from azure-pipelines-tasks.

Hey guys, sorry you're hitting this issue. I think I see what the problem is, it should be resolved shortly.

Thanks,
Dennis

from azure-pipelines-tasks.

Closing since the fix is about to rollout to hosted

from azure-pipelines-tasks.

Re-opening for Dennis to confirm

from azure-pipelines-tasks.

It looks like my original theory wasn't correct. Can either of you provide me your visualstudio.com account name so I can look up which data center you're account is in? You can email me at [email protected] if you're uncomfortable posting it here.

Other questions that may help us diagnose the issue:
Have you been able to do this exact step successfully in the past? If so, when did it last succeed and when did you first start seeing it fail?

Does the step fail for all npm packages or just certain ones? If so, which have you seen it succeed for and which failed?

Thanks,
Dennis

from azure-pipelines-tasks.

Hi, My VSO account name is rcs-pol. For the first time I used 'npm install' build step yesterday and I had these problems from this first time. The only successful step was about 3 hours ago. I thought you had solved the problem. However now this step fails again. Error from build log:
2015-07-16T13:06:32.9426115Z ##[error]npm ERR! EEXIST, open 'C:\NPM\Cache\a82d4a1c--Cache-lodash-3-10-0-package-tgz.lock'
2015-07-16T13:06:32.9436102Z ##[error]File exists: C:\NPM\Cache\a82d4a1c--Cache-lodash-3-10-0-package-tgz.lock
2015-07-16T13:06:32.9436102Z ##[error]Move it away, and try again.
2015-07-16T13:06:32.9446106Z ##[error]npm ERR! System Windows_NT 6.2.9200
2015-07-16T13:06:32.9456106Z ##[error]npm ERR! command "C:\Program Files\nodejs\node.exe" "C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js" "install"
2015-07-16T13:06:32.9496111Z ##[error]npm ERR! cwd C:\a\4f767487\EuradSB\src\EuradWebClient
2015-07-16T13:06:32.9516111Z ##[error]npm ERR! node -v v0.10.32
2015-07-16T13:06:32.9526111Z ##[error]npm ERR! npm -v 1.4.28
2015-07-16T13:06:32.9546110Z ##[error]npm ERR! path C:\NPM\Cache\a82d4a1c--Cache-lodash-3-10-0-package-tgz.lock
2015-07-16T13:06:32.9576107Z ##[error]npm ERR! code EEXIST
2015-07-16T13:06:32.9716105Z ##[error]npm ERR! errno 47
2015-07-16T13:06:33.6864108Z ##[error]npm
2015-07-16T13:06:34.1110534Z ##[error]Unexpected exit code 47 returned from tool npm.cmd

from azure-pipelines-tasks.

Our theory is currently this: http://stackoverflow.com/questions/26491311/lock-file-gets-in-the-way-when-trying-to-generate-yeoman-chrome-extension

If that is the issue, we need to update npm on our build image

from azure-pipelines-tasks.

I've gotten this issue randomly as well. I've run into an issue previously where if multiple builds that call "npm install" run at the same time, the lock file from the first npm install will cause the second one to fail. I didn't have much time to try and fix the issue with NPM and ended up only being able to limit builds to one concurrent NPM install at once. I did read that you can override the global cache folder per run using --cache=/folder but didn't get much farther on it.

npm/npm#2500

from azure-pipelines-tasks.

We're you running multiple agents on the same machine (that might be another workaround). Interesting that you can set cache folder - we could set local to each agent. Although it seems like it's npm's job to protect's it's cache. We'll look at the workaround. thx

from azure-pipelines-tasks.

I think local to each agent would be the best bet. My specific use case was with Jenkins and multiple agents but the idea was the same.

from azure-pipelines-tasks.

This happened to me too. A quick solution was to make a agent for a Linux machine already set up. The problem is that we have no control over the npm. Eg: install npm try to compile many packages by OS and depending on the package it may require admin access, installed tools on machine, or even a more advanced npm version (like node-sass package that requires npm ^ 2.x ). Things we can not do in HostedBuild

One of the things that can put an end to these problems will be docker support , which will give us control of the environment even in the hosted environment.

from azure-pipelines-tasks.

Also having the same issues here, I have been unable to use npm as a pre-build step as it always fails for the same error number 4048 (but a different file path each time).

EXEC : npm ERR! error : EPERM, rename 'C:\NPM\Cache\gulp\3.9.0\package\package.json' [d:\a\src\AppProject.csproj]
     npm ERR!     at Error (native)
     npm ERR!  { [Error: EPERM, rename 'C:\NPM\Cache\gulp\3.9.0\package\package.json']
     npm ERR!   errno: -4048,
     npm ERR!   code: 'EPERM',
     npm ERR!   path: 'C:\\NPM\\Cache\\gulp\\3.9.0\\package\\package.json',
     npm ERR!   parent: 'AppProject' }
     npm ERR! 
     npm ERR! Please try running this command again as root/Administrator.

from azure-pipelines-tasks.

The image with this fix went out on Friday/Monday. Are you still seeing the permission issue?

from azure-pipelines-tasks.

@thechrisjohnson Yes, I just queued up another build and still got the same failure message.

Build started 7/30/2015 4:39:31 PM.
...
npm install
npm ERR! Windows_NT 6.3.9600
npm ERR! argv "C:\\Program Files\\nodejs\\\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "install"
npm ERR! node v0.12.7
npm ERR! npm  v2.11.3
npm ERR! path C:\NPM\Cache\gulp\3.9.0\package\package.json
npm ERR! code EPERM
npm ERR! errno -4048

Anything else you need from me to investigate further?

from azure-pipelines-tasks.

I'm unable to repro this issue just running npm install on a test project I have. What packages are you trying to install when this occurs?

from azure-pipelines-tasks.

@thechrisjohnson It works on my machine, but only fails when running the build in VSO using the Hosted Build Controller.

from azure-pipelines-tasks.

Oh, this is a xaml based build?

from azure-pipelines-tasks.

Can you provide a package.json file that repros that we can try?

From: Werner van Deventer [mailto:[email protected]]
Sent: Thursday, July 30, 2015 2:08 PM
To: Microsoft/vso-agent-tasks [email protected]
Subject: Re: [vso-agent-tasks] npm install task fail on installing various packages (#323)

@thechrisjohnsonhttps://github.com/thechrisjohnson It works on my machine, but only fails when running the build in VSO using the Hosted Build Controller.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/323#issuecomment-126421083.

from azure-pipelines-tasks.

@thechrisjohnson
Created a codeless project just to build with Gulp, it has no assembly output.

Just some simple MSBUILD Exec tasks in the .csproj file.

<Project ToolsVersion="12.0" DefaultTargets="GulpBuild" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
...
<PropertyGroup>
  <CompileDependsOn>
    $(CompileDependsOn);
    GulpBuild;
  </CompileDependsOn>
</PropertyGroup>
...
<Target Name="GulpBuild">
    <Exec Command="if not exist C:\Users\buildguest\AppData\Roaming\npm mkdir C:\Users\buildguest\AppData\Roaming\npm" />
    <Exec Command="npm install" />
    <Exec Command="node_modules\.bin\bower install" />
    <Exec Command="gulp release" />
  </Target>

Deploy copy steps and so forth omitted since it can't get past the npm install. This was based on a number of different sites:
http://www.codecadwallader.com/2015/03/15/integrating-gulp-into-your-tfs-builds-and-web-deploy/
http://blogs.msdn.com/b/nicktrog/archive/2015/03/05/adding-gulp-to-visual-studio-online-builds.aspx
http://www.davepaquette.com/archive/2015/04/08/integrating-gulp-and-bower-with-visual-studio-online-hosted-builds.aspx

You can use any packages I guess, it fails with different names on each build so it appears random as to when exactly it fails. Here is a sample of my package.json.

{
  "name": "MyProject",
  "private": "true",
  "version": "0.0.1",
  "description": "",
  "dependencies": {},
  "devDependencies": {
    "bower": "^1.4.1",
    "browser-sync": "^2.7.12",
    "del": "~1.2.0",
    "gulp": "^3.9.0",
    "gulp-autoprefixer": "~2.3.1",
    "gulp-concat": "~2.5.2",
    "gulp-consolidate": "~0.1.2",
    "gulp-csso": "~1.0.0",
    "gulp-declare": "~0.3.0",
    "gulp-ext-replace": "^0.2.0",
    "gulp-foreach": "^0.1.0",
    "gulp-handlebars": "^4.0.0",
    "gulp-hb": "^2.6.0",
    "gulp-if": "~1.2.5",
    "gulp-imagemin": "~2.2.1",
    "gulp-jsbeautifier": "0.0.8",
    "gulp-jshint": "~1.11.0",
    "gulp-load-plugins": "~0.10.0",
    "gulp-minify-html": "~1.0.3",
    "gulp-minify-inline": "^0.1.0",
    "gulp-newer": "~0.5.0",
    "gulp-ng-annotate": "^1.0.0",
    "gulp-plumber": "~1.0.1",
    "gulp-print": "~1.1.0",
    "gulp-sass": "^2.0.3",
    "gulp-uglify": "~1.2.0",
    "gulp-usemin": "~0.3.11",
    "gulp-wrap": "~0.11.0",
    "imagemin-jpegtran": "^4.2.0",
    "imagemin-pngquant": "~4.1.0",
    "jshint-stylish": "~2.0.0",
    "merge-stream": "^0.1.7",
    "run-sequence": "~1.1.1"
  }
}

from azure-pipelines-tasks.

Thanks! I was able to repro the issue and will push out a fix within the next day or so.

from azure-pipelines-tasks.

Awesome! Let me know when it's done and I'll give it a test.

from azure-pipelines-tasks.

It would also be good to try the same thing from the new build system (+ definition from the web view)
http://aka.ms/vsopreview

from azure-pipelines-tasks.

I've tried to run my orginal build (based on new build system) and now NPM part works.

from azure-pipelines-tasks.

Awesome, thanks for confirming

from azure-pipelines-tasks.

Unfortunately not as lucky for me.

Gets a lot further but still eventually gets caught out in the middle of the npm install after 9 minutes:

##[error]EXEC(0,0): Error : EPERM, rename 'C:\NPM\Cache\process-nextick-args\1.0.2\package\package.json'
2>EXEC : npm ERR! error : EPERM, rename 'C:\NPM\Cache\process-nextick-args\1.0.2\package\package.json' [C:\a\c930acd7\MyProject\MyProject.csproj]
npm ERR!     at Error (native)
npm ERR!  { [Error: EPERM, rename 'C:\NPM\Cache\process-nextick-args\1.0.2\package\package.json']
npm ERR!   errno: -4048,
npm ERR!   code: 'EPERM',
npm ERR!   path: 'C:\\NPM\\Cache\\process-nextick-args\\1.0.2\\package\\package.json',
npm ERR!   parent: 'readable-stream' }
npm ERR! 
npm ERR! Please try running this command again as root/Administrator.

from azure-pipelines-tasks.

It looks to me like you are running NPM from within MSBuild. Is that the case? If so can you try adding /m:1 to the MSBuild args in your step?

From: Werner van Deventer [mailto:[email protected]]
Sent: Thursday, July 30, 2015 4:08 PM
To: Microsoft/vso-agent-tasks [email protected]
Cc: Chris Patterson [email protected]
Subject: Re: [vso-agent-tasks] npm install task fail on installing various packages (#323)

Unfortunately not as lucky for me.

Gets a lot further but still eventually gets caught out in the middle of the npm install after 9 minutes:

##[error]EXEC(0,0): Error : EPERM, rename 'C:\NPM\Cache\process-nextick-args\1.0.2\package\package.json'

2>EXEC : npm ERR! error : EPERM, rename 'C:\NPM\Cache\process-nextick-args\1.0.2\package\package.json' [C:\a\c930acd7\MyProject\MyProject.csproj]

npm ERR! at Error (native)

npm ERR! { [Error: EPERM, rename 'C:\NPM\Cache\process-nextick-args\1.0.2\package\package.json']

npm ERR! errno: -4048,

npm ERR! code: 'EPERM',

npm ERR! path: 'C:\NPM\Cache\process-nextick-args\1.0.2\package\package.json',

npm ERR! parent: 'readable-stream' }

npm ERR!

npm ERR! Please try running this command again as root/Administrator.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/323#issuecomment-126466001.

from azure-pipelines-tasks.

Thanks for the suggestion. I've changed a couple things and made sure that it builds perfectly on my machine with the same arguments as the new build/deploy setup.

Unfortunately it still doesn't get past the npm install step:

  npm ERR! Windows_NT 6.3.9600
  npm ERR! argv "C:\\Program Files\\nodejs\\\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "install"
  npm ERR! node v0.12.7
  npm ERR! npm  v2.11.3

  npm ERR! shasum check failed for C:\Users\BUILDG~1\AppData\Local\Temp\npm-2864-a6acc8bb\registry.npmjs.org\browser-sync-ui\-\browser-sync-ui-0.5.13.tgz
  npm ERR! Expected: b3c1661824efa957fc6a7a6dcb8db95369b6d38a
  npm ERR! Actual:   cd71ba0b316389da3b236879b63eb2a328065a38
  npm ERR! From:     https://registry.npmjs.org/browser-sync-ui/-/browser-sync-ui-0.5.13.tgz
  npm ERR! 
EXEC(0,0): Error at: 
EXEC : npm ERR! If you need help, you may report this error at:  [C:\a\06a92b97\MyProject\MyProject.csproj]
  npm ERR!     <https://github.com/npm/npm/issues>

from azure-pipelines-tasks.

Forced another build just to check it it was a dodgy download causing the SHA failure. Back with the permission errors again.

EXEC(0,0): Error : EPERM, rename 'C:\NPM\Cache\once\1.3.2\package\package.json'
EXEC : npm ERR! error : EPERM, rename 'C:\NPM\Cache\once\1.3.2\package\package.json' [C:\a\06a92b97\MyProject\MyProject.csproj]
  npm ERR!     at Error (native)
  npm ERR!  { [Error: EPERM, rename 'C:\NPM\Cache\once\1.3.2\package\package.json']
  npm ERR!   errno: -4048,
  npm ERR!   code: 'EPERM',
  npm ERR!   path: 'C:\\NPM\\Cache\\once\\1.3.2\\package\\package.json',
  npm ERR!   parent: 'end-of-stream' }
  npm ERR! 
  npm ERR! Please try running this command again as root/Administrator.

from azure-pipelines-tasks.

I believe we have this tracked down now. For some reason the permissions on that directory were modify instead of full control. We have a new image building and will be rolling in out tonight.

from azure-pipelines-tasks.

Just wanted to give some more feedback. Tested another build an hour ago and still getting the permission error. Different package this time but it's usually quite random.

EXEC(0,0): Error : EPERM, rename 'C:\NPM\Cache\inherits\2.0.1\package\package.json'
EXEC : npm ERR! error : EPERM, rename 'C:\NPM\Cache\inherits\2.0.1\package\package.json' [C:\a\06a92b97\MyProject\MyProject.csproj]
  npm ERR!     at Error (native)
  npm ERR!  { [Error: EPERM, rename 'C:\NPM\Cache\inherits\2.0.1\package\package.json']
  npm ERR!   errno: -4048,
  npm ERR!   code: 'EPERM',
  npm ERR!   path: 'C:\\NPM\\Cache\\inherits\\2.0.1\\package\\package.json',
  npm ERR!   parent: 'readable-stream' }
  npm ERR! 
  npm ERR! Please try running this command again as root/Administrator.

  npm ERR! Please include the following file with any support request:
  npm ERR!     C:\a\06a92b97\MyProject\npm-debug.log

MyProject.csproj(156,5): Error MSB3073: The command "npm install" exited with code -4048.

Let me know if there is anything else I need to provide to assist you in identifying a fix.

from azure-pipelines-tasks.

Same here:

npm ERR! Windows_NT 6.3.9600
npm ERR! argv "C:\\Program Files\\nodejs\\\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "install" "--loglevel" "error"
npm ERR! node v0.12.7
npm ERR! npm v2.11.3
npm ERR! path C:\NPM\Cache\inherits\2.0.1\package\package.json
npm ERR! code EPERM
npm ERR! errno -4048
npm ERR! Error: EPERM, rename 'C:\NPM\Cache\inherits\2.0.1\package\package.json'
npm ERR! at Error (native)
npm ERR! { [Error: EPERM, rename 'C:\NPM\Cache\inherits\2.0.1\package\package.json']
npm ERR! errno: -4048,
npm ERR! code: 'EPERM',
npm ERR! path: 'C:\\NPM\\Cache\\inherits\\2.0.1\\package\\package.json',
npm ERR! parent: 'readable-stream' }
npm ERR! 
npm ERR! Please try running this command again as root/Administrator.
npm ERR! Please include the following file with any support request:
npm ERR! C:\a\83f54761\REMOVED\npm-debug.log
Unexpected exit code -4048 returned from tool npm.cmd

from azure-pipelines-tasks.

Any updates on this issue? We're having the same problems with our VSO vNext build.

from azure-pipelines-tasks.

+1 Sometimes succeeds but most of the time fails.

from azure-pipelines-tasks.

We're working on deploying a fix to our hosted agent pools to fix this issue, it should be going out over the next few days.

from azure-pipelines-tasks.

@thechrisjohnson Can you notify us when the deployment is complete? Thanks.

from azure-pipelines-tasks.

I'm experiencing the same issues right now. Is it random? ill try a couple more builds

from azure-pipelines-tasks.

@Bartmax - are you experiencing on the hosted pool windows build?

from azure-pipelines-tasks.

@bryanmacfarlane
I'm new to build so the terminology I'm not sure.
I'm using build definitions not xaml definitions.
Picked from deployment Azure Website.
and using default hosted queue

I confirm that queuing again worked (after 3 time permission error in a row). (so I think it fails sometimes)

Let me know if you need more info please, I'm more than glad to help.

from azure-pipelines-tasks.

and I think it's windows because it calling program files directories for npm install if that was your question.

from azure-pipelines-tasks.

We are still experiencing this problem intermittently. When setting /m:1 it has become less frequent, but we still see it now and then.

I assume the fix mentioned earlier has been fully rolled out by now?

from azure-pipelines-tasks.

Yes, the fix has been rolled out. Are you seeing EPERM intermittently? I also have folks that see intermittent connection issues. Hosted build is more susceptible to that being a new machine everytime as opposed to incremental build which would go to npm much less often.

from azure-pipelines-tasks.

Yes, we are seeing this sometimes (In fact, as of late it occurs more often than not):

2015-09-08T08:20:06.7233736Z EnsureNode:
2015-09-08T08:20:06.7243740Z   NodeJS path is: 'C:\NPM\Modules'
2015-09-08T08:20:06.7243740Z RunNpm:
2015-09-08T08:20:06.7243740Z   "C:\NPM\Modules\npm" install --loglevel error --production
2015-09-08T08:21:43.8125851Z   npm ERR! Windows_NT 6.3.9600
2015-09-08T08:21:43.8146306Z   npm ERR! argv "node" "C:\\NPM\\Modules\\node_modules\\npm\\bin\\npm-cli.js" "install" "--loglevel" "error" "--production"
2015-09-08T08:21:43.8146306Z   npm ERR! node v0.12.7
2015-09-08T08:21:43.8155847Z   npm ERR! npm  v2.14.2
2015-09-08T08:21:43.8155847Z   npm ERR! path C:\NPM\Cache\readable-stream\1.0.33\package\package.json
2015-09-08T08:21:43.8165844Z   npm ERR! code EPERM
2015-09-08T08:21:43.8175844Z   npm ERR! errno -4048
2015-09-08T08:21:43.8175844Z   
2015-09-08T08:21:43.8356335Z ##[error]EXEC(0,0): Error : EPERM, rename 'C:\NPM\Cache\readable-stream\1.0.33\package\package.json'
2015-09-08T08:21:43.8365846Z EXEC : npm ERR! error : EPERM, rename 'C:\NPM\Cache\readable-stream\1.0.33\package\package.json' [C:\[Projectpath]\[ProjectName].csproj]
2015-09-08T08:21:43.8376459Z   npm ERR!     at Error (native)
2015-09-08T08:21:43.8386574Z   npm ERR!  { [Error: EPERM, rename 'C:\NPM\Cache\readable-stream\1.0.33\package\package.json']
2015-09-08T08:21:43.8396077Z   npm ERR!   errno: -4048,
2015-09-08T08:21:43.8405934Z   npm ERR!   code: 'EPERM',
2015-09-08T08:21:43.8405934Z   npm ERR!   path: 'C:\\NPM\\Cache\\readable-stream\\1.0.33\\package\\package.json',
2015-09-08T08:21:43.8416463Z   npm ERR!   parent: 'bl' }
2015-09-08T08:21:43.8426216Z   npm ERR! 
2015-09-08T08:21:43.8426216Z   npm ERR! Please try running this command again as root/Administrator.
2015-09-08T08:21:44.1325862Z   
2015-09-08T08:21:44.9570537Z   npm ERR! Please include the following file with any support request:
2015-09-08T08:21:44.9570537Z   npm ERR!     C:\[Projectpath]\npm-debug.log
2015-09-08T08:21:44.9610534Z ##[error]packages\MSBuild.Npm.0.4.4\build\MSBuild.Npm.targets(11,9): Error MSB3073: The command ""C:\NPM\Modules\npm" install --loglevel error --production" exited with code -4048.

from azure-pipelines-tasks.

Are you doing it as an npm task (in the build definition) or is npm being called as part of your msbuild step?

If it's an npm task:
As an experiment (if possible) try two npm tasks back to back with the first being continue on error (checkbox on task). What I'm curious about is when it fails - is it something consistent on that VM (permissions/state) or is it simply something intermittent in npm?

from azure-pipelines-tasks.

It is being called as a part of the build definition. The build runs a solution with three different web apps with different sets of modules. So we do already call npm three times in a build.

What we see, is that if varies what web app that gives this error, and sometimes more than one fails this way. There is no pattern: Sometimes the first project built gives the error, other times it's the second or last.

I could imagine that running the builds in paralell would cause problems, but even with /m:1 the problems continue.

from azure-pipelines-tasks.

I am facing the same issues are there any fixes available soon or will it anyhow be gone with the next node version?

Also on my environment removing parallel builds did not help. With this option it's just a little bit less likely the build will fail because of the described error.

from azure-pipelines-tasks.

This issue is tracking the npm task that this repo delivers.

The error given on the last report is from msbuild in an msbuild target.

##[error]packages\MSBuild.Npm.0.4.4\build\MSBuild.Npm.targets(11,9): Error MSB3073: The command ""C:\NPM\Modules\npm" install --loglevel error --production" exited with code -4048

from azure-pipelines-tasks.

I have a NPM Install build step which gives this error, but the package varies:

npm ERR! Windows_NT 6.3.9600
npm ERR! argv "C:\Program Files\nodejs\node.exe" "C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js" "install"
npm ERR! node v0.12.7
npm ERR! npm v2.11.3
npm ERR! path C:\NPM\Cache\inflight\1.0.4\package\package.json
npm ERR! code EPERM
npm ERR! errno -4048
npm ERR! Error: EPERM, rename 'C:\NPM\Cache\inflight\1.0.4\package\package.json'
npm ERR! at Error (native)
npm ERR! { [Error: EPERM, rename 'C:\NPM\Cache\inflight\1.0.4\package\package.json']
npm ERR! errno: -4048,
npm ERR! code: 'EPERM',
npm ERR! path: 'C:\NPM\Cache\inflight\1.0.4\package\package.json',
npm ERR! parent: 'glob' }
npm ERR!
npm ERR! Please try running this command again as root/Administrator.
npm ERR! Please include the following file with any support request:
npm ERR! C:\a\74efc3b8\G101\src\G101.App\npm-debug.log
Unexpected exit code -4048 returned from tool npm.cmd

Have tried adding another NPN Install step below the first one and setting the first one to continue on error, but still get the same result. Only had 1 build succeed so far!

from azure-pipelines-tasks.

Our success rate is 17 out of 38, less than 50%. Most of the time "run as administrator" error, sometimes "ECONNRESET".

At the moment our UI builds are completely unreliable. Really frustrating...

from azure-pipelines-tasks.

Just trying to narrow. As a troubleshooting mechanism, can you setup your own agent ( http://aka.ms/tfbuild has instructions ) in your own pool and try a bunch of builds. I'm trying to narrow down whether it's the networking in the hosted pool or it's something with the build process and/or npm.

you can email me and I can take each case and drill into specifics (hard to do here in a forum env)

from azure-pipelines-tasks.

@bryanmacfarlane What's your e-mail?

from azure-pipelines-tasks.

my alias is bryanmac and you know where I work :)

from azure-pipelines-tasks.

We've made quite a bit of progress in at least understanding the issue. Workarounds are at the bottom as we wait for the root issue.

Here is the related npm issue:
npm/npm#9696

That issue has the repro and the logs etc.. The issue is when you have an npm graph with a high number of duplicates (bunch of gulp tasks is the easy repro) then when it pulls the package, as npm is writing to the cache folder, something parallel/async is not synchronized and it tries to rename the same package.json in the cache folder and it fails with an EPERM. We also noticed this happens more frequently on slower I/O machines (VMs etc...). My fast local SSD box on Win10 never reproduces it.

Workarounds that avoid the issue on hosted:

1. Pass --force to the npm install cmd. This force it to install from the service everytime and it avoids the cache (which is where the issue is).
2. Invoke npm install task multiple times in your build definition. On all but the last check continue on error. It will get further each time and if not using --force, it will get farther with the aid of the cache.
3. Deploy your agent on your own azure VM. Since the agent is stable (not blown away each build) the cache gets warm and mitigates the issue. This also has the benefit of speeding up the build avoiding the redownloading of packages on every build.

We will also be adding retries within the task next sprint.

from azure-pipelines-tasks.

Can you point me on directions for workaround #3? Thanks.

from azure-pipelines-tasks.

@Bartmax I recommend a DS virtual machine with Premium Storage for better I/O. We have a very high success rate with it so far. Lots of or build have failed with a regular VM.

from azure-pipelines-tasks.

Thanks for avoiding me the huslle. Quite expensive for my needs. I think I'm moving away from tfs ci. Not worth it yet. I hope it will improve with time.

from azure-pipelines-tasks.

@Bartmax It's not really a VSO issue. It is definitely an NPM issue. We have experimented with a Linux machine as well. It seems to be more stable, but as soon as you want to run multiple builds (NPM install) at the same time on the same machine the same locking problem appears (different file system locations, different sessions, etc...).
When we run the npm install command manually on our dev machines (Windows 8.1 + SSD), or in the cloud (Windows Server 2012 R2), we get some fails, but mostly it succeeds. Regular VMs have produced a lot of failed builds. Only the high speed VM worked for us. And on the other hand by upgrading our build machine, our builds are up to 60% faster. Which is cool, because we have ~50 CI builds.

from azure-pipelines-tasks.

@Bartmax - instructions for #3 are @ http://aka.ms/tfbuild
Also note that #1 is a simple workaround that didn't slow down the build (hosted build). Of course #3 with high performance I/O will get you faster builds (and incremental which is key here) but at a cost.

As @Peter-Juhasz noted, these are just workarounds for the core issue - an issue in npm writing to the cache folder when you have a complex dependency graph (typical with gulp). Slower I/O exposes the issue and it will happen with any build system. I reproduced it from the command line on a plain new VM per my write-up in the npm issue.

from azure-pipelines-tasks.

Also, we should note that next sprint we will put retries in the task so you don't have to do #1 - each invocation will get you farther as you are protected by the cache (even on hosted build where the machine is cleaned).

from azure-pipelines-tasks.

It seems this thread may be going cold as I haven't seen anything here or on the npm thread Bryan linked to since late September. The --force workaround mentioned above doesn't always work either. Below is a build log from yesterday with the failure in the npm install step with the --force arg. I'm using the hosted agent pool in VSO and the out-of-the-box npm install build step. You'll notice this isn't the EPERM error, rather the ECONNRESET error, but we get either one intermittently and I'm guessing it's the same issue. Has there been any progress on this that anyone knows of? Thanks!

2015-11-05T14:28:22.3720730Z Executing the powershell script: C:\LR\MMS\Services\Mms\TaskAgentProvisioner\Tools\agents\default\tasks\NpmInstall\0.1.3\NpmInstall.ps1
2015-11-05T14:28:23.4800859Z C:\Program Files\nodejs\npm.cmd install --force
2015-11-05T14:28:35.8044835Z ##[warning]npm WARN using --force I sure hope you know what you are doing.
2015-11-05T14:29:41.3481288Z > [email protected] install C:\a\1\s\EP.WebClient\node_modules\phantomjs
2015-11-05T14:29:41.3491294Z > node install.js
2015-11-05T14:29:42.1581291Z Downloading https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.8-windows.zip
2015-11-05T14:29:42.1591294Z Saving to C:\Users\BUILDG1\AppData\Local\Temp\phantomjs\phantomjs-1.9.8-windows.zip
2015-11-05T14:29:42.1601298Z Receiving...
2015-11-05T14:29:42.6556257Z Received 7292K total.
2015-11-05T14:29:42.6576249Z Extracting zip contents
2015-11-05T14:29:44.0906265Z Removing C:\a\1\s\EP.WebClient\node_modules\phantomjs\lib\phantom
2015-11-05T14:29:44.0916263Z Copying extracted folder C:\Users\BUILDG1\AppData\Local\Temp\phantomjs\phantomjs-1.9.8-windows.zip-extract-1446733782656\phantomjs-1.9.8-windows -> C:\a\1\s\EP.WebClient\node_modules\phantomjs\lib\phantom
2015-11-05T14:29:46.3083449Z Writing location.js file
2015-11-05T14:29:46.3103450Z Done. Phantomjs binary available at C:\a\1\s\EP.WebClient\node_modules\phantomjs\lib\phantom\phantomjs.exe
2015-11-05T14:30:13.9478260Z > [email protected] install C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\engine.io\node_modules\ws\node_modules\utf-8-validate
2015-11-05T14:30:13.9499145Z > node-gyp rebuild
2015-11-05T14:30:14.0018258Z C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\engine.io\node_modules\ws\node_modules\utf-8-validate>if not defined npm_config_node_gyp (node "C:\Program Files\nodejs\node_modules\npm\bin\node-gyp-bin....\node_modules\node-gyp\bin\node-gyp.js" rebuild ) else (node rebuild )
2015-11-05T14:30:47.9372367Z Building the projects in this solution one at a time. To enable parallel build, please add the "/m" switch.
2015-11-05T14:31:01.2509324Z validation.cc
2015-11-05T14:31:13.4701084Z C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include\limits(214): warning C4577: 'noexcept' used with no exception handling mode specified; termination on exception is not guaranteed. Specify /EHsc [C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\engine.io\node_modules\ws\node_modules\utf-8-validate\build\validation.vcxproj]
2015-11-05T14:31:20.5208976Z Creating library C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\engine.io\node_modules\ws\node_modules\utf-8-validate\build\Release\validation.lib and object C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\engine.io\node_modules\ws\node_modules\utf-8-validate\build\Release\validation.exp
2015-11-05T14:31:20.5268996Z Generating code
2015-11-05T14:31:21.2963213Z Finished generating code
2015-11-05T14:31:23.0896089Z validation.vcxproj -> C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\engine.io\node_modules\ws\node_modules\utf-8-validate\build\Release\validation.node
2015-11-05T14:31:23.1806074Z ##[error]npm ERR! Windows_NT 6.3.9600
2015-11-05T14:31:23.1816091Z ##[error]npm ERR! argv "C:\Program Files\nodejs\node.exe" "C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js" "install" "--force"
2015-11-05T14:31:23.1816091Z ##[error]npm ERR! node v0.12.7
2015-11-05T14:31:23.1826093Z ##[error]npm ERR! npm v2.11.3
2015-11-05T14:31:23.1836088Z ##[error]npm ERR! code ECONNRESET
2015-11-05T14:31:23.1836088Z ##[error]npm ERR! errno ECONNRESET
2015-11-05T14:31:23.1846092Z ##[error]npm ERR! syscall read
2015-11-05T14:31:23.1846092Z ##[error]npm ERR! network read ECONNRESET
2015-11-05T14:31:23.1856086Z ##[error]npm ERR! network This is most likely not a problem with npm itself
2015-11-05T14:31:23.1856086Z ##[error]npm ERR! network and is related to network connectivity.
2015-11-05T14:31:23.1866093Z ##[error]npm ERR! network In most cases you are behind a proxy or have bad network settings.
2015-11-05T14:31:23.1886089Z ##[error]npm ERR! network
2015-11-05T14:31:23.1896094Z ##[error]npm ERR! network If you are behind a proxy, please make sure that the
2015-11-05T14:31:23.1906093Z ##[error]npm ERR! network 'proxy' config is set properly. See: 'npm help config'
2015-11-05T14:31:23.1916096Z > [email protected] install C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\engine.io\node_modules\ws\node_modules\bufferutil
2015-11-05T14:31:23.1916096Z > node-gyp rebuild
2015-11-05T14:31:23.1956094Z C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\engine.io\node_modules\ws\node_modules\bufferutil>if not defined npm_config_node_gyp (node "C:\Program Files\nodejs\node_modules\npm\bin\node-gyp-bin....\node_modules\node-gyp\bin\node-gyp.js" rebuild ) else (node rebuild )
2015-11-05T14:31:24.9346206Z Building the projects in this solution one at a time. To enable parallel build, please add the "/m" switch.
2015-11-05T14:31:25.3976236Z bufferutil.cc
2015-11-05T14:31:26.6336320Z C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include\limits(214): warning C4577: 'noexcept' used with no exception handling mode specified; termination on exception is not guaranteed. Specify /EHsc [C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\engine.io\node_modules\ws\node_modules\bufferutil\build\bufferutil.vcxproj]
2015-11-05T14:31:26.8426386Z Creating library C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\engine.io\node_modules\ws\node_modules\bufferutil\build\Release\bufferutil.lib and object C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\engine.io\node_modules\ws\node_modules\bufferutil\build\Release\bufferutil.exp
2015-11-05T14:31:26.8966331Z Generating code
2015-11-05T14:31:27.2668555Z Finished generating code
2015-11-05T14:31:27.7978584Z bufferutil.vcxproj -> C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\engine.io\node_modules\ws\node_modules\bufferutil\build\Release\bufferutil.node
2015-11-05T14:31:27.8618591Z > [email protected] install C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\socket.io-client\node_modules\engine.io-client\node_modules\ws\node_modules\utf-8-validate
2015-11-05T14:31:27.8618591Z > node-gyp rebuild
2015-11-05T14:31:27.8718598Z C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\socket.io-client\node_modules\engine.io-client\node_modules\ws\node_modules\utf-8-validate>if not defined npm_config_node_gyp (node "C:\Program Files\nodejs\node_modules\npm\bin\node-gyp-bin....\node_modules\node-gyp\bin\node-gyp.js" rebuild ) else (node rebuild )
2015-11-05T14:31:29.7993872Z Building the projects in this solution one at a time. To enable parallel build, please add the "/m" switch.
2015-11-05T14:31:30.1703955Z validation.cc
2015-11-05T14:31:31.5963992Z C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include\limits(214): warning C4577: 'noexcept' used with no exception handling mode specified; termination on exception is not guaranteed. Specify /EHsc [C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\socket.io-client\node_modules\engine.io-client\node_modules\ws\node_modules\utf-8-validate\build\validation.vcxproj]
2015-11-05T14:31:31.9144016Z Creating library C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\socket.io-client\node_modules\engine.io-client\node_modules\ws\node_modules\utf-8-validate\build\Release\validation.lib and object C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\socket.io-client\node_modules\engine.io-client\node_modules\ws\node_modules\utf-8-validate\build\Release\validation.exp
2015-11-05T14:31:31.9204017Z Generating code
2015-11-05T14:31:32.2444029Z Finished generating code
2015-11-05T14:31:32.6984068Z validation.vcxproj -> C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\socket.io-client\node_modules\engine.io-client\node_modules\ws\node_modules\utf-8-validate\build\Release\validation.node
2015-11-05T14:31:32.7294064Z > [email protected] install C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\socket.io-client\node_modules\engine.io-client\node_modules\ws\node_modules\bufferutil
2015-11-05T14:31:32.7304055Z > node-gyp rebuild
2015-11-05T14:31:32.7374057Z C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\socket.io-client\node_modules\engine.io-client\node_modules\ws\node_modules\bufferutil>if not defined npm_config_node_gyp (node "C:\Program Files\nodejs\node_modules\npm\bin\node-gyp-bin....\node_modules\node-gyp\bin\node-gyp.js" rebuild ) else (node rebuild )
2015-11-05T14:31:34.6165673Z Building the projects in this solution one at a time. To enable parallel build, please add the "/m" switch.
2015-11-05T14:31:34.9955704Z bufferutil.cc
2015-11-05T14:31:36.2335780Z C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include\limits(214): warning C4577: 'noexcept' used with no exception handling mode specified; termination on exception is not guaranteed. Specify /EHsc [C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\socket.io-client\node_modules\engine.io-client\node_modules\ws\node_modules\bufferutil\build\bufferutil.vcxproj]
2015-11-05T14:31:36.5954953Z Creating library C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\socket.io-client\node_modules\engine.io-client\node_modules\ws\node_modules\bufferutil\build\Release\bufferutil.lib and object C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\socket.io-client\node_modules\engine.io-client\node_modules\ws\node_modules\bufferutil\build\Release\bufferutil.exp
2015-11-05T14:31:36.6954960Z Generating code
2015-11-05T14:31:37.0984982Z Finished generating code
2015-11-05T14:31:37.6648305Z bufferutil.vcxproj -> C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\socket.io\node_modules\socket.io-client\node_modules\engine.io-client\node_modules\ws\node_modules\bufferutil\build\Release\bufferutil.node
2015-11-05T14:31:37.6988342Z > [email protected] install C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents
2015-11-05T14:31:37.6998357Z > node-pre-gyp install --fallback-to-build
2015-11-05T14:31:39.0979884Z C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents>if not defined npm_config_node_gyp (node "C:\Program Files\nodejs\node_modules\npm\bin\node-gyp-bin....\node_modules\node-gyp\bin\node-gyp.js" clean ) else (node clean )
2015-11-05T14:31:39.4999910Z C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents>if not defined npm_config_node_gyp (node "C:\Program Files\nodejs\node_modules\npm\bin\node-gyp-bin....\node_modules\node-gyp\bin\node-gyp.js" configure --fallback-to-build --module=C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents\lib\binding\Release\node-v14-win32-x64\fse.node --module_name=fse --module_path=C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents\lib\binding\Release\node-v14-win32-x64 ) else (node configure --fallback-to-build --module=C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents\lib\binding\Release\node-v14-win32-x64\fse.node --module_name=fse --module_path=C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents\lib\binding\Release\node-v14-win32-x64 )
2015-11-05T14:31:40.8039993Z ##[error]Traceback (most recent call last):
2015-11-05T14:31:40.8049977Z ##[error] File "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\gyp_main.py", line 18, in
2015-11-05T14:31:40.8059976Z ##[error] sys.exit(gyp.script_main())
2015-11-05T14:31:40.8059976Z ##[error] File "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp__init__.py", line 533, in script_main
2015-11-05T14:31:40.8069975Z ##[error] return main(sys.argv[1:])
2015-11-05T14:31:40.8069975Z ##[error] File "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp__init__.py", line 526, in main
2015-11-05T14:31:40.8079972Z ##[error] return gyp_main(args)
2015-11-05T14:31:40.8089982Z ##[error] File "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp__init__.py", line 511, in gyp_main
2015-11-05T14:31:40.8089982Z ##[error] generator.GenerateOutput(flat_list, targets, data, params)
2015-11-05T14:31:40.8099970Z ##[error] File "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\generator\msvs.py", line 2000, in GenerateOutput
2015-11-05T14:31:40.8129982Z ##[error] sln_projects, project_objects, flat=msvs_version.FlatSolution())
2015-11-05T14:31:40.8139982Z ##[error] File "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\generator\msvs.py", line 1759, in _GatherSolutionFolders
2015-11-05T14:31:40.8169981Z ##[error] return _DictsToFolders('', root, flat)
2015-11-05T14:31:40.8179980Z ##[error] File "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\generator\msvs.py", line 1712, in _DictsToFolders
2015-11-05T14:31:40.8209969Z ##[error] for folder, contents in bucket.iteritems():
2015-11-05T14:31:40.8219982Z ##[error]AttributeError: 'MSVSProject' object has no attribute 'iteritems'
2015-11-05T14:31:40.8369994Z ##[error]gyp ERR! configure error
2015-11-05T14:31:40.8379993Z ##[error]gyp ERR! stack Error: gyp failed with exit code: 1
2015-11-05T14:31:40.8379993Z ##[error]gyp ERR! stack at ChildProcess.onCpExit (C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\lib\configure.js:355:16)
2015-11-05T14:31:40.8389997Z ##[error]gyp ERR! stack at ChildProcess.emit (events.js:110:17)
2015-11-05T14:31:40.8399989Z ##[error]gyp ERR! stack at Process.ChildProcess._handle.onexit (child_process.js:1074:12)
2015-11-05T14:31:40.8399989Z ##[error]gyp ERR! System Windows_NT 6.3.9600
2015-11-05T14:31:40.8409992Z ##[error]gyp ERR! command "node" "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js" "configure" "--fallback-to-build" "--module=C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents\lib\binding\Release\node-v14-win32-x64\fse.node" "--module_name=fse" "--module_path=C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents\lib\binding\Release\node-v14-win32-x64"
2015-11-05T14:31:40.8419989Z ##[error]gyp ERR! cwd C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents
2015-11-05T14:31:40.8419989Z ##[error]gyp ERR! node -v v0.12.7
2015-11-05T14:31:40.8429985Z ##[error]gyp ERR! node-gyp -v v2.0.1
2015-11-05T14:31:40.8429985Z ##[error]gyp ERR! not ok
2015-11-05T14:31:40.8519991Z ##[error]node-pre-gyp ERR! build error
2015-11-05T14:31:40.8529995Z ##[error]node-pre-gyp ERR! stack Error: Failed to execute 'node-gyp.cmd configure --fallback-to-build --module=C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents\lib\binding\Release\node-v14-win32-x64\fse.node --module_name=fse --module_path=C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents\lib\binding\Release\node-v14-win32-x64' (1)
2015-11-05T14:31:40.8539998Z ##[error]node-pre-gyp ERR! stack at ChildProcess. (C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents\node_modules\node-pre-gyp\lib\util\compile.js:83:29)
2015-11-05T14:31:40.8549993Z ##[error]node-pre-gyp ERR! stack at ChildProcess.emit (events.js:110:17)
2015-11-05T14:31:40.8549993Z ##[error]node-pre-gyp ERR! stack at maybeClose (child_process.js:1015:16)
2015-11-05T14:31:40.8559994Z ##[error]node-pre-gyp ERR! stack at Process.ChildProcess._handle.onexit (child_process.js:1087:5)
2015-11-05T14:31:40.8559994Z ##[error]node-pre-gyp ERR! System Windows_NT 6.3.9600
2015-11-05T14:31:40.8569988Z ##[error]node-pre-gyp ERR! command "node" "C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents\node_modules\node-pre-gyp\bin\node-pre-gyp" "install" "--fallback-to-build"
2015-11-05T14:31:40.8579988Z ##[error]node-pre-gyp ERR! cwd C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents
2015-11-05T14:31:40.8579988Z ##[error]node-pre-gyp ERR! node -v v0.12.7
2015-11-05T14:31:40.8589998Z ##[error]node-pre-gyp ERR! node-pre-gyp -v v0.6.15
2015-11-05T14:31:40.8589998Z ##[error]node-pre-gyp ERR! not ok
2015-11-05T14:31:40.8600086Z Failed to execute 'node-gyp.cmd configure --fallback-to-build --module=C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents\lib\binding\Release\node-v14-win32-x64\fse.node --module_name=fse --module_path=C:\a\1\s\EP.WebClient\node_modules\karma\node_modules\chokidar\node_modules\fsevents\lib\binding\Release\node-v14-win32-x64' (1)
2015-11-05T14:31:40.9899996Z ##[error]npm ERR! Windows_NT 6.3.9600
2015-11-05T14:31:40.9920075Z ##[error]npm ERR! argv "C:\Program Files\nodejs\node.exe" "C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js" "install" "--force"
2015-11-05T14:31:40.9929999Z ##[error]npm ERR! node v0.12.7
2015-11-05T14:31:40.9940000Z ##[error]npm ERR! npm v2.11.3
2015-11-05T14:31:40.9950005Z ##[error]npm ERR! shasum check failed for C:\Users\BUILDG~1\AppData\Local\Temp\npm-2456-06618d10\registry.npmjs.org\babel-core-\babel-core-5.8.33.tgz
2015-11-05T14:31:40.9959996Z ##[error]npm ERR! Expected: 77e63eacff2072c695a827297a6d91b43546b2f4
2015-11-05T14:31:40.9959996Z ##[error]npm ERR! Actual: 37a677cdeca5a6b04f3567c66a75451075fcb6cc
2015-11-05T14:31:40.9969997Z ##[error]npm ERR! From: https://registry.npmjs.org/babel-core/-/babel-core-5.8.33.tgz
2015-11-05T14:31:40.9980004Z ##[error]npm ERR!
2015-11-05T14:31:40.9980004Z ##[error]npm ERR! If you need help, you may report this error at:
2015-11-05T14:31:40.9990000Z ##[error]npm ERR! https://github.com/npm/npm/issues
2015-11-05T14:31:40.9990000Z ##[error]npm ERR! Please include the following file with any support request:
2015-11-05T14:31:41.0000000Z ##[error]npm ERR! C:\a\1\s\EP.WebClient\npm-debug.log
2015-11-05T14:31:41.0759993Z ##[error]Unexpected exit code 1 returned from tool npm.cmd

from azure-pipelines-tasks.

Funny, I was just looking at this thread again this morning and dismayed to see no updates since September. We tried the --force parameter as well and it seemed to work for a bit but in the last 2-3 weeks the NPM failures have become a daily occurrence again, actually several times a day on our CI builds. Our build set up is very simple:

The builds usually fail with this report:

Summary | Build
6255 error(s), 4 warning(s)
Build - 6255 error(s), 4 warning(s)
(): npm ERR! Windows_NT 6.3.9600
(): npm ERR! argv "C:\Program Files\nodejs\node.exe" "C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js" "install" "--force"
(): npm ERR! node v0.12.7
(): npm ERR! npm v2.11.3
(): npm ERR! code ECONNRESET
(): npm ERR! errno ECONNRESET
(): npm ERR! syscall read
(): npm ERR! network read ECONNRESET
(): npm ERR! network This is most likely not a problem with npm itself
(): npm ERR! network and is related to network connectivity.

It know it was a suggestion early one to create your build server with high I/O throughput but I'd like to avoid another server to maintain for our simplistic build needs. The new vNext build system is really nice but these errors are starting to interrupt our production workflow. May I suggest that MS investigate creating a second tier of higher performing build servers for those who need it.

-Paul

from azure-pipelines-tasks.

Same problem here:

npm ERR! Windows_NT 6.3.9600
npm ERR! argv "C:\Program Files\nodejs\node.exe" "C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js" "install"
npm ERR! node v0.12.7
npm ERR! npm v2.11.3
npm ERR! code ECONNRESET
npm ERR! errno ECONNRESET
npm ERR! syscall read
npm ERR! network read ECONNRESET
npm ERR! network This is most likely not a problem with npm itself
npm ERR! network and is related to network connectivity.
npm ERR! network In most cases you are behind a proxy or have bad network settings.
npm ERR! network
npm ERR! network If you are behind a proxy, please make sure that the
npm ERR! network 'proxy' config is set properly. See: 'npm help config'

Absolutely defeats using the unreliable hosted build environment...

from azure-pipelines-tasks.

Though I am not using vso-agent-tasks this was the only place I've found a solid connection to the errors we've been running into doing npm install on AWS EC2 instances. EPERM, rename errors would come up more often than not but we could never reproduce it locally. When we logged into the boxes we couldn't reproduce it. Following @bryanmacfarlane's idea I bumped up the EC2 machines to be i2.xlarge which have better disk I/O performance and I haven't had anymore problems. I had tried M3 large, M4 large, C4, C3 before this. We are running Windows Server 2012 and 2008. I'll report back if this doesn't continue working. Thanks!

from azure-pipelines-tasks.

@barretts - note that you don't necessarily have to jump to the expensive xlarge etc... VMs. There's an option in azure called Premium Storage that you opt for on disks. We use for D2 skus.

@tempworks - we're investigating just giving Premium I/O to the hosted pool. But we're running tests now. Hopefully you can just get it but we'll see where that lands. We're doing it this sprint.

from azure-pipelines-tasks.

@bryanmacfarlane thanks for the Azure tip! we've been evaluating them as well for the larger array of Windows versions and Edge browser testing so that will come in handy

I see some of you having ECONNRESET issues, we were able to resolve that by pointing NPM to the http registry after reading http://stackoverflow.com/a/18428563/604861

from azure-pipelines-tasks.

@barretts - thanks we'll investigate the ECONRESET (diff issue than this thread).

from azure-pipelines-tasks.

@bryanmacfarlane So to recap the recent discussions today.

Hosting your own vso-agents with high performing I/O will not experience this problem.

There is no promise of a delivery or timeline of high performing vso-agent tiers.

So for those experiencing this problem, it's Microsoft's opinion is to host your own vso-agents.

from azure-pipelines-tasks.

Hi I'm having the same problem:

npm ERR! Windows_NT 6.3.9600
npm ERR! argv "C:\\Program Files\\nodejs\\\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "install"
npm ERR! node v0.12.7
npm ERR! npm v2.11.3
npm ERR! shasum check failed for C:\Users\BUILDG~1\AppData\Local\Temp\npm-2792-3de7b145\registry.npmjs.org\less\-\less-1.7.5.tgz
npm ERR! Expected: 4f220cf7288a27eaca739df6e4808a2d4c0d5756
npm ERR! Actual: 3acfbcebee77bd1ef7a044c90a48b97316af4b6c
npm ERR! From: https://registry.npmjs.org/less/-/less-1.7.5.tgz
npm ERR!
npm ERR! If you need help, you may report this error at:
npm ERR! <https://github.com/npm/npm/issues>
npm ERR! Please include the following file with any support request:
npm ERR! C:\a\1\s\dist\npm-debug.log
Unexpected exit code 1 returned from tool npm.cmd
******************************************************************************
Finishing task: NpmInstall
******************************************************************************
Task NpmInstall failed. This caused the job to fail. Look at the logs for the task for more details.
******************************************************************************

from azure-pipelines-tasks.

@JustSayNO That doesn't look like the same problem. This thread is about an EPERM race condition in npm. It's not a catch all for all errors that could come out of npm

from azure-pipelines-tasks.

Sorry about that I was on a call to a Microsoft person in New Zealand and we both mistakenly got the impression this issue was a similar thing because of the above comment by @brutaldev and @pescado that does include an error about :

 shasum check failed for .....

I'll look through issues and if I cannot find one for this issue then I will open one.

from azure-pipelines-tasks.

@JustSayNO - thanks. Very soon, I'm going to summarize this thread/issue into an npm doc/faq.

from azure-pipelines-tasks.

@bryanmacfarlane I wish I had better news but even with those over powered machines I still got hit by the error. I've thrown up my hands and implemented an npm install retry batch file for my situation. I haven't seen it error twice in a row yet but I've got it setup to try 3 times before failing. Hope y'all can figure out what the actual issue is. Good luck!

Edit: overpowered being the i2.xlarge which was said might not be needed.

from azure-pipelines-tasks.

@barretts - Thanks. I don't know what over powered machines means.

The issue is a deep npm dependency graph with a high number of duplicates in that graph with race conditions over locking and updating files.

Here's the root issue: npm/npm#9696

from azure-pipelines-tasks.

@barretts
Here's the specific option for disks in azure to minimize: https://azure.microsoft.com/en-us/documentation/articles/storage-premium-storage-preview-portal

As I stated earlier, we are investigating enabling on hosted build machines this sprint.

from azure-pipelines-tasks.

Can't this be solved it instead trying to lower the ocurrence with SSD disks (expensive solution) ?

TravisCI works with no issue on the free tier without SSD.

from azure-pipelines-tasks.

@Bartmax - the npm bug is a race condition that repros on windows with low I/O. This isn't a CI issue - I sent npm a repro (way up in the thread) with a package.json and npm cmd line (no CI involved). We just run cmd line just like Travis. Travis also doesn't build on windows.

And yes, as I pointed out in the thread way above, there's an inexpensive workaround which is to pass --force as additional args to npm. That causes it to bypass the cache which avoids the race and deep graphs with multiple writes without locking to the cache dir ( npm/npm#9696 ). We are considering just doing that on hosted (with retries like other CIs do) since it's always a fresh machine and the cache doesn't help much.

from azure-pipelines-tasks.

We're getting this error using the NPM task (with --force) on our self-hosted instance consistently (we just set it up in the last couple of days):

npm ERR! syscall rename
npm ERR! Error: EPERM: operation not permitted, rename     'C:\Users\tfsbuild\AppData\Roaming\npm-    cache\minimist\1.2.0\package\package.json.414d5643225a5c0e9714b0cdc0eaf8b4' ->     'C:\Users\tfsbuild\AppData\Roaming\npm-cache\minimist\1.2.0\package\package.json'
npm ERR! at Error (native)
npm ERR! { [Error: EPERM: operation not permitted, rename 'C:\Users\tfsbuild\AppData\Roaming\npm-cache\minimist\1.2.0\package\package.json.414d5643225a5c0e9714b0cdc0eaf8b4' -> 'C:\Users\tfsbuild\AppData\Roaming\npm-cache\minimist\1.2.0\package\package.json']
npm ERR! errno: -4048,
npm ERR! code: 'EPERM',
npm ERR! syscall: 'rename',
npm ERR! path: 'C:\\Users\\tfsbuild\\AppData\\Roaming\\npm-cache\\minimist\\1.2.0\\package\\package.json.414d5643225a5c0e9714b0cdc0eaf8b4',
npm ERR! dest: 'C:\\Users\\tfsbuild\\AppData\\Roaming\\npm-cache\\minimist\\1.2.0\\package\\package.json',
npm ERR! parent: 'gulp-util' }
npm ERR!
npm ERR! Please try running this command again as root/Administrator.

Is there any other way to get this working?

from azure-pipelines-tasks.

@tghamm - npm is writing to the cache folder:

EPERM: operation not permitted, rename 'C:\Users\tfsbuild\AppData\Roaming\npm-cache ...

Which implies --force is not being passed (force fetches and bypassed the cache)

Can you:

• gather your CI steps and inputs
• set system.debug to true on the variables tab of the definition
• gather task output
• since you have your own vm, gather the npm log the output refers to
• one more - your package.json (or at a minimum the dependencies section)
  Zip that up and send it bigbldt at microsoft dot com

from azure-pipelines-tasks.

@bryanmacfarlane will do. It's actually working intermittently now so I'm waiting for it to fail again.

from azure-pipelines-tasks.

Using the "--force" works for now but is far from an optimal solution. We're trying to roll out a rather large pipeline using npm right now and I'm afraid this might get a serious issue.

from azure-pipelines-tasks.

@konrad-ha - agreed. The only right fix is npm fixing the race condition. --force, retries, and faster I/O are all workarounds/mitigations.

from azure-pipelines-tasks.

How about a focused hackathon by Microsoft dedicated to fix this for npm?

from azure-pipelines-tasks.

I have been running my own VSO agent on a VM with premium storage data disk for 10 days and even though it dramatically improves the situation (and build times) - I still see the occasional EPERM error (very rare though so far)

from azure-pipelines-tasks.

what about a central cache/disk for npm in the microsoft infrastructure. Instead of downloading everything, the vm can attach the cache drive and download there or use it as cache improving speed by a lot and prevent this kind of errors. (same can be done for nuget).

if the vm is run without cache (some option?), no disk will be attached and you can opt in for the current behavior (for those needing that stuff actually gets downloaded)

Personally, It will make me very happy to use hosted agent and don't waste download speed / bandwidth.

from azure-pipelines-tasks.

Our team city agents also run into this issue. Using the --force argument does not work unfortunately. Here is a sample build log:

    [11:25:59][Step 1/1] Execute npm_install (13m:23s)
    [11:25:59][Execute npm_install] Running npm with parameters: ["install", "--force"]
    [11:26:03][Execute npm_install] npm WARN using --force I sure hope you know what you are doing.
    ...
    ...
    11:38:47][Execute npm_install] npm ERR! Windows_NT 6.1.7601
    [11:38:48][Execute npm_install] npm ERR! argv "C:\\Program Files\\nodejs\\\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "install" "--force"
    [11:38:48][Execute npm_install] npm ERR! node v0.12.7
    [11:38:48][Execute npm_install] npm ERR! npm  v2.11.3
    [11:38:48][Execute npm_install] npm ERR! path C:\Users\teamcityagent\AppData\Roaming\npm-cache\inherits\2.0.1\package\package.json
    [11:38:48][Execute npm_install] npm ERR! code EPERM
    [11:38:48][Execute npm_install] npm ERR! errno -4048
    [11:38:48][Execute npm_install] 
    [11:38:48][Execute npm_install] npm ERR! Error: EPERM, rename 'C:\Users\teamcityagent\AppData\Roaming\npm-cache\inherits\2.0.1\package\package.json'
    [11:38:48][Execute npm_install] npm ERR!     at Error (native)
    [11:38:48][Execute npm_install] npm ERR!  { [Error: EPERM, rename 'C:\Users\teamcityagent\AppData\Roaming\npm-cache\inherits\2.0.1\package\package.json']
    [11:38:48][Execute npm_install] npm ERR!   errno: -4048,
    [11:38:48][Execute npm_install] npm ERR!   code: 'EPERM',
    [11:38:48][Execute npm_install] npm ERR!   path: 'C:\\Users\\teamcityagent\\AppData\\Roaming\\npm-cache\\inherits\\2.0.1\\package\\package.json',
    [11:38:48][Execute npm_install] npm ERR!   parent: 'npmconf' }
    [11:38:48][Execute npm_install] npm ERR! 
    [11:38:48][Execute npm_install] npm ERR! Please try running this command again as root/Administrator.
    [11:39:22][Execute npm_install] 

Our agents are running on Windows 7 and currently it is not feasible to upgrade the agents to use an SSD...

from azure-pipelines-tasks.

Lots of progress in finding the root issue from Microsoft open tech. The issue is in npm2. More here: npm/npm#9696

The good news is it's fixed in npm3 ([sudo] npm install npm@3). We're progressing to see if a fix can be pushed to npm2.

We are looking to move hosted build pool to npm3 and we're almost done with premium storage in the hosted pool (even if npm3 fixes, still much faster I/O).

So to sum up the workarounds:

• own VM with premium storage, and/or
• own VM with npm3
• on hosted pool use --force

from azure-pipelines-tasks.

Very cool!

from azure-pipelines-tasks.

@Maarten88 +1

from azure-pipelines-tasks.

Hosted build image is getting npm3 this sprint.

Note that if you have your own agent and you install npm3 (or node5.x) and it resolves in your path (confirm with npm -v), then you can use the npm task (no need use command line task but that's fine as well).

from azure-pipelines-tasks.

We had to roll back npm3 because it broke Cordova so it will be a bit longer before we can move to it.

From: Bryan MacFarlane [mailto:[email protected]]
Sent: Friday, January 15, 2016 8:09 AM
To: Microsoft/vso-agent-tasks [email protected]
Cc: Chris Patterson [email protected]
Subject: Re: [vso-agent-tasks] npm install task (not msbuild) fails with EPERM on hosted build (#323)

Hosted build image is getting npm3 this sprint.

Note that if you have your own agent and you install npm3 (or node5.x) and it resolves in your path (confirm with npm -v), then you can use the npm task (no need use command line task but that's fine as well).

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/323#issuecomment-171955379.

from azure-pipelines-tasks.

What is the progress on this?

Would it help to simply include node_modules into the repository and basically skip the npm install? Is it a good idea?

from azure-pipelines-tasks.

I would not do that. You kind of break the concept behind npm install. npm install npm@3 takes usually less than 2 minutes for us. npm install of course has to download quite some stuff (depending on project), but thats the point, to keep the dependencies up to date and to latest release.

from azure-pipelines-tasks.

Progress:

Microsoft open tech fixed the root problem in npm2.
npm/npm#9696
npm/npm@e982858

When that is released, we will update our hosted image to it. On your machines you can do the same or install npm@3

We are also in the process of rolling out premium storage disks to our hosted pool scale units at no cost to our customers. Early numbers show significant I/O gains and have cut the time in half for some of our sample builds. This will minimize the likelyhood of hitting the bug.

This upcoming sprint, we will offer a drop down in the npm task to specify which npm version. It will be leveraging a tools concept we will be adding: https://github.com/Microsoft/vso-agent-tasks/blob/master/docs/tools.md

from azure-pipelines-tasks.

Good news Bryan and thanks to MS open tech. Will there be an announcement on a blog somewhere when the upcoming sprint is deployed? Better yet would be a note here, it would be fantastic for those subscribed to this thread. -Paul

from azure-pipelines-tasks.

Thanks for the information. If I understand you correctly: when using the Hosted Agent, we need to wait until the fix is released. However, the upcoming sprint will bring some improvements sooner and the fix will be released a bit later.

from azure-pipelines-tasks.