Comments (4)
deployable
commented on July 29, 2024
2
So it looks like the cause of the truncations was external. There's a router this data was going via that also manages vpn connections that was performing particularly badly today. I dropped all the vpn connections and then the npm-register timeouts that also truncate appear to have stopped happening. The router is possibly dumping the occasional tcp connection.
For the app side of it, I can put together a PR from my testing that does a shasum check on the stream and will only move the downloaded file into place if it matches the expected shasum. That should cover any type of tcp or file weirdness that could possibly happen and not leave the registry in a state with a bad/incomplete file. Does that sound useful?
The file deletion on timeout seems to make things slower, at least in my slightly contrived testing examples. If there happens to be something big or slow that regularly hits the timeout then the file never appears and the client has to retry a lot. At least without the delete, the timed out client connection ends up with a downloaded a file, eventually.
I also needed to add a bunch of debug logging to look at this as I couldn't reproduce it in a debugger. I replaced a lot of the console output with a winston logger that supports timestamps and log levels. Even using something simple like debug would be useful. If you're interested in something like that let me know and I'll pull it out into a PR too.
from npm-register.
deployable
commented on July 29, 2024
The corrupt files have consistently had their ends truncated so far.
± |logging_and_files {1} ✓| →tar -tvf tmp/tarballs/typescript/typescript-2.0.3/33dec9eae86b8eee327dd419ca050c853cabd514.tgz
-rw-rw-rw- 0 0 0 2985 22 Sep 15:34 package/package.json
-rw-rw-rw- 0 0 0 178 22 Sep 15:08 package/.npmignore
-rw-rw-rw- 0 0 0 4311 22 Sep 15:34 package/README.md
-rw-rw-rw- 0 0 0 3486 22 Sep 15:08 package/ThirdPartyNoticeText.txt
-rw-rw-rw- 0 0 0 2443 22 Sep 15:08 package/AUTHORS.md
-rw-rw-rw- 0 0 0 9008 22 Sep 15:34 package/CONTRIBUTING.md
-rw-rw-rw- 0 0 0 834 22 Sep 15:08 package/CopyrightNotice.txt
-rw-rw-rw- 0 0 0 41273 22 Sep 15:34 package/Gulpfile.ts
-rw-rw-rw- 0 0 0 9197 22 Sep 15:08 package/LICENSE.txt
-rw-rw-rw- 0 0 0 8538 22 Sep 15:08 package/.mailmap
-rw-rw-rw- 0 0 0 45 22 Sep 15:08 package/bin/tsc
-rw-rw-rw- 0 0 0 50 22 Sep 15:08 package/bin/tsserver
-rw-rw-rw- 0 0 0 275 22 Sep 15:08 package/lib/README.md
-rw-rw-rw- 0 0 0 3542841 22 Sep 15:34 package/lib/typescript.js
-rw-rw-rw- 0 0 0 2765717 22 Sep 15:34 package/lib/tsserverlibrary.js
-rw-rw-rw- 0 0 0 2774735 22 Sep 15:34 package/lib/tsserver.js
-rw-rw-rw- 0 0 0 2025376 22 Sep 15:34 package/lib/tsc.js
-rw-rw-rw- 0 0 0 3542841 22 Sep 15:34 package/lib/typescriptServices.js
-rw-rw-rw- 0 0 0 1692 22 Sep 15:08 package/lib/lib.es2017.sharedmemory.d.ts
-rw-rw-rw- 0 0 0 1370 22 Sep 15:08 package/lib/lib.es2015.d.ts
-rw-rw-rw- 0 0 0 15136 22 Sep 15:34 package/lib/lib.es2015.iterable.d.ts
-rw-rw-rw- 0 0 0 9159 22 Sep 15:08 package/lib/lib.es2015.promise.d.ts
-rw-rw-rw- 0 0 0 1920 22 Sep 15:34 package/lib/lib.es2015.proxy.d.ts
-rw-rw-rw- 0 0 0 1952 22 Sep 15:08 package/lib/lib.es2015.reflect.d.ts
-rw-rw-rw- 0 0 0 1910 22 Sep 15:08 package/lib/lib.es2015.symbol.d.ts
-rw-rw-rw- 0 0 0 12268 22 Sep 15:08 package/lib/lib.es2015.symbol.wellknown.d.ts
-rw-rw-rw- 0 0 0 4645 22 Sep 15:08 package/lib/lib.es2016.array.include.d.ts
-rw-rw-rw- 0 0 0 852204 22 Sep 15:34 package/lib/lib.d.ts
-rw-rw-rw- 0 0 0 1018 22 Sep 15:08 package/lib/lib.es2017.d.ts
-rw-rw-rw- 0 0 0 1584 22 Sep 15:08 package/lib/lib.es2017.object.d.ts
-rw-rw-rw- 0 0 0 1296 22 Sep 15:08 package/lib/lib.es2015.generator.d.ts
-rw-rw-rw- 0 0 0 197351 22 Sep 15:34 package/lib/lib.es5.d.ts
-rw-rw-rw- 0 0 0 911987 22 Sep 15:34 package/lib/lib.es6.d.ts
-rw-rw-rw- 0 0 0 9360 22 Sep 15:08 package/lib/lib.scripthost.d.ts
-rw-rw-rw- 0 0 0 41129 22 Sep 15:34 package/lib/lib.webworker.d.ts
-rw-rw-rw- 0 0 0 21276 22 Sep 15:34 package/lib/lib.es2015.core.d.ts
-rw-rw-rw- 0 0 0 2425 22 Sep 15:34 package/lib/lib.es2015.collection.d.ts
-rw-rw-rw- 0 0 0 344693 22 Sep 15:34 package/lib/tsserverlibrary.d.ts
-rw-rw-rw- 0 0 0 1181 22 Sep 15:08 package/lib/lib.dom.iterable.d.ts
-rw-rw-rw- 0 0 0 100233 22 Sep 15:34 package/lib/typescript.d.ts
-rw-rw-rw- 0 0 0 647044 22 Sep 15:34 package/lib/lib.dom.d.ts
-rw-rw-rw- 0 0 0 100219 22 Sep 15:34 package/lib/typescriptServices.d.ts
-rw-rw-rw- 0 0 0 970 22 Sep 15:08 package/lib/lib.es2016.d.ts
-rw-rw-rw- 0 0 0 103 22 Sep 15:51 package/test.config
± |logging_and_files {1} ✓| →tar -tvf tmp/tarballs/typescript/typescript-2.0.3/33dec9eae86b8eee327dd419ca050c853cabd514.tgz.trunc
-rw-rw-rw- 0 0 0 2985 22 Sep 15:34 package/package.json
-rw-rw-rw- 0 0 0 178 22 Sep 15:08 package/.npmignore
-rw-rw-rw- 0 0 0 4311 22 Sep 15:34 package/README.md
-rw-rw-rw- 0 0 0 3486 22 Sep 15:08 package/ThirdPartyNoticeText.txt
-rw-rw-rw- 0 0 0 2443 22 Sep 15:08 package/AUTHORS.md
-rw-rw-rw- 0 0 0 9008 22 Sep 15:34 package/CONTRIBUTING.md
-rw-rw-rw- 0 0 0 834 22 Sep 15:08 package/CopyrightNotice.txt
-rw-rw-rw- 0 0 0 41273 22 Sep 15:34 package/Gulpfile.ts
-rw-rw-rw- 0 0 0 9197 22 Sep 15:08 package/LICENSE.txt
-rw-rw-rw- 0 0 0 8538 22 Sep 15:08 package/.mailmap
-rw-rw-rw- 0 0 0 45 22 Sep 15:08 package/bin/tsc
-rw-rw-rw- 0 0 0 50 22 Sep 15:08 package/bin/tsserver
-rw-rw-rw- 0 0 0 275 22 Sep 15:08 package/lib/README.md
-rw-rw-rw- 0 0 0 3542841 22 Sep 15:34 package/lib/typescript.js
-rw-rw-rw- 0 0 0 2765717 22 Sep 15:34 package/lib/tsserverlibrary.js
-rw-rw-rw- 0 0 0 2774735 22 Sep 15:34 package/lib/tsserver.js
-rw-rw-rw- 0 0 0 2025376 22 Sep 15:34 package/lib/tsc.js
tar: Truncated input file (needed 2025472 bytes, only 0 available)
tar: Error exit delayed from previous errors.
from npm-register.
jdx
commented on July 29, 2024
I put together a first pass at fixing this. Let me know what you think.
from npm-register.
deployable
commented on July 29, 2024
Cool thanks, although same problem testing this! Interestingly it's occurring more frequently now. Something odd is going on here, let me check a few things...
from npm-register.
Related Issues (20)
- Heroku Button Deploy not working HOT 1
- Incompatible with node 10
- Docker hub image HOT 2
- Passing authorization details from env vars? HOT 2
- Permission denied when trying to create /data/tarballs
- Unsupported operation: `npm audit` HOT 1
- Unhandled rejection TypeError: Cannot read property '0.0.3' of undefined HOT 5
- heroku build fails because of "Outdated Yarn lockfile" HOT 3
- Circle CI Access Key for AWS causes Failed Integration HOT 6
- How to disable caching? HOT 6
- NPM Installing packages 404's HOT 1
- TypeError: Cannot promisify an API that has normal methods with 'Async'-suffix HOT 5
- Circle CI 2.0 Migration HOT 1
- Add Search to UI
- AWS > 2.304 Breaks Bluebird Promisify in NPM-Register HOT 4
- Support AWS SDK > 2.304 and Bluebird PromisifyAll
- Protect frontend ?
- Dockerfile is broken HOT 1
- Is this project maintained?
- Offering to add free integrity verification
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from npm-register.