larsw / npm-leech Goto Github PK
View Code? Open in Web Editor NEWSmall utility to leech all direct and transitive npm packages for a given package.json or package-lock.json file.
License: MIT License
Small utility to leech all direct and transitive npm packages for a given package.json or package-lock.json file.
License: MIT License
Hi,
While using npm-leech to download a package with a lot of dependencies, some dependencies failed to download with an error message like this:
pkg-err { Error: Request failed with status code 405
at createError (/usr/lib/node_modules/npm-leech/node_modules/axios/lib/core/createError.js:16:15)
at settle (/usr/lib/node_modules/npm-leech/node_modules/axios/lib/core/settle.js:17:12)
at IncomingMessage.handleStreamEnd (/usr/lib/node_modules/npm-leech/node_modules/axios/lib/adapters/http.js:236:11)
at emitNone (events.js:91:20)
at IncomingMessage.emit (events.js:185:7)
at endReadableNT (_stream_readable.js:978:12)
at _combinedTickCallback (internal/process/next_tick.js:80:11)
at process._tickCallback (internal/process/next_tick.js:104:9)
config:
{ url: 'http://registry.npmjs.org/accepts/~1.3.7',
...
_header: 'GET /accepts/~1.3.7 HTTP/1.1\r\nAccept: application/json, text/plain, */*\r\nUser-Agent: axios/0.19.2\r\nHost: registry.npmjs.org\r\nConnection: close\r\n\r\n',
...
responseUrl: 'http://registry.npmjs.org/accepts/~1.3.7',
redirects: [],
read: [Function] } },
response:
{ status: 405,
statusText: 'Method Not Allowed',
The package.json used with npm-leech version 1.2.1:
"name": "foo",
"version:": "1.0.0",
"dependencies": {
"express": "latest"
}
}
The package 'express' dependencies, only showing the two first:
npm show express --json dependencies
{
"accepts": "~1.3.7",
"array-flatten": "1.1.1",
The dependency 'array-flatten' has a "normal" semver and is downloaded without problems.
But the dependency 'accepts' has a tihlde in the semver. This causes npm-leech to download this version without resolving the tihlde to a "normal" semver first. But the registry does not understand "GET http://registry.npmjs.org/accepts/~1.3.7".
Using wireshark, this is the TCP stream when downloading the package 'accepts':
GET /accepts/~1.3.7 HTTP/1.1
Accept: application/json, text/plain, */*
User-Agent: axios/0.19.2
Host: registry.npmjs.org
Connection: close
HTTP/1.1 405 Method Not Allowed
Date: Wed, 05 May 2021 15:47:03 GMT
Content-Type: application/json
Content-Length: 63
Connection: close
Set-Cookie: __cfduid=db421dde183435235cd4cb62992f582121620229622; expires=Fri, 04-Jun-21 15:47:02 GMT; path=/; domain=.npmjs.org; HttpOnly; SameSite=Lax
CF-Ray: 64ab1ae7687ffac0-OSL
Allow: PUT
CF-Cache-Status: DYNAMIC
cf-request-id: 09decf24a30000fac0cd049000000001
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
{"code":"MethodNotAllowedError","message":"GET is not allowed"}
So, shouldn't npm-leech resolve tihlde and caret semvers into normal semvers before downloading packages?
Hi,
In he npmjs.org registry scoped packages has a tarball path like this:
# npm info '@types/qs@latest' --json dist.tarball
"https://registry.npmjs.org/@types/qs/-/qs-6.9.6.tgz"
I.e. "@scope/packagename/-/packagename-version.tgz"
When downloaded by npm-leech, the file structure in the tar file is:
# tar tvf npm-tarballs.tar
-rw-r--r-- 0/0 2405 2021-05-05 18:23 @types/qs-6.9.6.tgz
When this tar-file is imported to Artifactory using the web-interface, this layout from the tar file is kept in Artifactory. This is not the "correct" path inside Artifactory, it uses a path like this for scoped packages:
@scope/packagename/~/@scope/packagename-version.tgz
When importing the tar file from npm-leech into Artifactory, the incorrect layout will cause issues with scoped packages that also exist without a scope with the same name and version. Like "@types/qs" and "qs". It will cause checksum error when trying to install (npm install) a package like this using Artifactory as a registry. See this bug for more info:
https://www.jfrog.com/jira/browse/RTFACT-7668
Several issues like this is reported for Artifactory, but no response from the developer that I could see.
Pushing the packages to Artifactory using "npm publish" will go through the Artifactory API and store the file with the layout expected by Artifactory.
But I hoped this could be fixed in npm-leech by adding a option to store in Artifactory layout inside the tar file. This way the bulk import of the tar file from Artifactory web interface would work out of the box.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.