unjs / giget Goto Github PK
View Code? Open in Web Editor NEW✨ Download templates and git repositories with pleasure!
License: MIT License
✨ Download templates and git repositories with pleasure!
License: MIT License
Support a built-in provider to automatically resolve from a URL to a tar file
❯ node -v
v20.10.0
❯ npm -v
10.2.3
❯ npx [email protected] gh:unjs/template
/Users/byron/.npm/_npx/a37ce48c408dc016/node_modules/.bin/giget: line 1: import: command not found
/Users/byron/.npm/_npx/a37ce48c408dc016/node_modules/.bin/giget: line 2: import: command not found
/Users/byron/.npm/_npx/a37ce48c408dc016/node_modules/.bin/giget: line 3: import: command not found
/Users/byron/.npm/_npx/a37ce48c408dc016/node_modules/.bin/giget: line 4: import: command not found
/Users/byron/.npm/_npx/a37ce48c408dc016/node_modules/.bin/giget: line 5: import: command not found
/Users/byron/.npm/_npx/a37ce48c408dc016/node_modules/.bin/giget: line 6: import: command not found
/Users/byron/.npm/_npx/a37ce48c408dc016/node_modules/.bin/giget: line 7: import: command not found
/Users/byron/.npm/_npx/a37ce48c408dc016/node_modules/.bin/giget: line 8: import: command not found
/Users/byron/.npm/_npx/a37ce48c408dc016/node_modules/.bin/giget: line 9: import: command not found
/Users/byron/.npm/_npx/a37ce48c408dc016/node_modules/.bin/giget: line 10: import: command not found
/Users/byron/.npm/_npx/a37ce48c408dc016/node_modules/.bin/giget: line 11: import: command not found
/Users/byron/.npm/_npx/a37ce48c408dc016/node_modules/.bin/giget: line 12: import: command not found
/Users/byron/.npm/_npx/a37ce48c408dc016/node_modules/.bin/giget: line 13: import: command not found
/Users/byron/.npm/_npx/a37ce48c408dc016/node_modules/.bin/giget: line 14: import: command not found
/Users/byron/.npm/_npx/a37ce48c408dc016/node_modules/.bin/giget: line 15: import: command not found
/Users/byron/.npm/_npx/a37ce48c408dc016/node_modules/.bin/giget: line 17: const: command not found
/Users/byron/.npm/_npx/a37ce48c408dc016/node_modules/.bin/giget: line 18: const: command not found
/Users/byron/.npm/_npx/a37ce48c408dc016/node_modules/.bin/giget: line 19: const: command not found
/Users/byron/.npm/_npx/a37ce48c408dc016/node_modules/.bin/giget: line 20: const: command not found
/Users/byron/.npm/_npx/a37ce48c408dc016/node_modules/.bin/giget: line 21: const: command not found
/Users/byron/.npm/_npx/a37ce48c408dc016/node_modules/.bin/giget: line 22: const: command not found
/Users/byron/.npm/_npx/a37ce48c408dc016/node_modules/.bin/giget: line 23: const: command not found
/Users/byron/.npm/_npx/a37ce48c408dc016/node_modules/.bin/giget: line 24: const: command not found
/Users/byron/.npm/_npx/a37ce48c408dc016/node_modules/.bin/giget: line 25: .:: command not found
/Users/byron/.npm/_npx/a37ce48c408dc016/node_modules/.bin/giget: line 26: import:: command not found
/Users/byron/.npm/_npx/a37ce48c408dc016/node_modules/.bin/giget: line 27: types:: command not found
/Users/byron/.npm/_npx/a37ce48c408dc016/node_modules/.bin/giget: line 28: default:: command not found
/Users/byron/.npm/_npx/a37ce48c408dc016/node_modules/.bin/giget: line 29: },: command not found
/Users/byron/.npm/_npx/a37ce48c408dc016/node_modules/.bin/giget: line 30: require:: command not found
/Users/byron/.npm/_npx/a37ce48c408dc016/node_modules/.bin/giget: line 31: types:: command not found
/Users/byron/.npm/_npx/a37ce48c408dc016/node_modules/.bin/giget: line 32: default:: command not found
/Users/byron/.npm/_npx/a37ce48c408dc016/node_modules/.bin/giget: line 33: syntax error near unexpected token }' /Users/byron/.npm/_npx/a37ce48c408dc016/node_modules/.bin/giget: line 33:
}'
Look like the cli file not run in node environment.
I think the reason for that is in here.
No response
No response
stream/promises
was introduced in Node 15.
When I try to download ts, you add .json
at the end. Why ?
From what I have tested so far for private gitlab repo is
export NODE_EXTRA_CA_CERTS=/path/to/cert.pem
in .zshrc
/giget/shared/gitget.xxxx.mjs
https://gitlab.domain.com/${parsed.repo}/tree/${parsed.ref}${parsed.subdir}
,https://gitlab.domain.com/${parsed.repo}/-/archive/${parsed.ref}.tar.gz
is there anything I can do for gitlab custom domain, for example to set GIGET_GITLAB_URL in .env so that it will pick up custom domain?
GIGET_REGISTRY
, GIGET_AUTH
in .envThe only downside is I need to modify the gitlab domain myself.
_Originally posted by @hanzoz in nuxt/nuxt#13367
OS: Windows 10
Node: v16.17.0
i created a new private empty bitbucket repo for this issue, so i don't mind sharing the key :).
npx giget bitbucket:riad_hachemane/test-giget-bitbucket --auth kuhyURjJB7gMoqURJMIP
@pi0
am having trouble with a private bitbucket repo, the auth token is not working,
i created a new private empty bitbucket repo for this issue, so i don't mind sharing the key :).
steps i made to get to this issue:
--auth
, doesn't work ❌No response
No response
Hello 👋
I'm using a Nuxt layer with the GitHub repo approach which lacks the versioning we would get if it were a npm package.
Hence I want to make sure that I'm using the latest layer commit and otherwise fail to catch bugs as early as possible.
Currently, If I set any incorrect GIGET_AUTH
env (Repo is private) it doesn't throw an error and uses the currently folder in node_modules
.
A flag that throws an error and/or warning if fetching the repo fails would be great! Thanks a lot!
3 months and counting... is nobody else confused?
Line 13 in fbce78a
Perhaps an origin story unfolding...
Hey! After seeing this being published, definitely gonna be my default cloner after using degit/tiged. Curious what are the differences between other packages? I'd love to help contribute the readme.
Node: v18.15.0
Nuxt: v3.7.0
Using nuxt:
Using pure giget:
Either way the fetch will fail
I am working on a Nuxt Layers project. Nuxt uses uses unjs/giget
internally to fetch remote git repositories, but is unable to fetch private Bitbucket repositories. I already raised an issue at the Nuxt repository, but was forwarded to create an issue here.
nuxt setup:
GIGET_AUTH=XXXXXXXXXXXXXXXX`
export default defineNuxtConfig({
extends: [
'bitbucket:user_name/project_name#branch_name',
],
// ...
})
Internally unjs/giget converts this to https://bitbucket.org/user_name/project_name/get/branch_name.tar.gz
.
If I try to execute this with npx, the fetch fails:
npx giget@latest https://bitbucket.org/user_name/project_name/get/branch_name.tar.gz --auth XXXXXXXXXXXXXXXX
In my opionion something like https://api.bitbucket.org/2.0/repositories/user_name/project_name/src/branch_name/
would be more correct.
I found an issue for private Github repos which seems pretty similar to my problem. The issue was fixed but is not published yet. Someone also already addressed the problem with private Bitbucket repositories. However the response was insufficient to me.
Am I missing something? Is this a known bug? If so, will it be fixed in a future release? Any help is appreciated.
No response
No response
I'd like to inform the user what is the git commit hash that is being downloaded when a branch or tag is passed.
Since giget doesn't depend on git, we'd need to assume that the tarball downloaded from the providers are generated using git archive
.
If that's the case, the commit id could be retrieved from the tarball.
Example:
wget https://codeload.github.com/unjs/giget/tar.gz/tags/v1.1.2
zcat v1.1.2 | git get-tar-commit-id
8ddd2a4eb6045710c1bac9761e97152cf6af4b8a
I've been happy to see that you've added support for accessing private repos via Token.
In our organisation we use the new fine-graded tokens. Which permissions do we need to set in order to use giget?
Content + Metadata readonly doesn't seem to work (returns a 404 error).
Would be great to get some documentation
Thanks a lot! :)
>= 18
^1.2.1
In your terminal, run:
npx giget@latest --verbose github:bigcommerce/catalyst/apps/core#@bigcommerce/[email protected]
Expected Result:
Giget should successfully clone bigcommerce-catalyst
to bigcommerce-catalyst
at commit SHA 5955b5aeb0894ddb6748e4b60745899f1720f6ec
(the commit SHA associated with tag @bigcommerce/[email protected]
).
Actual Result:
Giget clones main
instead of the desired tag. You'll notice the logs confirm this:
npx giget@latest --verbose github:bigcommerce/catalyst/apps/core#@bigcommerce/[email protected]
[giget] Downloaded https://api.github.com/repos/bigcommerce/catalyst/tarball/main to /Users/matt.volk/.cache/giget/github/bigcommerce-catalyst/main.tar.gz in 537ms
parseGitURI
's inputRegex
is as follows:
Line 49 in 2398346
We have a repository (bigcommerce/catalyst) containing tags with @
characters (example: https://github.com/bigcommerce/catalyst/tree/@bigcommerce/[email protected])
Since the @
character is not included in the <ref>
character set [\w./-]
, the tag above won't be matched by this pattern, and will fallback to main
.
Even if we try to URI encode the tag name (so that it turns into %40bigcommerce%2Fcatalyst-core%400.1.0
), the character %
is also not included in the <ref>
character set, so it still falls back to main
.
The following change seems to fix the issue, though I'm not sure if there was an intentional reason to omit @
from the inputRegex
string:
diff --git a/src/_utils.ts b/src/_utils.ts
index 6e6d8c4..f6d56db 100644
--- a/src/_utils.ts
+++ b/src/_utils.ts
@@ -46,7 +46,7 @@ export async function download(
}
const inputRegex =
- /^(?<repo>[\w.-]+\/[\w.-]+)(?<subdir>[^#]+)?(?<ref>#[\w./-]+)?/;
+ /^(?<repo>[\w.-]+\/[\w.-]+)(?<subdir>[^#]+)?(?<ref>#[\w./@-]+)?/;
export function parseGitURI(input: string): GitInfo {
const m = input.match(inputRegex)?.groups || {};
[giget] Downloaded https://api.github.com/repos/bigcommerce/catalyst/tarball/main to /Users/matt.volk/.cache/giget/github/bigcommerce-catalyst/main.tar.gz in 537ms
[giget] Extracted to /Users/matt.volk/code/bigcommerce-catalyst in 105ms
✨ Successfully cloned bigcommerce-catalyst to bigcommerce-catalyst
We could improve online performance by reducing one HTTP redirect and directly use codeload.github.com
.
https://codeload.github.com/unjs/template/tar.gz/refs/heads/main
The challenge is that we must guess the ref type and change heads/
to either tags/
or commits/
.
Hello,
Currently, there is a default registry for unjs template. To use a custom registry, we have to use --registry
and pass a long URL like https://raw.githubusercontent.com/unjs/giget/main/templates
.
I'm currently creating my own registry. To fetch from my registry, I've to pass the long URL which is annoying. I could put it in a var env but this will overwrite the unjs registry.
Using a config file with unjs/rc9, we could easily support multiple registries. We could also support multiple registries in var env using a separator like a comma.
If it's ok for you, I'm ok to try to implement this feature.
Node v16.19.0, pnpm, macOS
Your own CI
Since the latest update (1.1.0), downloadTemplate
returns a 400 on Node 16 while hitting GitHub. This happens in giget
CI, but is also breaking people downstream, including create-astro
.
The original PR (#77) also had this test fail.
No response
Failed to download https://github.com/unjs/template/archive/main.tar.gz: 400 Bad Request
To avoid cd into the new repo after the cloning, add on the CLI an opt-in feature to do the directory change automatically
Bonus: an opt-out feature to automatically install deps using ni
?
https://github.com/unjs/nanotar
nanotar is much more lightweight.
Only reason we cannot simply migrate from current tar
package is that it relies on modern Node 18+ API / Web API for .gzip
decompression which will be a breaking change for package.
When trying to giget
a template that doesn't exist,
$ giget gh:unjs/empty
the folder ends up being created:
|- unjs-empty
Should we make the behavior of giget
and git
consistent, Clear the pre-created folder when the template doesn't exist ?
Ref: https://github.com/nuxt/nuxt.js/issues/15101
we shall honor http_proxy
/HTTP_PROXY
and https_proxy
/ HTTPS_PROXY
values to suppory proxy.
/
try pnpm dlx giget@latest nitro .
to view how it fails
/
No response
No response
Hey Pooya,
Congrats on another exceptionally useful repository.
I regularly have need to download examples – not templates – from repos, for example:
These kind of examples folder setups are pretty common; I wonder if a direct URL download option could be added to the package, such as:
# download single folder
npx giget@latest https://github.com/wxt-dev/wxt-examples/tree/main/examples/vue
# download several folders
https://github.com/wxt-dev/wxt-examples/tree/main/examples/vue*
That way it's a super-easy copy/paste job.
Could perhaps the folders be filtered after the download of the main repo?
I took a look at the source files, and perhaps it might even be manageable for me to produce a PR!
LMK, and thanks again for all the great code 🙏
[email protected]
Node 20.7.0
npm 10.1.0
pnpm 8.15.1
npx giget@latest nitro nitro-app --install
or
pnpx giget@latest nitro nitro-app --install
Because nypm's package manager detection depends on either the lockfile being present or the package manager being specified in package.json, following the quick start guide for Nitro, which uses --install
, causes it to complain that no package managers were detected.
No response
> pnpm dlx giget@latest nitro --install
Packages: +34
++++++++++++++++++++++++++++++++++
Progress: resolved 34, reused 34, downloaded 0, added 34, done
ERROR No package manager auto-detected.
Try to install any GitLab repository that is located in a subgroup.
In GitLab, you can seperate your projects into subgroups. Currently its not possible to download projects in subgroups due to how repository urls are resolved.
For example runningnpx giget@latest gitlab:group/subgroup/project
will try download the repository from https://gitlab.com/group/subgroup/-/archive/main.tar.gz
.
No response
No response
any
Try to fetch a file that is within a private repo using the registry provider.
Cannot fetch json to then download template since registry is private and needs auth token to access.
No response
No response
Shouldn't make a difference, tested with Node v18.15.0
Attempt a download of any git branch that includes slashes.
The ref capturing group of inputRegex
used by parseGitURI
is too strict, does not include /
.
This can result in failed download attempts.
This might not technically
be a bug, but there's no downside to adjusting the regex to allow slashes afaik.
It would be nice to get this working as org/repo#user/ABC-123
is a pretty common pattern.
I have adjusted the regex and added a test for it, only opening this issue to reference it in the PR.
Resolved by #102
No response
If I already have a template project locally, can I get a new project by copying the template?
Does giget support it?
Normally giget uses tarbal snapshot downloads. But for supporting a 3rd party provider, we might support generic git provider supporting git:
and git via HTTPS (.git
).
We might need some internal refactors too for providers giving a directory instead of tarball path OR that this git providers makes a tarbal after cloning.
Hi :) Let's say we have a repository name within a directory in GitLab, such as xxx/dir/website-1#master
. The regular expression you are using here is not working for this type of repository name.
Line 49 in 83eb734
resulting:
"url": "https://git.xxx.com/xxx/dir/tree/master/website-1",
and should be
"url": "https://git.xxx.com/xxx/dir/website-1/tree/master/website-1-master.tar.bz"
Whith own registry, not working
Whith own registry, not working
Whith own registry, not working
No response
No response
This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.
These updates have all been created already. Click a checkbox below to force a retry/rebase of any.
@types/node
, @vitest/coverage-v8
, eslint-config-unjs
, jiti
, pnpm
, prettier
, vitest
)These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.
.github/workflows/autofix.yml
actions/checkout v4
actions/setup-node v4
autofix-ci/action ea32e3a12414e6d3183163c3424a7d7a8631ad84
.github/workflows/ci.yml
actions/checkout v4@692973e3d937129bcbf40652eb9f2f61becf3332
actions/setup-node v4
codecov/codecov-action v4
package.json
citty ^0.1.6
consola ^3.2.3
defu ^6.1.4
node-fetch-native ^1.6.4
nypm ^0.3.8
ohash ^1.1.3
pathe ^1.1.2
tar ^6.2.1
@types/node ^20.12.7
@types/tar ^6.1.13
@vitest/coverage-v8 ^1.5.0
changelogen ^0.5.5
eslint ^8.57.0
eslint-config-unjs ^0.2.1
jiti ^1.21.0
prettier ^3.2.5
typescript ^5.4.5
unbuild ^2.0.0
vitest ^1.5.0
pnpm 9.0.5
Node: v18.12.1
Package: npx giget@latest
npx giget@latest gh:TechAkayy/icones
Look closely at this screenshot:
Cloned repo - gh:TechAkayy/icones
Status message - Successfully cloned TechAkayy-icones to TechAkayy-icones
No response
No response
Bun: 1.0.25
Git repo: https://github.com/tkjaergaard/giget-bun-reproduction
Steps to reproduce
bun install
to install dependencies$ bun build index.ts --outdir=dist --target=node
Error
11 | import { fetch } from 'node-fetch-native/proxy';
^
error: No matching export in "node_modules/node-fetch-native/dist/proxy.cjs" for import "fetch"
at /Users/thomaskjaergaard/projects/internal/giget-bun-reproduction/node_modules/giget/dist/index.mjs:11:10
Originally posted by @tkjaergaard in #140 (comment)
node 20.6.1
giget 1.1.3
Start any proxy server with logging:
// https://github.com/berstend/straightforward
$ npx straightforward
straightforward forward-proxy running on localhost:9191
Set environment variable
$env:HTTPS_PROXY="http://127.0.0.1:9191"
export HTTPS_PROXY="http://127.0.0.1:9191"
Run curl -x http://127.0.0.1:9191 https://example.com
to confirm the proxy is working and is printing proxy connections:
CONNECT example.com:443
Run npx giget github:withastro/astro/examples/basics#latest ./output --force-clean --verbose
.
No connection is made through the proxy.
giget should use the proxy set in environment variable.
https://github.com/unjs/node-fetch-native/blob/47e92fdae1356d3970e40dfafa5614d2c0f68a87/src/index.ts#L22-L29
giget by default is using node's built in fetch
in node 20.
The built-in fetch
is the same as undici.fetch
.
https://github.com/nodejs/node/blob/3e14cfbbcfcedbb18235ffd5cc28983ac5d5ca52/lib/internal/process/pre_execution.js#L314-L316
undici does not support http.Agent
but uses a different API: https://undici.nodejs.org/#/docs/api/ProxyAgent
Workaround: set the environment variable FORCE_NODE_FETCH
.
No response
giget 1.2.1
Node 18
Nuxt 3.8.2, 3.9.0, 3.9.1
Nitro 2.8.1
Sorry unable to provide repro as this only appears in workers environment
This was reported on Cloudflare, Vercel
Does not appear in NodeJS, local environments
Initial bug report was filed in Nuxt-Security there
We initially thought the issue was on our side but were unable to reproduce
Then one user filed logs, and we realized the error was triggered by giget
while trying to use node-fetch-native
in Workers environments
nitropack
to 2.8.1[email protected]
upgraded node-fetch-native
to 1.6.1giget
is trying to use node-fetch-native
's proxy feature, which was not available in 1.4.1node-fetch-native
1.4.1This looks like a version dependency mismatch on node-fetch-native
This is very strange because package managers should usually be able to resolve the correct version without any issue
In fact, in NodeJS environments the issue does not exist
However for some unknown reason, it seems to affect Workers environment
(node-resolve plugin) Could not resolve import "node-fetch-native/native/proxy" in /opt/_work/myproject/node_modules/.pnpm/[email protected]/node_modules/giget/dist/index.mjs using exports defined in /opt/_work/myproject/node_modules/.pnpm/[email protected]/node_modules/node-fetch-native/package.json.
Error: [nitro] Error: Cannot resolve "node-fetch-native/native/proxy" from "/opt/_work/myproject/node_modules/.pnpm/[email protected]/node_modules/giget/dist/index.mjs" and externals are not allowed!
undefined
Error: Cannot resolve "node-fetch-native/native/proxy" from "/opt/_work/myproject/node_modules/.pnpm/[email protected]/node_modules/giget/dist/index.mjs" and externals are not allowed!
at Object.resolveId (node_modules/.pnpm/[email protected]/node_modules/nitropack/dist/shared/nitro.4ea992bc.mjs:1973:17)
at async PluginDriver.hookFirstAndGetPlugin (node_modules/.pnpm/[email protected]/node_modules/rollup/dist/es/shared/node-entry.js:18542:28)
at async resolveId (node_modules/.pnpm/[email protected]/node_modules/rollup/dist/es/shared/node-entry.js:17211:26)
at async ModuleLoader.resolveId (node_modules/.pnpm/[email protected]/node_modules/rollup/dist/es/shared/node-entry.js:17625:15)
at async Object.resolveId (node_modules/.pnpm/@[email protected][email protected]/node_modules/@rollup/plugin-commonjs/dist/es/index.js:588:10)
at async PluginDriver.hookFirstAndGetPlugin (node_modules/.pnpm/[email protected]/node_modules/rollup/dist/es/shared/node-entry.js:18542:28)
at async resolveId (node_modules/.pnpm/[email protected]/node_modules/rollup/dist/es/shared/node-entry.js:17211:26)
at async ModuleLoader.resolveId (node_modules/.pnpm/[email protected]/node_modules/rollup/dist/es/shared/node-entry.js:17625:15)
at async PluginDriver.hookFirstAndGetPlugin (node_modules/.pnpm/[email protected]/node_modules/rollup/dist/es/shared/node-entry.js:18542:28)
at async resolveId (node_modules/.pnpm/[email protected]/node_modules/rollup/dist/es/shared/node-entry.js:17211:26)
Error: Cannot resolve "node-fetch-native/native/proxy" from "/opt/_work/myproject/node_modules/.pnpm/[email protected]/node_modules/giget/dist/index.mjs" and externals are not allowed!
ELIFECYCLE Command failed with exit code 1.
Error: Process completed with exit code 1.
giget tries requesting https://github.com/yjl9903/<my-private-repo>/archive/main.tar.gz
without any auth info, so that it receives 404 Not Found
.
Github already provides this locally if you auth with gh
in this case it can cover most private repo cases without creating a new token . This can used by trying to run it with execa fallbacks to undefined
E.g
options.auth = options.auth || (await execa('gh', ['auth', 'token']).then(r => r.stdout).catch(() => undefined))
Maybe it can be a CLI param like --use-gh-auth
that enables this function to that might be more viable.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.