This module is used to batch git clone the large file system
needed lib:
git-lfs: https://askubuntu.com/questions/799341/how-to-install-git-lfs-on-ubuntu-16-04
GitPython : https://gitpython.readthedocs.io/en/stable/tutorial.html
Run under back ground:
sudo nohup python3 gitDownloader.py &
Check the process or stop:
sudo ps ax | grep python3
Check folder size:
du -sh /mnt/storage/llmModules/Vicuna-7B/
| Model | Download link available | Download state | License |
|---|---|---|---|
| GPT-4 | No | No | Proprietary |
| Claude-1 | No | No | Proprietary |
| Claude-2 | No | No | Proprietary |
| Claude-instant-1 | No | No | Proprietary |
| GPT-3.5-turbo | Yes | No | Proprietary |
| Vicuna-33B | Yes | Yes | Non-commercial |
| Llama-2-70b-chat | Yes | Yes | Llama 2 Community |
| WizardLM-13b-v1.2 | Yes | Yes | Llama 2 Community |
| Vicuna-13B | Yes | Yes | Llama 2 Community |
| MPT-30B-chat | Yes | Yes | CC-BY-NC-SA-4.0 |
| Guanaco-33B | Yes | Yes | Non-commercial |
| CodeLlama-34B-instruct | No | Yes | Llama 2 Community |
| PaLM-Chat-Bison-001 | Yes | No | Proprietary |
| Vicuna-7B | Yes | Yes | Llama 2 Community |
| Llama-2-13b-chat | Yes | Yes | Llama 2 Community |
| Llama-2-7b-chat | Yes | Yes | Llama 2 Community |
| Koala-13B | No | No | Non-commercial |
| ChatGLM2-6B | Yes | No | Apache-2.0 |
| GPT4All-13B-Snoozy | Yes | Yes | Non-commercial |
Log:
[sudo] password for ncl:
Start to clone the repo: Vicuna-7B
Finished
Start to clone the repo: Vicuna-33B
Finished
Start to clone the repo: Llama-2-70b-chat
error: Cmd('git') failed due to: exit code(128)
cmdline: git clone -v -- https://huggingface.co/meta-llama/Llama-2-70b-chat-hf Llama-2-70b-chat
stderr: 'Cloning into 'Llama-2-70b-chat'...
'
Start to clone the repo: WizardLM-13b-v1.2
Finished
Start to clone the repo: Vicuna-13B
Finished
Start to clone the repo: MPT-30B-chat
Finished
Start to clone the repo: Guanaco-33B
Finished
Start to clone the repo: CodeLlama-34B-instruct
Finished
Start to clone the repo: Llama-2-13b-chat
error: Cmd('git') failed due to: exit code(128)
cmdline: git clone -v -- https://huggingface.co/meta-llama/Llama-2-13b-chat-hf Llama-2-13b-chat
stderr: 'Cloning into 'Llama-2-13b-chat'...
fatal: could not read Username for 'https://huggingface.co': No such device or address
'
Start to clone the repo: Llama-2-7b-chat
error: Cmd('git') failed due to: exit code(128)
cmdline: git clone -v -- https://huggingface.co/meta-llama/Llama-2-7b-chat-hf Llama-2-7b-chat
stderr: 'Cloning into 'Llama-2-7b-chat'...
fatal: could not read Username for 'https://huggingface.co': No such device or address
'
Start to clone the repo: GPT4All-13B-Snoozy
Finished
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent