360cvgroup / seechat Goto Github PK

when I run 'sh demo_stage2.sh', I meet the error below:
ValueError: You passed inputs_embeds to .generate(), but the model class ChatGLMForConditionalGeneration doesn't have its forwarding implemented. See the GPT2 implementation for an
example (huggingface/transformers#21405), and feel free to open a PR with it!
Could you please give me some insight.

where is the model FileNotFoundError: [Errno 2] No such file or directory: '../models/chatglm-6b/checkpoint_100.pth'?

vit 和 qformer 都是来自 blip2，vicuna和glm也是不训练的，那就是只需要训练 qformer 到 llm 的全连接层呗？这样效果会好么？

参与训练：

self.llama_proj = nn.Linear(
            self.Qformer.config.hidden_size, self.llama_model.config.hidden_size
        )

不参与训练：

vit_model="eva_clip_g",

q_former_model="https://storage.googleapis.com/sfr-vision-language-research/LAVIS/models/BLIP2/blip2_pretrained_flant5xxl.pth",

for name, param in self.llama_model.named_parameters():
            param.requires_grad = False

Loading VIT
Loading VIT Done
Loading Q-Former
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/xxx/anaconda3/envs/seechat/lib/python3.8/site-packages/transformers/utils/hub.py:409 │
│ in cached_file │
│ │
│ 406 │ user_agent = http_user_agent(user_agent) │
│ 407 │ try: │
│ 408 │ │ # Load from URL or cache if already cached │
│ ❱ 409 │ │ resolved_file = hf_hub_download( │
│ 410 │ │ │ path_or_repo_id, │
│ 411 │ │ │ filename, │
│ 412 │ │ │ subfolder=None if len(subfolder) == 0 else subfolder, │
│ │
│ /home/xxx/anaconda3/envs/seechat/lib/python3.8/site-packages/huggingface_hub/utils/_valida │
│ tors.py:120 in _inner_fn │
│ │
│ 117 │ │ if check_use_auth_token: │
│ 118 │ │ │ kwargs = smoothly_deprecate_use_auth_token(fn_name=fn.name, has_token=ha │
│ 119 │ │ │
│ ❱ 120 │ │ return fn(*args, **kwargs) │
│ 121 │ │
│ 122 │ return _inner_fn # type: ignore │
│ 123 │
│ │
│ /home/xxx/anaconda3/envs/seechat/lib/python3.8/site-packages/huggingface_hub/file_download │
│ .py:1259 in hf_hub_download │
│ │
│ 1256 │ │ │ │ " and downloads online, set 'local_files_only' to False." │
│ 1257 │ │ │ ) │
│ 1258 │ │ else: │
│ ❱ 1259 │ │ │ raise LocalEntryNotFoundError( │
│ 1260 │ │ │ │ "Connection error, and we cannot find the requested files in" │
│ 1261 │ │ │ │ " the disk cache. Please try again or make sure your Internet" │
│ 1262 │ │ │ │ " connection is on." │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
LocalEntryNotFoundError: Connection error, and we cannot find the requested files in the disk cache. Please try again or make sure your Internet connection is on.