optimalscale / detgpt Goto Github PK

can you please provide details on os and gpu to be used ?

When I try the demo_detgpt.py after environment setup, I got below error:

(detgpt) admin@k8s:~/DetGPT$ python demo_detgpt.py --cfg-path configs/detgpt_tasktune_7b_coco.yaml
Initializing Chat
final text_encoder_type: bert-base-uncased
Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.seq_relationship.bias', 'cls.predictions.decoder.weight', 'cls.predictions.bias', 'cls.seq_relationship.weight', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias']

• This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).

• This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
  Model loaded from /home/itadmin/.cache/huggingface/hub/models--ShilongLiu--GroundingDINO/snapshots/a94c9b567a2a374598f05c584e96798a170c56fb/groundingdino_swinb_cogcoor.pth
  => _IncompatibleKeys(missing_keys=[], unexpected_keys=['label_enc.weight'])
  /home/itadmin/anaconda3/envs/detgpt2/lib/python3.9/site-packages/torch/cuda/init.py:104: UserWarning:
  NVIDIA A100-PCIE-40GB with CUDA capability sm_80 is not compatible with the current PyTorch installation.
  The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
  If you want to use the NVIDIA A100-PCIE-40GB GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))

loaded detector

Loading VIT
Loading VIT Done
Loading Q-Former
Loading Q-Former Done
Loading LLAMA
Traceback (most recent call last):
File "/home/itadmin/hans/2d/DetGPT/demo_detgpt.py", line 198, in
model_llm = model_cls.from_config(model_config).to(cuda_llm)
File "/home/itadmin/hans/2d/DetGPT/detgpt/models/detgpt.py", line 278, in from_config
model = cls(
File "/home/itadmin/hans/2d/DetGPT/detgpt/models/detgpt.py", line 95, in init
self.llama_tokenizer = LlamaTokenizer.from_pretrained(llama_model, use_fast=False)
File "/home/itadmin/anaconda3/envs/detgpt2/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 1795, in from_pretrained
raise EnvironmentError(
OSError: Can't load tokenizer for './output_models/vicuna-7b'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure './output_models/vicuna-7b' is the correct path to a directory containing all relevant files for a LlamaTokenizer tokenizer.

你们这个项目非常棒，请问什么时候会发布训练或微调代码呢，谢谢！

hello,

Is there a way to load and serve multiple models on a single GPU
my computer:
gpu: RTX 3060
cpu: i7-12700
ram: 16gb

pip install -e .

Collecting supervision (from detgpt==0.0.1)
Using cached supervision-0.7.0-py3-none-any.whl (38 kB)
INFO: pip is looking at multiple versions of detgpt to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement decord==0.6.0 (from detgpt) (from versions: none)
ERROR: No matching distribution found for decord==0.6.0

uname -a
Linux 309bbf56dcab 5.15.49-linuxkit #1 SMP PREEMPT Tue Sep 13 07:51:32 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

All demos are down.

Thank you for your outstanding work.
I try the demo_detgpt.py on local machine, but it can't detect anything.
I've tried a couple of examples that don't output anything，here's why？

Hi Team ,
on link is working for demo even the backup ones please check and update it.

In the train log, loss is always nan.

Hi Developers, Thank You for this wonderful job !!!

Trying to run DetGPT locally, following readme file but i feel steps are not clear please fix this.

1. Installation
   git clone https://github.com/OptimalScale/DetGPT.git
   cd DetGPT
   conda create -n detgpt python=3.9 -y
   conda activate detgpt
   pip install -e .

Step 1 : Completed

2. Install GroundingDino
   python -m pip install -e GroundingDINO
3. Download the pretrained checkpoint
   cd output_models
   bash download.sh all
   cd -

content inside dir after running -> bash download.sh all
output_models/
coco_task_annotation.json
download.sh
pretrained_minigpt4_7b.pth
pretrained_minigpt4_13b.pth
task_tuned.pth

Step 2 : Completed

having problem here
-> Merge the robin lora model with the original llama model and save the merged model to output_models/robin-7b, w[here] (https://github.com/OptimalScale/DetGPT/blob/main/detgpt/configs/models/detgpt_robin_7b.yaml#L16) the corresponding model path is specified in this config file here.

To obtain the original llama model, one may refer to this doc. To merge a lora model with a base model, one may refer to PEFT or use the merge script provided by LMFlow.

Questios:

1. How to merge the robin lora model with the original llama model?
2. this output_models/robin-7b doesn't exist do i need to create this?
3. merge a lora model with a base model, is llama model a base model? what is base model here?
4. how to Replace 'path/to/pretrained_linear_weights' in the config file to the real path.

trying to use this script but not sure how to get this {huggingface-model-name-or-path-to-base-model} , {path-to-lora-model
and {path-to-merged-model}

python examples/merge_lora.py
--model_name_or_path {huggingface-model-name-or-path-to-base-model}
--lora_model_path {path-to-lora-model}
--output_model_path {path-to-merged-model}

what path to specify here -> ckpt: 'path/to/pretrained_linear_weights'

Please help in running this locally with required steps.

Collecting supervision (from detgpt==0.0.1)
Using cached supervision-0.8.0-py3-none-any.whl (42 kB)
INFO: pip is looking at multiple versions of detgpt to determine which version is compatible with other requirements. This could take a while.
ERROR: Ignored the following versions that require a different python version: 1.6.2 Requires-Python >=3.7,<3.10; 1.6.3 Requires-Python >=3.7,<3.10; 1.7.0 Requires-Python >=3.7,<3.10; 1.7.0rc1 Requires-Python >=3.7,<3.10; 1.7.0rc2 Requires-Python >=3.7,<3.10; 1.7.1 Requires-Python >=3.7,<3.10
ERROR: Could not find a version that satisfies the requirement decord==0.6.0 (from detgpt) (from versions: none)
ERROR: No matching distribution found for decord==0.6.0

It seems the model files cannot be downloaded.

(detgpt) itadmin@k8s172:~/DetGPT/output_models$ bash download.sh all
downloading robin-7b
--2023-05-12 10:22:51-- http://lmflow.org:5000/robin-7b.tar.gz
Connecting to 153.88.253.150:8080... connected.
Proxy request sent, awaiting response... 503 Service Unavailable
2023-05-12 10:23:51 ERROR 503: Service Unavailable.

tar (child): robin-7b.tar.gz: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
rm: cannot remove 'robin-7b.tar.gz': No such file or directory

hi~ the three demo you provided all not working！！！！！ please update！！！plz!!!!!!!!!!!!!!!!!!!!!!!!!!

Object detection backend seems to be class-agnostic. Could you please provide details on that?

so when will you release your training code, thanks a lot.

Trying local version. Ran into CUDA memory problem even after reducing batch. What else should i do?

Hi developers, thanks for your wonderful job and I have got good performance when I input my own test images on your web demo. So I tried to run your model demo locally. But I got confused when I followed your readme step by step.
First, there is only 2 files in http://lmflow.org:5000/detgpt/

But more files in the upper level directory. Are there some of them should be put in /detgpt or useful for DetGPT?

Second, In readme, Step 2 suggests 'use Robin from LMFlow team', merge Robin with origin Llama model. But in 'Deploy Demo Locally' part, it seems that the demo is using 'linear weights based on Vicuna-13B-v1.1'. So Vicuna-13B is used as the LLM part in demo?
And I can not find the pretrained linear weights following the link. It seems only complete vicuna models on your hugging face account.

All above is my questions. I am confused. Looking forward to your responses! Thanks

The README mentions linear weights based on Vicuna-13B-v1.1, but I couldn't find them in the repo. Can you provide some guidance on where to find these weights or when they will be available if they're not yet released?

CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling cublasLtMatmul from torch.nn.functional.linear()

Hello! Thank you for your great job. I wonder where I can download the dataset you constructed. I cannot find it anywhere.

final text_encoder_type: bert-base-uncased
Traceback (most recent call last):
File "/app/DetGPT/demo_detgpt.py", line 189, in
detector = load_model_hf(config_file, ckpt_repo_id, ckpt_filenmae)
File "/app/DetGPT/demo_detgpt.py", line 84, in load_model_hf
model = build_model(model_args)
File "/app/DetGPT/GroundingDINO/groundingdino/models/init.py", line 17, in build_model
model = build_func(args)
File "/app/DetGPT/GroundingDINO/groundingdino/models/GroundingDINO/groundingdino.py", line 372, in build_groundingdino
model = GroundingDINO(
File "/app/DetGPT/GroundingDINO/groundingdino/models/GroundingDINO/groundingdino.py", line 108, in init
self.bert = get_tokenlizer.get_pretrained_language_model(text_encoder_type)
File "/app/DetGPT/GroundingDINO/groundingdino/util/get_tokenlizer.py", line 23, in get_pretrained_language_model
return BertModel.from_pretrained(text_encoder_type)
File "/app/anaconda3/envs/detgpt/lib/python3.9/site-packages/transformers/modeling_utils.py", line 2560, in from_pretrained
state_dict = load_state_dict(resolved_archive_file)
File "/app/anaconda3/envs/detgpt/lib/python3.9/site-packages/transformers/modeling_utils.py", line 429, in load_state_dict
with safe_open(checkpoint_file, framework="pt") as f:
safetensors_rust.SafetensorError: Error while deserializing header: MetadataIncompleteBuffer

The installation guide is really not clear. Should be very basic:
(1) weight files to a path that is also synced to config files
(2) a script that we can run from terminal to debug, and not straight from command line terminal

OSError: Can't load tokenizer for './output_models/vicuna-13b'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure './output_models/vicuna-13b' is the correct path to a directory containing all relevant files for a LlamaTokenizer tokenizer.

Please switch the demo to replicate.com instead, which allows for API usage.

Hello Developers, tried downloading the models but,

downloading robin-7b
--2023-06-05 12:27:24-- http://lmflow.org:5000/robin-7b.tar.gz
Resolving lmflow.org (lmflow.org)... 3.84.211.127
Connecting to lmflow.org (lmflow.org)|3.84.211.127|:5000... connected.
HTTP request sent, awaiting response... 404 File not found
2023-06-05 12:27:25 ERROR 404: File not found.

tar: Error opening archive: Failed to open 'robin-7b.tar.gz'
rm: robin-7b.tar.gz: No such file or directory
downloading robin-13b
--2023-06-05 12:27:25-- http://lmflow.org:5000/robin-13b.tar.gz
Resolving lmflow.org (lmflow.org)... 3.84.211.127
Connecting to lmflow.org (lmflow.org)|3.84.211.127|:5000... connected.
HTTP request sent, awaiting response... 404 File not found
2023-06-05 12:27:25 ERROR 404: File not found.

tar: Error opening archive: Failed to open 'robin-13b.tar.gz'
rm: robin-13b.tar.gz: No such file or directory

how to download these models ?? help!

all demo link reported "No interface is running right now" error.
Please update the demo link, thanks

Firstly, thanks LMFlow team for your amazing work in LLM and Multi-Modal !!!

When I run the following script:
CUDA_VISIBLE_DEVICES=0,1 python demo_detgpt.py --cfg-path configs/detgpt_tasktune_13b_coco.yaml
All the models are loaded successfully:

But when I select the first demo, and run the model, the output is "<object_names>":

I doubt the following layer( in DetGPT/detgpt/models/detgpt.py) is only initilize, but not loaded pretrained weights:

error: subprocess-exited-with-error

× python setup.py develop did not run successfully.
│ exit code: 1
╰─> [31 lines of output]
/home/sdb/anaconda3/envs/groundingDino/lib/python3.10/site-packages/setuptools/_distutils/dist.py:265: UserWarning: Unknown distribution option: 'requires_python'
warnings.warn(msg)
running develop
/home/sdb/anaconda3/envs/groundingDino/lib/python3.10/site-packages/setuptools/command/develop.py:40: EasyInstallDeprecationWarning: easy_install command is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` and ``easy_install``.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

i install the project, but failed. what should i do to solve it. thanks

I'm wondering can I skip the merging part of the set up if I just download the file from here?

-> Merge the robin lora model with the original llama model and save the merged model to output_models/robin-7b, w[here] (https://github.com/OptimalScale/DetGPT/blob/main/detgpt/configs/models/detgpt_robin_7b.yaml#L16) the corresponding model path is specified in this config file here.

To obtain the original llama model, one may refer to this doc. To merge a lora model with a base model, one may refer to PEFT or use the merge script provided by LMFlow.

For example. I ask: "how many apples are there in the image". Boxing all the apples in the image and count, give the correct answer.

When run demo_detgpt.py, an error is reported NameError: name '_C' is not defined.
In GroundingDINO/groundingdino/models/GroundingDINO/ms_deform_attn.py, "from groundingdino import _C" not work.