yuweihao / mm-vet Goto Github PK

Hi,
For all the results, do you perform zero-shot testing or fine-tune on your dataset on the respective models?

Thanks for the wonderful evaluation work!
But the Otter evaluation seems based on our early version (around May 2023), we have a stronger MPT7B version Otter since last month based on OpenFlamingoV2. Would you like to try to evaluate it instead of the LLama7B version.

Both OpenFlamingoV2 and Otter-MPT7B have much better performance than previous version. Please consider using them to rejustify the evaluation results.

We also have a video version of Otter, would you like to add it to further VideoLLM evaluation?

Otter-Image: https://huggingface.co/luodian/OTTER-Image-MPT7B
Otter-Video: https://huggingface.co/luodian/OTTER-Video-LLaMA7B-DenseCaption

Would you mind to share your GPT-4V inference and evaluation script? Really appreciate your help.
I have tried by myself, and only achieve score 61.8, far from 67.7.

        rec   ocr  know   gen  spat  math  total  std    runs
gpt4v  54.8  74.1  39.6  42.9  74.7  75.2   61.8  0.0  [61.8]
       know_gen_rec   rec  spat_ocr  math_spat_ocr  spat_rec   ocr  math_ocr  know_rec  know_ocr_gen_rec  ...  spat_ocr_rec  ocr_rec  know_spat_ocr  spat_know_rec  spat_ocr_gen  math_spat_ocr_rec  total  std    runs
gpt4v          33.9  81.1      79.6           92.1      66.7  73.3      59.0      27.8              71.2  ...          42.9    100.0           66.7          100.0          85.0                0.0   61.8  0.0  [61.8]

How do you evaluate BARD on your dataset? Do you manually input the questions, or do you have a API you access? It would be much appreciated if you could share your BARD inference scripts, thanks!