VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding

Xiang Li, Jian Ding, Mohamed Elhoseiny

RSGPT: A Remote Sensing Vision Language Model and Benchmark

Yuan Hu, Jianlong Yuan, Congcong Wen, Xiaonan Lu, Xiang Li☨

VRSBench is a Versatile Vision-Language Benchmark for Remote Sensing Image Understanding. It consists of 29,614 remote sensing images with detailed captions, 52,472 object refers, and 3123,221 visual question-answer pairs. It facilitates the training and evaluation of vision-language models across a broad spectrum of remote sensing image understanding tasks.

• Release code and models of baseline models.
• [2024.06.19] We release the instructions and code for calling GPT-4V to get initial annotations.
• [2024.06.19] We release the VRSBench, A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding. VRSBench contains 29,614 images, with 29,614 human-verified detailed captions, 52,472 object references, and 123,221 question-answer pairs. check VRSBench Project Page.

The dataset can be downloaded from link and used via the Hugging Face datasets library. To load the dataset, you can use the following code snippet:

from datasets import load_dataset
fw = load_dataset("xiang709/VRSBench", streaming=True)

To construct our VRSBench dataset, we employed multiple data engineering steps, including attribute extraction, prompting engineering, GPT-4 inference, and human verification.

• Attribute Extraction: we extract image information, including the source and resolution, as well as object information—such as the object category, bounding box, color, position (absolute and relative), and size (absolute and relative)—from existing object detection datasets. Please check extract_patch_json.py for attribute extraction details.
• Prompting Engineering: We carefully design instructions to prompt GPT-4V to create detailed image captions, object referring, and question-answer pairs. Please check instruction.txt for detialed instruction.
• GPT-4 inference: Given input prompts, we call OpenAI API to automatically generate image captions, object referring, and question-answer pairs based on the prompts. Use the extract_patch_json.py to get initial annotations for image captioning, visual grounding, and VQA tasks using GPT-4V.
• Human verification: To improve the quality of the dataset, we engage human annotators to validate each annotation generated by GPT-4V.

For the above three tasks, we benchmark state-of-the-art models, including LLaVA-1.5, MiniGPT-v2, Mini-Gemini, and GeoChat, to demonstrate the potential of LVMs for remote sensing image understanding. To ensure a fair comparison, we reload the models that are initially trained on large-scale image-text alignment datasets, and then finetune each method using the training set of our RSVBench dataset. For each comparing method, we finetune the model on the training set of our RSVBench dataset for 5 epochs. Following GeoChat, we use LoRA finetuning to finetune all comparing methods, with a rank of 64.

Use the prepare_geochat_eval_all.ipynb to prepare the VRSBench evaluation file for image captioning, visual grounding, and VQA tasks.

Caption: Detailed image caption performance on the VRSBench dataset. Avg_L denotes the average word length of generated captions.

Caption: Visual grounding performance on the VRSBench dataset.

Caption: Visual question answering performance on the {\papernameAbbrev} dataset. Rows in bold indicate newly added results.

The dataset is released under the CC-BY-4.0, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

• RSGPT. The first GPT-based Large Vision-Language Model in remote sensing. RSGPT
• Survey. A comprehensive survey about vision-language models in remote sensing. RSVLM.
• MiniGPT-v2. MiniGPT-v2

@article{li2024vrsbench,
  title={VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding},
  author={Xiang Li, Jian Ding, and Mohamed Elhoseiny},
  journal={arXiv:2406.12384},
  year={2024}
}

Our VRSBench dataset is built based on DOTA-v2 and DIOR datasets.

We are thankful to LLaVA-1.5, MiniGPT-v2, Mini-Gemini, and GeoChat for releasing their models and code as open-source contributions.