PLAMi (Pixel-Level Analysis in Medical Imaging) precisely aligns image and text data through Visual Instruction Tuning technology, significantly enhancing the accuracy of medical image analysis. This framework incorporates a specialized visual encoder and a pixel-level regional feature extractor, both tailored for medical imaging, to capture and deeply analyze subtle features within images. PLAMi not only improves diagnostic accuracy but also provides detailed visual and textual analysis when handling complex medical images, offering robust support for clinical diagnosis.

This overview is based on findings from the article "Enhancing Pixel-Level Analysis in Medical Imaging through Visual Instruction Tuning: Introducing PLAMi."

Clone the repository and set up the environment with all necessary packages using these commands:

git clone https://github.com/MaochengBai/PLAMi
cd PLAMi
conda create -n plami python=3.10 -y
conda activate plami
pip install --upgrade pip
pip install -e .
pip install -e ".[train]"
pip install flash-attn --no-build-isolation

To run the demo, navigate to the demo directory and execute app.py. Before starting, ensure you have set the model_path for the model, data_folder for the image data, json_file for the image instructions, and description_type to specify the description format in app.py. Make sure all parameters are correctly configured to match your setup.

cd demo
python app.py 

Utilize the LLaVA training approach to train text-image alignment projectors and refine the mm_projector. This stage does not introduce a feature extractor.

• Set model_name_or_path to the path of Mistral-7B-Instruct-v0.2 in stage2.sh.
• Set pretrain_mm_mlp_adapter to the path of mm_projector in stage2.sh.
• Run sh scripts/stage2.sh.

• Set model_name_or_path to the path of stage2 checkpoint in stage2.sh.
• Set vision_tower to the path of BioMedCLIP in stage2.sh.
• Run sh scripts/stage3.sh.

The datasets generated and/or analyzed during the current study are available from the author on reasonable request. The data will be made publicly available after the publication of the paper. If you are interested in accessing the data before then, please contact the author via email at [email protected].