Enhancing Pixel-Level Analysis in Medical Imaging through Visual Instruction Tuning: Introducing PLAMi
PLAMi (Pixel-Level Analysis in Medical Imaging) precisely aligns image and text data through Visual Instruction Tuning technology, significantly enhancing the accuracy of medical image analysis. This framework incorporates a specialized visual encoder and a pixel-level regional feature extractor, both tailored for medical imaging, to capture and deeply analyze subtle features within images. PLAMi not only improves diagnostic accuracy but also provides detailed visual and textual analysis when handling complex medical images, offering robust support for clinical diagnosis.
This overview is based on findings from the article "Enhancing Pixel-Level Analysis in Medical Imaging through Visual Instruction Tuning: Introducing PLAMi."
Clone the repository and set up the environment with all necessary packages using these commands:
git clone https://github.com/MaochengBai/PLAMi
cd PLAMi
conda create -n plami python=3.10 -y
conda activate plami
pip install --upgrade pip
pip install -e .
pip install -e ".[train]"
pip install flash-attn --no-build-isolation
To run the demo, navigate to the demo directory and execute app.py. Before starting, ensure you have set the model_path for the model, data_folder for the image data, json_file for the image instructions, and description_type to specify the description format in app.py. Make sure all parameters are correctly configured to match your setup.
cd demo
python app.py
Utilize the LLaVA training approach to train text-image alignment projectors and refine the mm_projector
. This stage does not introduce a feature extractor.
- Set
model_name_or_path
to the path ofMistral-7B-Instruct-v0.2
instage2.sh
. - Set
pretrain_mm_mlp_adapter
to the path ofmm_projector
instage2.sh
. - Run
sh scripts/stage2.sh
.
- Set
model_name_or_path
to the path ofstage2 checkpoint
instage2.sh
. - Set
vision_tower
to the path ofBioMedCLIP
instage2.sh
. - Run
sh scripts/stage3.sh
.
The datasets generated and/or analyzed during the current study are available from the author on reasonable request. The data will be made publicly available after the publication of the paper. If you are interested in accessing the data before then, please contact the author via email at [email protected].