Giter Club home page Giter Club logo

vilmedic's Introduction

ViLMedic: a framework for research at the intersection of vision and language in medical AI

ViLMedic has a dedicated website at: https://vilmedic.app/



Documentation Status MIT License


New!🔥 Code for Improving the Factual Correctness of Radiology Report Generation with Semantic Rewards

New!🔥 Checkout our live radiology report generation 📝 space on HuggingFace🤗

@inproceedings{delbrouck-etal-2022-vilmedic,
    title = "{V}i{LM}edic: a framework for research at the intersection of vision and language in medical {AI}",
    author = "Delbrouck, Jean-benoit  and
      Saab, Khaled  and
      Varma, Maya  and
      Eyuboglu, Sabri  and
      Chambon, Pierre  and
      Dunnmon, Jared  and
      Zambrano, Juan  and
      Chaudhari, Akshay  and
      Langlotz, Curtis",
    booktitle = "Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations",
    month = may,
    year = "2022",
    address = "Dublin, Ireland",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.acl-demo.3",
    pages = "23--34",
}

Quickstart

More at: https://vilmedic.app/

Installation

conda create --name vilmedic python==3.8 -y
git clone https://github.com/jbdel/vilmedic
python setup.py develop

Documentation

Learn more about ViLMedic here.

Model Zoo

ViLMedic hosts a zoo of pretrained models.

from vilmedic import AutoModel
model, processor = AutoModel.from_pretrained("selfsup/convirt-mimic")
batch = processor.inference(seq=["no acute cardiopulmonary process"],
                            image=["my_chest_xray.jpg"])

out = model(**batch)
print(out.keys())
# dict_keys(['loss', 'loss_l', 'loss_v', 'linguistic', 'visual'])
Name dataset Report preprocessing
Radiology report generation
rrg/biomed-roberta-baseline-mimic mimic-cxr r2gen
rrg/biomed-roberta-baseline-indiana indiana r2gen
rrg/baseline-padchest padchest -
Radiology report summarization
rrs/biomed-roberta-baseline-mimic mimic-cxr rouge
rrs/biomed-roberta-baseline-indiana indiana r2gen
Self-supervision
selfsup/convirt-mimic mimic-cxr r2gen
selfsup/convirt-mimic-balanced mimic-cxr r2gen
selfsup/convirt-padchest-16 padchest gloria
selfsup/convirt-padchest-32 padchest gloria
selfsup/convirt-indiana-16 indiana r2gen
selfsup/convirt-indiana-32 indiana r2gen
selfsup/convirt-indiana-64 indiana r2gen
selfsup/gloria-chexpert CheXpert gloria
selfsup/gloria-mimic-48 mimic-cxr r2gen
selfsup/simclr-mimic-16 mimic-cxr
selfsup/simclr-mimic-32 mimic-cxr
selfsup/simclr-mimic-64 mimic-cxr
selfsup/vae-mimic mimic-cxr
selfsup/vae-indiana indiana
selfsup/vae-padchest padchest
Medical VQA
mvqa/mvqa-imageclef ImageCLEF-VQAMed

Implemented solutions

ViLMedic replicates solutions from the multimodal medical literature.

Solutions
Medical Visual Question Answering
SYSU-HCP at VQA-Med 2021
Radiology report generation
Generating Radiology Reports via Memory-driven Transformer
Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports
Improving Factual Completeness and Consistency of Image-to-text Radiology Report Generation
Radiology report summarization
Multimodal Radiology Report Summarization
Multimodal self-supervised Learning
Contrastive Learning of Medical Visual Representations from Paired Images and Text
DALLE: Zero-Shot Text-to-Image Generation
CLIP: Learning Transferable Visual Models From Natural Language Supervision
SimCLR: A Simple Framework for Contrastive Learning of Visual Representations
GLoRIA: A Multimodal Global-Local Representation Learning Framework for Label-efficient Medical Image Recognition

Citation

If you use ViLMedic in your work or use any models published in ViLMedic, please cite:

License

ViLMedic is MIT-licensed. The license applies to the pre-trained models as well.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.