Giter Club home page Giter Club logo

medical-image-captioning-on-chest-x-rays's Introduction

Medical-Image-Captioning-on-Chest-X-rays

Medical imaging is the process of creating visual representations of the interior of a body for clinical analysis as well as visual representation of the function of some organs or tissues. They are widely used in hospitals and clinics to determine fractures and diseases. The medical images are read and interpreted by specialized medical professionals and their findings regarding each body of area examined are communicated via written Medical Reports. The process of writing medical reports usually takes around 5โ€“10 minutes per report. In a day the doctors have to write medical reports that number in 100s which can take a lot of their time. The objective of this case study is to build a deep learning model that automatically write the impression part of medical report of chest X-rays and alleviate some of the burden of the medical professional. Here I will be taking a publicly available dataset from Indiana University which consists of chest X-ray images and reports (in XML format) which contain information regarding the findings and impression of the X-ray. The goal is to predict the impressions of the medical report attached to the images.

This web application takes input X-Ray images and returns the predicted impression of the medical report.

My detailed approach can be viewed in this medium article.

Streamlit App

Hnet-image

Results

Sl No. Model BLEU-1 BLEU-2 BLEU-3 BLEU-4
1. Attention Model (greedy search) 0.306819 0.302596 0.339031 0.383689
2. Custom Final Model (greedy search) 0.214501 0.243265 0.303785 0.36675
3. Simple Encoder Decoder (greedy search) 0.317412 0.308454 0.333496 0.366244

Contents of the Code Files are given below :-

Code File Description
1_EDA_Medical_Report.ipynb Exploratory Data Analysis
2_Simple_encoder_decoder_Medical_Report.ipynb Simple Encoder Decoder Model
3_Attention_Model_Medical_Report.ipynb Attention Model
4_Custom_Final_Model.ipynb Model based on Q. Tang, F. Liu, T. Zhang, J. Jiang, Y. Zhang, Attention-guided Chained Context Aggregation for Semantic Segmentation (2020) paper
5_Final.ipynb Function 1 - takes input images, returns predicted caption,Function 2 - takes input images returns BLEU scores (This file contains full data pipeline)
create_model.py Contains the code for creating the model
final.py Contains the code for web application
chexnet_weights Contains the weights for the ChexNet model
Encode_Decoder_global_attention.h5 Contains the weights for the Attention Model (best model)
tokenizer.pkl Contains the tokenizer

Experimentation was done in Jupyter Notebook and the code was then converted to .py files for deployment. The code for the web application is in final.py file. The web application is deployed using Streamlit.

medical-image-captioning-on-chest-x-rays's People

Contributors

agno-nymous avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar

medical-image-captioning-on-chest-x-rays's Issues

A little help

Dear Ashish,

Thanks very much for the Blog on image captioning!

I was trying to reproduce your work and wanted to know if you had a script to turn the XML files into the pandas dataframe because it's not included in the Github page. Or did you do it all on the regex site?

Best,
Oz

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.