A visual question answering based model helping in segregation of question answers based on different medical images as input.
Pytorch, Transformers, Natural Language Processing, Computer Vision
: Used insights from BiomedCLIPs, Cross-Modal Attention-based model, SAN-stacked attention network for fine-tuning of the provided dataset ( VQA-RAD dataset ) in various pre-trained models.
: Trained a model from scratch using PubMedBERT (for text embeddings ) and Vision Transformers (for Image Embeddings ) and used a multi-head cross-attention layer on top of image and text embeddings.
: The proposed model was able to achieve an accuracy of 72-75% on the classification labels (the top 10 most frequent answers in the dataset).