awesome foundation and multimodal models

👁️ + 💬 + 🎧 = 🤖

foundation model - a pre-trained machine learning model that serves as a base for a wide range of downstream tasks. It captures general knowledge from a large dataset and can be fine-tuned to perform specific tasks more effectively.

multimodal model - a model that can process multiple modalities (e.g. text, image, video, audio, etc.) at the same time.

🗞️ papers

AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining

Haohe Liu, Qiao Tian, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Yuping Wang, Wenwu Wang, Yuxuan Wang, Mark D. Plumbley

Date: 10-08-2023
Modalities: 💬️ + 🎧
Tasks: Text-to-Audio, Text-to-Speech

OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models

Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, Ludwig Schmidt

Date: 02-08-2023
Modalities: 👁️ + 💬
Tasks: Image Classification, Image Captioning, VQA

Kosmos-2: Grounding Multimodal Large Language Models to the World

Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, Furu Wei

Date: 26-07-2023
Modalities: 👁️ + 💬
Tasks: Image Captioning, VQA, Phrase Grounding

LLaVA: Large Language and Vision Assistant

Haotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae Lee

Date: 17-04-2023
Modalities: 👁️ + 💬
Tasks:

ImageBind: One Embedding Space To Bind Them All

Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, Ishan Misra

Date: 09-05-2023
Modalities: 👁️ + 💬 + 🎧
Tasks:

Segment Anything

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, Ross Girshick

Date: 05-04-2023
Modalities: 👁️
Tasks:

Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection

Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, Lei Zhang

Date: 09-03-2023
Modalities: 👁️ + 💬
Tasks:

BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

Junnan Li, Dongxu Li, Silvio Savarese, Steven Hoi

Date: 30-01-2023
Modalities: 👁️ + 💬
Tasks:

OWL-ST: Scaling Open-Vocabulary Object Detection

Matthias Minderer, Alexey Gritsenko, Neil Houlsby

Date: 16-01-2023
Modalities: 👁️ + 💬
Tasks:

Whisper: Robust Speech Recognition via Large-Scale Weak Supervision

Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, Ilya Sutskever

Date: 06-12-2022
Modalities: 💬️ + 🎧
Tasks:

OWL-ViT: Simple Open-Vocabulary Object Detection with Vision Transformers

Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, Xiao Wang, Xiaohua Zhai, Thomas Kipf, Neil Houlsby

Date: 12-05-2022
Modalities: 👁️ + 💬
Tasks:

CLIP: Learning Transferable Visual Models From Natural Language Supervision

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever

Date: 26-02-2021
Modalities: 👁️ + 💬
Tasks:

🦸 contribution

We would love your help in making this repository even better! If you know of an amazing paper that isn't listed here, or if you have any suggestions for improvement, feel free to open an issue or submit a pull request.

onuralpszr / awesome-foundation-and-multimodal-models Goto Github PK