Giter Club home page Giter Club logo

awesome-foundation-and-multimodal-models's Introduction

awesome foundation and multimodal models

πŸ‘οΈ + πŸ’¬ + 🎧 = πŸ€–

foundation model - a pre-trained machine learning model that serves as a base for a wide range of downstream tasks. It captures general knowledge from a large dataset and can be fine-tuned to perform specific tasks more effectively.

multimodal model - a model that can process multiple modalities (e.g. text, image, video, audio, etc.) at the same time.

πŸ€– models

YOLO-World: Real-Time Open-Vocabulary Object Detection

arXiv GitHub Gradio Colab

Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, Ying Shan

  • Date: 2024-01-30
  • Modalities: πŸ‘οΈ + πŸ’¬
  • Tasks: Zero-Shot Object Detection

Depth Anything

arXiv GitHub Gradio Hugging Face Models Colab

Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, Hengshuang Zhao

  • Date: 2024-01-19
  • Modalities: πŸ‘οΈ
  • Tasks: Depth Estimation

EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything

arXiv GitHub Gradio Hugging Face Models

Yunyang Xiong, Bala Varadarajan, Lemeng Wu, Xiaoyu Xiang, Fanyi Xiao, Chenchen Zhu, Xiaoliang Dai, Dilin Wang, Fei Sun, Forrest Iandola, Raghuraman Krishnamoorthi, Vikas Chandra

  • Date: 2023-12-01
  • Modalities: πŸ‘οΈ
  • Tasks: Zero-Shot Object Segmentation

Qwen-VL-Plus / Max

arXiv GitHub Gradio Hugging Face Models

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, Jingren Zhou

  • Date: 2023-11-28
  • Modalities: πŸ‘οΈ + πŸ’¬
  • Tasks: Image Captioning, VQA, Zero-Shot Object Detection

CogVLM: Visual Expert for Pretrained Language Models

arXiv GitHub Gradio Hugging Face Models

Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, Jie Tang

  • Date: 2023-11-06
  • Modalities: πŸ‘οΈ + πŸ’¬
  • Tasks: Image Captioning, VQA

Fuyu-8B: A Multimodal Architecture for AI Agents

Gradio Hugging Face Models

Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, Sağnak Taşırlar

  • Date: 2023-10-17
  • Modalities: πŸ‘οΈ + πŸ’¬
  • Tasks: Image Classification, Image Captioning, VQA, Find Text in Image

Ferret: Refer and Ground Anything Anywhere at Any Granularity

arXiv GitHub

Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, Yinfei Yang

  • Date: 2023-10-11
  • Modalities: πŸ‘οΈ + πŸ’¬
  • Tasks: Image Captioning, VQA, Phrase Grounding, Object Detection

MetaCLIP: Demystifying CLIP Data

arXiv GitHub Gradio Hugging Face Models Colab

Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, Christoph Feichtenhofer

  • Date: 2023-09-28
  • Modalities: πŸ‘οΈ + πŸ’¬
  • Tasks: Zero-Shot Classification

Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond

arXiv GitHub Gradio Hugging Face Models

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, Jingren Zhou

  • Date: 2023-09-24
  • Modalities: πŸ‘οΈ + πŸ’¬
  • Tasks: Image Captioning, VQA

SigLIP: Sigmoid Loss for Language Image Pre-Training

arXiv GitHub Gradio Hugging Face Models Colab

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, Lucas Beyer

  • Date: 2023-08-27
  • Modalities: πŸ‘οΈπŸ’¬
  • Tasks: Zero-Shot Image Classification

Nougat: Neural Optical Understanding for Academic Documents

arXiv GitHub Gradio Hugging Face Models Colab

Lukas Blecher, Guillem Cucurull, Thomas Scialom, Robert Stojnic

  • Date: 2023-08-25
  • Modalities: πŸ‘οΈπŸ’¬
  • Tasks: Visual Question Answering

AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining

arXiv GitHub Gradio Hugging Face Models

Haohe Liu, Qiao Tian, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Yuping Wang, Wenwu Wang, Yuxuan Wang, Mark D. Plumbley

  • Date: 2023-08-10
  • Modalities: πŸ’¬οΈ + 🎧
  • Tasks: Text-to-Audio, Text-to-Speech

OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models

arXiv GitHub Gradio Hugging Face Models

Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, Ludwig Schmidt

  • Date: 2023-08-02
  • Modalities: πŸ‘οΈ + πŸ’¬
  • Tasks: Image Classification, Image Captioning, VQA

Kosmos-2: Grounding Multimodal Large Language Models to the World

arXiv GitHub Gradio Hugging Face Models

Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, Furu Wei

  • Date: 2023-07-26
  • Modalities: πŸ‘οΈ + πŸ’¬
  • Tasks: Image Captioning, VQA, Phrase Grounding

OWLv2: Scaling Open-Vocabulary Object Detection

arXiv GitHub Gradio Hugging Face Models Colab

Matthias Minderer, Alexey Gritsenko, Neil Houlsby

  • Date: 2023-06-17
  • Modalities: πŸ‘οΈ
  • Tasks: Zero-Shot Object Detection

ImageBind: One Embedding Space To Bind Them All

arXiv GitHub Gradio

Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, Ishan Misra

  • Date: 2023-05-09
  • Modalities: πŸ‘οΈ + πŸ’¬ + 🎧
  • Tasks:

LLaVA: Large Language and Vision Assistant

arXiv GitHub Gradio Hugging Face Models

Haotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae Lee

  • Date: 2023-04-17
  • Modalities: πŸ‘οΈ + πŸ’¬
  • Tasks: Vision Language Modeling

Segment Anything

arXiv GitHub Gradio Hugging Face Models Colab

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr DollΓ‘r, Ross Girshick

  • Date: 2023-04-05
  • Modalities: πŸ‘οΈ
  • Tasks: Zero-Shot Object Segmentation

Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection

arXiv GitHub Gradio Hugging Face Models Colab

Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, Lei Zhang

  • Date: 2023-03-09
  • Modalities: πŸ‘οΈ + πŸ’¬
  • Tasks: Phrase Grounding, Zero-Shot Object Detection

BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

arXiv GitHub Gradio Hugging Face Models Colab

Junnan Li, Dongxu Li, Silvio Savarese, Steven Hoi

  • Date: 2023-01-30
  • Modalities: πŸ‘οΈ + πŸ’¬
  • Tasks: Image Captioning, Visual Question Answering

Whisper: Robust Speech Recognition via Large-Scale Weak Supervision

arXiv GitHub Gradio Hugging Face Models Colab

Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, Ilya Sutskever

  • Date: 2022-12-06
  • Modalities: πŸ’¬οΈ + 🎧
  • Tasks: Speech-to-Text

OWL-ViT: Simple Open-Vocabulary Object Detection with Vision Transformers

arXiv GitHub Gradio Hugging Face Models

Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, Xiao Wang, Xiaohua Zhai, Thomas Kipf, Neil Houlsby

  • Date: 2022-05-12
  • Modalities: πŸ‘οΈ + πŸ’¬
  • Tasks: Zero-Shot Object Detection

CLIP: Learning Transferable Visual Models From Natural Language Supervision

arXiv GitHub Hugging Face Models Colab

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever

  • Date: 2021-02-26
  • Modalities: πŸ‘οΈ + πŸ’¬
  • Tasks: Zero-Shot Classification

🦸 contribution

We would love your help in making this repository even better! If you know of an amazing paper that isn't listed here, or if you have any suggestions for improvement, feel free to open an issue or submit a pull request.

awesome-foundation-and-multimodal-models's People

Contributors

fnauman avatar merveenoyan avatar skalskip avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.