| Maker | Models |
|---|---|
| OpenAI | CLIP(2021), OpenCLIP(2021), SynthCLIP(2024) |
| SigLIP(2023), PaLI-3(2023) | |
| Salesforce | BLIP(2022), BLIP-2(2023), LAVIS(2023) |
| FILIP(2021) | |
| moondream.ai | moondream(2024), moondream2(2024) |
| DeepMind | Flamingo(2022), OpenFlamingo(2023) |
| Molmo(2024) | |
| Cambrian-1(2024) | |
| HuggingFace | SmolVLM(2024) |
| HuggingFace | Idefics1(2023), Idefics(2024), Idefics(2024) |
| Microsoft | Florence(2021), Florence-2(2023) |
| Microsoft | Phi3-Vision(2024), Phi3.5-Vision(2024) |
| Microsoft | KOSMOS-2(2023) |
| PaliGemma(2024), PaliGemma-2(2024) | |
| Meta(Facebook) | MDETR(2021), FLAVA(2022) |
| Apple | 4M(2023), 4M-21(2024), MM1(2024), MM1.5(2024) |
| IBM | Granite-3.1-Vision(2025) |
| NVIDIA | VILA(2023) |
| LLaVA(2023), LLaVA-1.5(2023), LLaVA-NeXT(2024), LLaVA-OneVision(2024) | |
| PKU | Video-LLaVA(2024), LLaVA-o1(2024) -> LLaVA-CoT(2025) |
| MBZUAI | Robust-LLaVA(2025) |
| Meituan | MobileVLM(2023), MobileVLM-V2(2024), VisionLLaMA(2024) |
| X-VLM(2022) | |
| TinyGPT-V(2023) | |
| OpenBMB | MiniCPM, MiniCPM-V-2, MiniCPM-o(2025) |
| MiniGPT-4(2023), MiniGPT4-Video(2024) | |
| Mini-Omni(2024), Mini-Omni2(2024) | |
| Tencent | VITA-1.0(2024), VITA-1.5(2024) |
| THUDM | CogVLM(2023), CogVLM2(2024) |
| DeepSeek | DeepSeek-VL(2024), DeepSeek-VL2(2024) |
| Alibaba | Qwen-VL(2024), Qwen2-VL(2024), Qwen2.5-VL(2024) |
| Alibaba | Ovis1.0(2024), Ovis1.6(2024) |
| OpenGVLab | InternVL(2023), InternVL2(2024), InternVL2.5(2024) |
| OpenGVLab | VisionLLM(2023), VisionLLM-2(2024) |
| Maker | Models |
|---|---|
| OpenVLA(2024), OpenVLA-mini(2024) | |
| TinyVLA(2024) | |
| Pi0(2024) | |
| UCB | LLARVA(2025) |
- How to make its vision model from the base LLM?
- 2025/02/21 SigLIP 2: A better multilingual vision language encoder
- 2025/02/20 SmolVLM2: Bringing Video Understanding to Every Device
- 2025/01/29 Smol-ERVLM: Lightweight Vision-Language Model for Efficient AI
- 2025/01/23 SmolVLM Grows Smaller – Introducing the 250M & 500M Models!
- 2024/12/05 Welcome PaliGemma 2 – New vision language models by Google
- 2024/01/24 We just gave sight to smolagents
- 2024/01/23 SmolVLM Grows Smaller – Introducing the 250M & 500M Models!
- 2024/11/26 SmolVLM - small yet mighty Vision Language Model
- 2024/11/08 Guide to Vision-Language Models (VLMs)
- 2024/10/23 A comparative evaluation of open-source vision language models
- 2024/09/19 A Comprehensive Guide to Vision Language Models (VLMs)
- 2024/09/10 Finetune PaliGemma: Vision-Language Model on Free Colab
- 2024/09/06 Top Open-Source Vision Language Models (VLMs)
- 2024/05/26 Understanding CLIP — Implementing CLIP from scratch
- 2024/04/11 Vision Language Models Explained 🤗
- 2024/01/13 Exploring “Small” Vision-Language Models with TinyGPT-V
- 2023/10/17 Introduction to Visual-Language Model
- 2023/03/01 A Brief Introduction to Vision Language Models
- 2023/02/03 A Dive into Vision-Language Models 🤗
- OpenVLM Leaderboard
- Language Model : Qwen, DeepSeek, GLM, InternLM, BlueLM, Bailing, Gemma2, Llama, Aria, Yi, NeMo, Phi, Mixtral, Vicuna, MiniCPM, SmolLM, Falcon
- Vision Models: BLIP, CLIP, SigLIP, ViT, QwenViT, InternViT, SAM, EVA
- Primers • AI
- MM-LLMs - Recent Advances in MultiModal Large Language Models
- 2025 FoundationStereo: Zero-Shot Stereo Matching
- 2025 PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding
- 2024 Towards Vision-Language Geo-Foundation Model: A Survey
- 2024 LLM2CLIP: Powerful Language Model Unlocks Richer Visual Representation
- 2024 MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning
- 2024 Efficient Multimodal Large Language Models: A Survey
- 2024 An Introduction to Vision-Language Modeling 👍
- 2024 Foundation Models for Video Understanding: A Survey
- 2024 What matters when building vision-language models?
- 2024 MM-LLMs: Recent Advances in MultiModal Large Language Models
- https://github.com/uncbiag/Awesome-Foundation-Models - A curated list of foundation models for vision and language tasks 👍
- https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models - Latest Advances on Multimodal Large Language Models
- https://github.com/swordlidev/Efficient-Multimodal-LLMs-Survey - Efficient Multimodal Large Language Models: A Survey
- https://github.com/deepseek-ai/DeepSeek-VL - DeepSeek-VL: Towards Real-World Vision-Language Understanding
- https://github.com/deepseek-ai/DeepSeek-VL2 - DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding
- https://github.com/Stanford-ILIAD/openvla-mini - OpenVLA: An open-source vision-language-action model for robotic manipulation
- https://github.com/openvla/openvla - OpenVLA: An open-source vision-language-action model for robotic manipulation
- https://github.com/OpenGVLab/VisionLLM - VisionLLM Series
- https://github.com/vikhyat/moondream - tiny vision language model
- https://github.com/salesforce/LAVIS - A One-stop Library for Language-Vision Intelligence
- https://github.com/mlfoundations/open_flamingo - An open-source framework for training large multimodal models.
- https://github.com/salesforce/BLIP - PyTorch code for BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
- https://github.com/openai/CLIP - CLIP (Contrastive Language-Image Pretraining), Predict the most relevant text snippet given an image
- https://github.com/zengyan-97/X-VLM - X-VLM: Multi-Grained Vision Language Pre-Training (ICML 2022)
- https://github.com/cambrian-mllm/cambrian - Cambrian-1 is a family of multimodal LLMs with a vision-centric design
- https://github.com/google-research/big_vision - Official codebase used to develop Vision Transformer, SigLIP, MLP-Mixer, LiT and more
- https://github.com/THUDM/CogVLM - a state-of-the-art-level open visual language model | 多模态预训练模型
- https://github.com/THUDM/CogVLM2 - GPT4V-level open-source multi-modal model based on Llama3-8B
- https://github.com/QwenLM/Qwen2.5-VL - Qwen2.5-VL is the multimodal large language model series developed by Qwen team, Alibaba Cloud
- https://github.com/roboflow/notebooks - a comprehensive collection of tutorials on state-of-the-art computer vision models and techniques. Explore everything from foundational architectures like ResNet to cutting-edge models like YOLO11, RT-DETR, SAM 2, Florence-2, PaliGemma 2, and Qwen2.5VL
- https://github.com/gpt-omni/mini-omni - open-source multimodal large language model that can hear, talk while thinking. Featuring real-time end-to-end speech input and streaming audio output conversational capabilities
- https://github.com/Deep-Agent/R1-V - Witness the aha moment of VLM with less than $3
- https://github.com/andimarafioti/open-r1-multimodal - A fork to add multimodal model training to open-r1
- https://github.com/roboflow/maestro - streamline the fine-tuning process for multimodal models: PaliGemma, Florence-2, and Qwen2-VL
- https://github.com/merveenoyan/smol-vision - Recipes for shrinking, optimizing, customizing cutting edge vision models
- https://github.com/google-research/big_vision - Official codebase used to develop Vision Transformer, SigLIP, MLP-Mixer, LiT and more
- https://github.com/Abonia1/VLM-Architecture - VLM Architecture Guide
- https://github.com/shukur-alom/AgenticVision - Agentic Vision utilizing Microsoft's Florence-2
- https://github.com/lucasjinreal/Namo-R1 - A CPU Realtime VLM in 500M. Surpassed Moondream2 and SmolVLM. Training from scratch with ease.
