LLaVA (Liu 2023年)（ラバ）
2023年Haotian Liu et al. (UW-Madison+MS Research)発表LLaVA・Industry-leading open-source vision-language model + Industry-leading visual instruction tuning + Industry-leading CLIP + LLaMA + Industry-leading first open VLM foundation。
概要
LLaVA は、2023年Haotian Liu et al. (UW-Madison+MS Research)発表LLaVA・Industry-leading open-source vision-language model 2023年 + Industry-leading first open VLM foundation position確立。LLaVA specifications = Industry-leading open-source vision-language model (Industry-leading LLaVA Liu 2023 + Industry-leading open-source VLM + Industry-leading LLaVA flagship) + Industry-leading visual instruction tuning + Industry-leading CLIP + LLaMA + Industry-leading first open VLM foundation。
主な特徴・仕組み
- Authors: Industry-leading Haotian Liu + Chunyuan Li + Qingyang Wu + Yong Jae Lee (UW-Madison + Microsoft Research + Columbia)
- Year: 2023年 (NeurIPS 2023 paper)
- Paper: Industry-leading "Visual Instruction Tuning" arXiv 2304.08485
- Method Type: Industry-leading open-source vision-language model
- Visual Instruction Tuning: Industry-leading visual instruction tuning signature
- CLIP + LLaMA: Industry-leading CLIP ViT-L/14 vision encoder + LLaMA / Vicuna language model
- GPT-4 Generated Data: Industry-leading GPT-4 generated visual instruction data
- LLaVA Models: Industry-leading LLaVA-7B + LLaVA-13B + LLaVA-1.5 + LLaVA-NeXT
- First Open VLM: Industry-leading first open-source VLM foundation
- Open Source: Industry-leading GitHub LLaVA open implementation
- Industry Impact: Industry-leading open VLM ecosystem foundation (LLaVA-1.5/NeXT/VILA/Bunny)
- Industry-Leading: Industry-leading first open-source VLM foundation Liu UW-Madison 2023
スペック比較表
| LLM VLM (2023-2024) | Year | Org | Method | Industry Position |
|---|---|---|---|---|
| LLaVA | 2023 | Liu UW-Madison+MS | CLIP+LLaMA + visual instruct | Industry-leading open VLM foundation |
| GPT-4V | 2023 | OpenAI | Proprietary VLM | Industry-leading OpenAI VLM |
| Gemini Vision | 2023 | Google DeepMind | Native multimodal | Industry-leading native multimodal |
| Claude 3 Vision | 2024 | Anthropic | Claude 3 multimodal | Industry-leading Claude VLM |
| Qwen2-VL | 2024 | Alibaba | Naive dynamic resolution | Industry-leading Qwen VLM |
具体例・対応製品
- LLaVA (2023年, Liu et al. UW-Madison+MS Research): Industry-leading first open-source VLM foundation
- Industry-leading visual instruction tuning signature: Industry-leading visual instruction tuning
- Industry-leading CLIP ViT-L/14 + LLaMA/Vicuna: Industry-leading CLIP+LLaMA
- Industry-leading GPT-4 generated visual instruction data: Industry-leading GPT-4 generated data
- Industry-leading LLaVA-1.5 + LLaVA-NeXT + VILA + Bunny ecosystem foundation: Industry-leading open VLM ecosystem
- 競合 GPT-4V + Gemini Vision + Claude 3 Vision + Qwen2-VL: Industry-leading VLM competitors
選び方・注意点
LLaVA は「Industry-leading open-source VLM + visual instruction tuning」「Industry-leading CLIP+LLaMA + first open VLM foundation」用途のIndustry-leading 2023年Liu et al. UW-Madison+MS Research発表product。Industry-leading visual instruction tuning (Industry-leading visual instruction tuning signature + Industry-leading LLaVA visual instruction signature + Industry-leading multimodal instruction tuning innovation) で Industry-leading visual instruction + Industry-leading LLaVA signature + Industry-leading multimodal innovation。Industry-leading CLIP+LLaMA (Industry-leading CLIP ViT-L/14 vision encoder + LLaMA + Vicuna language model + Industry-leading LLaVA CLIP+LLaMA signature + Industry-leading projection layer connection) で Industry-leading CLIP+LLaMA + Industry-leading LLaVA + Industry-leading projection layer。Industry-leading GPT-4 generated (Industry-leading GPT-4 generated visual instruction data + Industry-leading LLaVA GPT-4 teacher signature + Industry-leading 158K visual instructions) で Industry-leading GPT-4 generated + Industry-leading LLaVA teacher + Industry-leading 158K visual instructions。Industry-leading LLaVA family (Industry-leading LLaVA-7B + LLaVA-13B + LLaVA-1.5 + LLaVA-NeXT + Industry-leading LLaVA model family signature + Industry-leading open VLM ecosystem foundation) で Industry-leading LLaVA family + Industry-leading LLaVA-1.5/NeXT + Industry-leading open VLM ecosystem。Industry-leading first open VLM (Industry-leading first open-source VLM foundation + Industry-leading LLaVA-1.5/NeXT/VILA/Bunny ecosystem foundation + Industry-leading Haotian Liu + Chunyuan Li + Yong Jae Lee UW-Madison + Microsoft Research + Columbia + Industry-leading NeurIPS 2023 acceptance + Industry-leading GitHub LLaVA open implementation) で Industry-leading first open VLM + Industry-leading ecosystem foundation + Industry-leading Liu+Li+Lee UW-Madison+MS+Columbia + Industry-leading NeurIPS 2023 + Industry-leading GitHub open。但しIndustry-leading GPT-4V + Gemini Vision + Claude 3 Vision + Qwen2-VL competition で Industry-leading 4-VLM competitors vs LLaVA + Industry-leading open-source vision-language model + visual instruction tuning + CLIP ViT-L/14 vision encoder + LLaMA + Vicuna language model + GPT-4 generated visual instruction data + LLaVA-7B + LLaVA-13B + LLaVA-1.5 + LLaVA-NeXT + first open-source VLM foundation + LLaVA-1.5/NeXT/VILA/Bunny ecosystem + Liu + Li + Lee UW-Madison + Microsoft Research + Columbia NeurIPS 2023 unique advantage adoption alignment必須。
関連用語との違い
- vs GPT-4V (OpenAI 2023): LLaVAはopen-source foundation method 2023・GPT-4Vはproprietary OpenAI VLM 2023
- vs Gemini Vision (Google 2023): LLaVAはCLIP+LLaMA two-stage + open・Gemini Visionはnative multimodal Google + proprietary
- vs Qwen2-VL (Alibaba 2024): LLaVAは2023 first open VLM foundation・Qwen2-VLは2024 Alibaba VLM Naive dynamic resolution
よくある質問（FAQ）
Q1: LLaVA vs GPT-4V 違いは？ A: LLaVA (Industry-leading open-source vision-language model + visual instruction tuning + CLIP ViT-L/14 + LLaMA/Vicuna + GPT-4 generated visual instruction data + LLaVA-7B/13B/1.5/NeXT + first open VLM foundation + Liu UW-Madison + MS Research NeurIPS 2023) vs GPT-4V (Industry-leading proprietary OpenAI VLM + GPT-4 base + multimodal + OpenAI 2023)・Industry-leading open-source + visual instruction tuning + CLIP+LLaMA + LLaVA family + first open + UW-Madison+MS = LLaVA + Industry-leading proprietary + GPT-4 base + OpenAI = GPT-4V preference judgment。
Q2: Industry-leading visual instruction tuning + CLIP+LLaMA value は？ A: Industry-leading visual instruction + CLIP+LLaMA (Industry-leading visual instruction tuning signature + Industry-leading CLIP ViT-L/14 + LLaMA/Vicuna + Industry-leading LLaVA dual signature)。
Q3: Industry-leading first open VLM + ecosystem foundation value は？ A: Industry-leading first open VLM + ecosystem (Industry-leading first open-source VLM foundation + Industry-leading LLaVA-1.5/NeXT/VILA/Bunny ecosystem foundation + Industry-leading Liu+Li+Lee UW-Madison+MS+Columbia NeurIPS 2023)。
まとめ
LLaVA = 2023年Liu et al. UW-Madison + Microsoft Research + Columbia発表のfirst open-source VLM foundation method。Industry-leading open-source vision-language model + Industry-leading visual instruction tuning signature + Industry-leading CLIP ViT-L/14 vision encoder + Industry-leading LLaMA + Vicuna language model + Industry-leading GPT-4 generated visual instruction data 158K + Industry-leading LLaVA-7B + LLaVA-13B + LLaVA-1.5 + LLaVA-NeXT models + Industry-leading first open-source VLM foundation + Industry-leading LLaVA-1.5/NeXT/VILA/Bunny ecosystem foundation + Industry-leading Haotian Liu + Chunyuan Li + Qingyang Wu + Yong Jae Lee UW-Madison + Microsoft Research + Columbia authors + Industry-leading "Visual Instruction Tuning" arXiv 2304.08485 NeurIPS 2023 + Industry-leading first open-source VLM foundation Liu UW-Madison 2023 position確立。