LongChat (Li 2023年)（ロングチャット）
2023年Dacheng Li et al. (LMSYS)発表LongChat・Industry-leading 32K context LLaMA + Industry-leading Condensing Rotary Embedding + Industry-leading LongEval benchmark + Industry-leading LMSYS Vicuna-based long context。
概要
LongChat は、2023年Dacheng Li et al. (LMSYS)発表LongChat・Industry-leading 32K context LLaMA 2023年 + Industry-leading LMSYS Vicuna-based long context position確立。LongChat specifications = Industry-leading 32K context LLaMA (Industry-leading LongChat Li LMSYS 2023 + Industry-leading 32K context + Industry-leading LongChat flagship) + Industry-leading Condensing Rotary Embedding + Industry-leading LongEval benchmark + Industry-leading LMSYS Vicuna-based long context。
主な特徴・仕組み
- Authors: Industry-leading Dacheng Li + Rulin Shao + Anze Xie + Ying Sheng + Lianmin Zheng + Joseph E. Gonzalez + Ion Stoica + Xuezhe Ma + Hao Zhang (LMSYS Org + UC Berkeley + CMU + UCSD)
- Year: 2023年 (LMSYS blog 2023年6月発表)
- Blog Post: Industry-leading "How Long Can Open-Source LLMs Truly Promise on Context Length?" LMSYS blog 2023
- Method Type: Industry-leading 32K context LLaMA + Vicuna-based
- Condensing Rotary: Industry-leading Condensing Rotary Embedding signature
- 32K Context: Industry-leading 32K context LongChat-7B/13B achievement
- LongEval Benchmark: Industry-leading LongEval long context benchmark
- Vicuna-Based: Industry-leading LMSYS Vicuna foundation
- Topics Retrieval: Industry-leading Topics retrieval evaluation
- Open Source: Industry-leading LongChat-7B/13B open release
- LMSYS Org: Industry-leading LMSYS Org (Chatbot Arena + Vicuna + LongChat)
- Industry-Leading: Industry-leading LMSYS Vicuna-based long context Li 2023
スペック比較表
| LLM Long Context (2023-2024) | Year | Org | Method | Industry Position |
|---|---|---|---|---|
| LongChat | 2023 | Li et al. LMSYS | Condensing rotary embedding | Industry-leading LongChat 32K |
| LongLoRA | 2023 | Chen et al. MIT+CUHK | S2-Attn + LoRA | Industry-leading efficient long context |
| YaRN | 2023 | Peng et al. Nous+EleutherAI | RoPE NTK-aware | Industry-leading RoPE extension widespread |
| Position Interpolation | 2023 | Chen et al. Meta | Linear interpolation RoPE | Industry-leading PI foundation |
| RoPE-based Extrapolation | 2024 | Various | RoPE extrapolation | Industry-leading RoPE extrapolation |
具体例・対応製品
- LongChat (2023年, Li et al. LMSYS): Industry-leading 32K context LLaMA + Vicuna-based
- Industry-leading Condensing Rotary Embedding signature: Industry-leading Condensing Rotary
- Industry-leading 32K context LongChat-7B/13B: Industry-leading 32K context
- Industry-leading LongEval long context benchmark: Industry-leading LongEval benchmark
- Industry-leading LMSYS Vicuna foundation: Industry-leading LMSYS Vicuna
- 競合 LongLoRA + YaRN + Position Interpolation + RoPE-based Extrapolation: Industry-leading long context competitors
選び方・注意点
LongChat は「Industry-leading 32K context LLaMA + Condensing Rotary Embedding」「Industry-leading LongEval benchmark + LMSYS Vicuna-based」用途のIndustry-leading 2023年Li et al. LMSYS発表product。Industry-leading 32K context (Industry-leading 32K context LongChat-7B/13B achievement + Industry-leading LongChat 32K signature + Industry-leading LLaMA 32K extension) で Industry-leading 32K context + Industry-leading LongChat signature + Industry-leading LLaMA 32K。Industry-leading Condensing Rotary (Industry-leading Condensing Rotary Embedding signature + Industry-leading LongChat Condensing signature + Industry-leading rotary embedding compression) で Industry-leading Condensing Rotary + Industry-leading LongChat signature + Industry-leading rotary compression。Industry-leading LongEval (Industry-leading LongEval long context benchmark + Industry-leading LongChat LongEval signature + Industry-leading Topics retrieval + Lines retrieval evaluation) で Industry-leading LongEval + Industry-leading LongChat benchmark + Industry-leading Topics+Lines retrieval。Industry-leading LMSYS Vicuna (Industry-leading LMSYS Vicuna foundation + Industry-leading LMSYS Org Chatbot Arena + Vicuna + LongChat ecosystem + Industry-leading Vicuna-based long context) で Industry-leading LMSYS Vicuna + Industry-leading LMSYS Org + Industry-leading Vicuna-based。Industry-leading open source (Industry-leading LongChat-7B/13B open release + Industry-leading LMSYS open release signature + Industry-leading Dacheng Li + Lianmin Zheng + Joseph Gonzalez + Ion Stoica + Hao Zhang LMSYS+UC Berkeley+CMU+UCSD) で Industry-leading open source + Industry-leading LMSYS open + Industry-leading Li+Zheng+Gonzalez+Stoica+Zhang。但しIndustry-leading LongLoRA + YaRN + Position Interpolation + RoPE-based Extrapolation competition で Industry-leading 4-long context competitors vs LongChat + Industry-leading Condensing Rotary Embedding + 32K context LongChat-7B/13B + LongEval long context benchmark + Topics+Lines retrieval + LMSYS Vicuna foundation + Dacheng Li + Lianmin Zheng + Joseph Gonzalez + Ion Stoica + Hao Zhang LMSYS Org + UC Berkeley + CMU + UCSD 2023 unique advantage adoption alignment必須。
関連用語との違い
- vs LongLoRA (Chen MIT+CUHK 2023): LongChatはCondensing Rotary + 32K + Vicuna・LongLoRAはS2-Attn + LoRA + 100K + LLaMA
- vs YaRN (Peng Nous 2023): LongChatはCondensing + 32K + LMSYS・YaRNはNTK-aware + 128K + widespread
- vs Position Interpolation (Chen Meta 2023): LongChatはCondensing + Vicuna + LMSYS・PIはLinear interpolation + Meta foundation
よくある質問（FAQ）
Q1: LongChat vs YaRN 違いは？ A: LongChat (Industry-leading Condensing Rotary Embedding + 32K context LongChat-7B/13B + LongEval long context benchmark + LMSYS Vicuna foundation + Topics+Lines retrieval + Dacheng Li + Lianmin Zheng + Joseph Gonzalez + Ion Stoica + Hao Zhang LMSYS Org + UC Berkeley + CMU + UCSD 2023) vs YaRN (Industry-leading RoPE NTK-aware extension + Yet another RoPE + NTK-by-parts + 128K + Mistral+Mixtral+Qwen+Yi widespread + Peng Nous+EleutherAI 2023)・Industry-leading Condensing Rotary + 32K + LongEval + LMSYS Vicuna + Topics/Lines retrieval + LMSYS Org = LongChat + Industry-leading NTK-aware + Yet another RoPE + NTK-by-parts + 128K + widespread Mistral+Mixtral+Qwen+Yi = YaRN preference judgment。
Q2: Industry-leading Condensing Rotary + 32K value は？ A: Industry-leading Condensing Rotary + 32K (Industry-leading Condensing Rotary Embedding signature + Industry-leading 32K context LongChat-7B/13B achievement + Industry-leading LongChat dual signature)。
Q3: Industry-leading LongEval + LMSYS Vicuna value は？ A: Industry-leading LongEval + LMSYS Vicuna (Industry-leading LongEval long context benchmark + Industry-leading LMSYS Vicuna foundation + Industry-leading LMSYS Org Chatbot Arena + Vicuna + LongChat ecosystem)。
まとめ
LongChat = 2023年Li et al. LMSYS発表のlong context Vicuna-based LLaMA。Industry-leading Condensing Rotary Embedding signature + Industry-leading 32K context LongChat-7B/13B achievement + Industry-leading LongEval long context benchmark + Industry-leading LMSYS Vicuna foundation + Industry-leading Topics + Lines retrieval evaluation + Industry-leading LongChat-7B/13B open release + Industry-leading Dacheng Li + Rulin Shao + Anze Xie + Ying Sheng + Lianmin Zheng + Joseph E. Gonzalez + Ion Stoica + Xuezhe Ma + Hao Zhang LMSYS Org + UC Berkeley + CMU + UCSD authors + Industry-leading "How Long Can Open-Source LLMs Truly Promise on Context Length?" LMSYS blog 2023 + Industry-leading LMSYS Vicuna-based long context Li 2023 position確立。