YaRN (Peng 2023年)（ヤーン）
2023年Bowen Peng et al. (Nous Research+EleutherAI)発表YaRN・Industry-leading RoPE NTK-aware extension + Industry-leading Yet another RoPE extensioN + Industry-leading 128K context + Industry-leading widespread RoPE long context method。
概要
YaRN は、2023年Bowen Peng et al. (Nous Research+EleutherAI)発表YaRN・Industry-leading RoPE NTK-aware extension 2023年 + Industry-leading widespread RoPE long context method position確立。YaRN specifications = Industry-leading RoPE NTK-aware extension (Industry-leading YaRN Peng 2023 + Industry-leading RoPE NTK-aware + Industry-leading YaRN flagship) + Industry-leading Yet another RoPE extensioN + Industry-leading 128K context + Industry-leading widespread RoPE long context method。
主な特徴・仕組み
- Authors: Industry-leading Bowen Peng + Jeffrey Quesnelle + Honglu Fan + Enrico Shippole (Nous Research + EleutherAI)
- Year: 2023年 (arXiv 2023年9月発表 ICLR 2024)
- Paper: Industry-leading "YaRN: Efficient Context Window Extension of Large Language Models" arXiv 2309.00071
- Method Type: Industry-leading RoPE NTK-aware extension method
- YaRN Acronym: Industry-leading "Yet another RoPE extensioN" YaRN signature
- NTK-Aware: Industry-leading NTK Neural Tangent Kernel aware signature
- NTK-by-parts: Industry-leading NTK-by-parts interpolation signature
- 128K Context: Industry-leading 128K context achievement
- Widespread Adoption: Industry-leading Mistral + Mixtral + Qwen + Yi LLM adoption
- Open Source: Industry-leading GitHub YaRN open implementation
- Training-Free + Fine-tune: Industry-leading training-free + fine-tuning variants
- Industry-Leading: Industry-leading widespread RoPE long context method Peng Nous Research+EleutherAI 2023
スペック比較表
| LLM Long Context (2023-2024) | Year | Org | Method | Industry Position |
|---|---|---|---|---|
| YaRN | 2023 | Peng et al. Nous+EleutherAI | RoPE NTK-aware | Industry-leading RoPE extension widespread |
| LongLoRA | 2023 | Chen et al. MIT+CUHK | S2-Attn + LoRA | Industry-leading efficient long context |
| LongChat | 2023 | Li et al. LMSYS | Condensing rotary embedding | Industry-leading LongChat 32K |
| Position Interpolation | 2023 | Chen et al. Meta | Linear interpolation RoPE | Industry-leading PI foundation |
| RoPE-based Extrapolation | 2024 | Various | RoPE extrapolation | Industry-leading RoPE extrapolation |
具体例・対応製品
- YaRN (2023年, Peng et al. Nous Research+EleutherAI): Industry-leading RoPE NTK-aware extension
- Industry-leading "Yet another RoPE extensioN" YaRN: Industry-leading YaRN acronym
- Industry-leading NTK Neural Tangent Kernel aware: Industry-leading NTK-aware
- Industry-leading NTK-by-parts interpolation: Industry-leading NTK-by-parts
- Industry-leading Mistral + Mixtral + Qwen + Yi adoption: Industry-leading Mistral+Mixtral+Qwen+Yi widespread
- 競合 LongLoRA + LongChat + Position Interpolation + RoPE-based Extrapolation: Industry-leading long context competitors
選び方・注意点
YaRN は「Industry-leading RoPE NTK-aware extension + Yet another RoPE extensioN」「Industry-leading 128K context + widespread RoPE method」用途のIndustry-leading 2023年Peng et al. Nous Research+EleutherAI発表product。Industry-leading RoPE NTK-aware (Industry-leading RoPE NTK-aware extension + Industry-leading NTK Neural Tangent Kernel aware signature + Industry-leading YaRN NTK-aware signature) で Industry-leading RoPE NTK-aware + Industry-leading NTK signature + Industry-leading YaRN signature。Industry-leading Yet another RoPE (Industry-leading "Yet another RoPE extensioN" YaRN signature + Industry-leading YaRN acronym + Industry-leading RoPE extension family) で Industry-leading Yet another RoPE + Industry-leading YaRN acronym + Industry-leading RoPE family。Industry-leading 128K context (Industry-leading 128K context achievement + Industry-leading YaRN 128K signature + Industry-leading 128K context extension) で Industry-leading 128K context + Industry-leading YaRN signature + Industry-leading 128K extension。Industry-leading NTK-by-parts (Industry-leading NTK-by-parts interpolation signature + Industry-leading YaRN NTK-by-parts signature + Industry-leading frequency-dependent interpolation) で Industry-leading NTK-by-parts + Industry-leading YaRN signature + Industry-leading frequency-dependent。Industry-leading widespread (Industry-leading Mistral + Mixtral + Qwen + Yi LLM adoption + Industry-leading widespread RoPE long context method + Industry-leading Peng + Quesnelle + Fan + Shippole Nous Research+EleutherAI ICLR 2024) で Industry-leading widespread + Industry-leading Mistral+Mixtral+Qwen+Yi + Industry-leading Peng+Nous+EleutherAI。但しIndustry-leading LongLoRA + LongChat + Position Interpolation + RoPE-based Extrapolation competition で Industry-leading 4-long context competitors vs YaRN + Industry-leading RoPE NTK-aware extension + Yet another RoPE extensioN + NTK Neural Tangent Kernel aware + NTK-by-parts interpolation + 128K context + Mistral+Mixtral+Qwen+Yi widespread adoption + training-free + fine-tuning variants + Peng + Quesnelle + Fan + Shippole Nous Research + EleutherAI ICLR 2024 unique advantage adoption alignment必須。
関連用語との違い
- vs LongLoRA (Chen MIT+CUHK 2023): YaRNはRoPE NTK-aware extension + widespread・LongLoRAはS2-Attn + LoRA + 100K
- vs Position Interpolation (Chen Meta 2023): YaRNはNTK-by-parts + frequency-aware・PIはlinear interpolation foundation
- vs LongChat (Li LMSYS 2023): YaRNは128K + widespread Mistral/Qwen・LongChatはCondensing + 32K
よくある質問（FAQ）
Q1: YaRN vs Position Interpolation 違いは？ A: YaRN (Industry-leading RoPE NTK-aware extension + Yet another RoPE extensioN + NTK-by-parts interpolation + 128K context + Mistral+Mixtral+Qwen+Yi widespread adoption + Peng + Quesnelle Nous Research+EleutherAI 2023) vs Position Interpolation (Industry-leading Linear interpolation RoPE + PI foundation method + Chen et al. Meta + RoPE-based + 2023)・Industry-leading NTK-aware + Yet another RoPE + NTK-by-parts + 128K + widespread + Peng+Nous = YaRN + Industry-leading Linear interpolation + PI foundation + Chen+Meta + 2023 = PI preference judgment (NTK-aware vs Linear differ)。
Q2: Industry-leading RoPE NTK-aware + Yet another RoPE value は？ A: Industry-leading RoPE NTK-aware + Yet another RoPE (Industry-leading RoPE NTK-aware extension signature + Industry-leading "Yet another RoPE extensioN" YaRN acronym + Industry-leading YaRN dual signature)。
Q3: Industry-leading 128K + widespread Mistral+Mixtral+Qwen+Yi value は？ A: Industry-leading 128K + widespread (Industry-leading 128K context achievement + Industry-leading Mistral + Mixtral + Qwen + Yi LLM widespread adoption + Industry-leading YaRN widespread RoPE long context method position)。
まとめ
YaRN = 2023年Peng et al. Nous Research + EleutherAI発表のRoPE NTK-aware extension method。Industry-leading RoPE NTK-aware extension signature + Industry-leading "Yet another RoPE extensioN" YaRN acronym + Industry-leading NTK Neural Tangent Kernel aware + Industry-leading NTK-by-parts interpolation + Industry-leading 128K context achievement + Industry-leading Mistral + Mixtral + Qwen + Yi LLM widespread adoption + Industry-leading training-free + fine-tuning variants + Industry-leading Bowen Peng + Jeffrey Quesnelle + Honglu Fan + Enrico Shippole Nous Research + EleutherAI authors + Industry-leading "YaRN: Efficient Context Window Extension of Large Language Models" arXiv 2309.00071 ICLR 2024 + Industry-leading widespread RoPE long context method Peng Nous Research+EleutherAI 2023 position確立。