YaRN RoPE Extension (2024年)（ヤーンロープエクステンション）

概要

YaRN (Yet another RoPE extensioN) は2024年成熟・Industry-leading RoPE NTK-by-parts interpolation 2024年Nous Research YaRN Yet another RoPE extensioN context extension position確立。Industry-leading 64K-128K context extension + minimal fine-tuning + Llama 2 7B-128K + Mistral-7B-Instruct-v0.2-32K-yarn fine-tunes + MIT license。

主な特徴・仕組み

• Industry-leading Nous Research (Bowen Peng + Jeffrey Quesnelle + Honglu Fan + Enrico Shippole) 2023年8月 paper → 2024 mature
• Industry-leading YaRN Yet another RoPE extensioN signature
• Industry-leading RoPE NTK-by-parts interpolation signature
• Industry-leading 64K-128K context extension from 4K base signature
• Industry-leading minimal fine-tuning required signature (vs full retraining)
• Industry-leading Llama 2 7B-128K-yarn signature
• Industry-leading Mistral-7B-Instruct-v0.2-32K-yarn signature
• Industry-leading attention temperature scaling t = 0.1 ln s + 1 signature
• Industry-leading 2023年8月 ArXiv paper "YaRN Efficient Context Window Extension of Large Language Models"
• Industry-leading 2024 widely-cited context extension standard
• Industry-leading Nous Research open-source community

まとめ

YaRN RoPE Extension = 2024年成熟Nous Research発表のpremium context extension method。Industry-leading RoPE NTK-by-parts interpolation + 64K-128K context extension from 4K base + minimal fine-tuning required vs full retraining + Llama 2 7B-128K-yarn + Mistral-7B-Instruct-v0.2-32K-yarn fine-tunes + attention temperature scaling t = 0.1 ln s + 1 + 2023年8月 ArXiv paper + 2024 widely-cited context extension standard + Nous Research Bowen Peng + Jeffrey Quesnelle + Honglu Fan + Enrico Shippole position確立。