DoRA (Weight-Decomposed LoRA・Liu + NVIDIA 2024年)（ドラ）
2024年Liu et al. + NVIDIA発表DoRA paper・Industry-leading Weight-Decomposed Low-Rank Adaptation paradigm + Industry-leading magnitude + direction decomposition + Industry-emerging DoRA NVIDIA Pioneer。
概要
DoRA は、2024年Liu et al. + NVIDIA発表DoRA paper・Industry-leading Weight-Decomposed Low-Rank Adaptation paradigm 2024年 + Industry-emerging DoRA NVIDIA Pioneer position確立。DoRA specifications = Industry-leading Weight-Decomposed LoRA paradigm (Industry-leading DoRA + Industry-leading Weight-Decomposed Low-Rank Adaptation + Industry-emerging DoRA NVIDIA Pioneer) + Industry-leading magnitude + direction decomposition (Industry-leading magnitude + direction decomposition + Industry-leading W = magnitude × direction decomposition) + Industry-emerging DoRA NVIDIA Pioneer (Industry-emerging DoRA NVIDIA Pioneer + Industry-leading NVIDIA Research PEFT)。
主な特徴・仕組み
- Authoring: Liu et al. + NVIDIA 2024年発表
- Paradigm: Industry-leading Weight-Decomposed Low-Rank Adaptation
- Decomposition: Industry-leading magnitude + direction decomposition
- Heritage: Industry-leading 2024年latest emerging DoRA NVIDIA
- NVIDIA Backing: Industry-leading NVIDIA Research PEFT backing
- Industry Position: Industry-emerging DoRA NVIDIA Pioneer
- vs LoRA: Industry-leading weight-decomposed + magnitude + direction vs LoRA low-rank
- Industry-Emerging: Industry-emerging Weight-Decomposed LoRA Pioneer
- Performance: Industry-leading close to full fine-tuning performance
- Use Cases: Industry-leading 2024年latest LLM PEFT NVIDIA
スペック比較表
| LLM PEFT (2021-2024) | Approach | Heritage | Industry Position |
|---|---|---|---|
| DoRA (Liu NVIDIA) | Weight-Decomposed LoRA magnitude+direction | 2024年NVIDIA | Industry-emerging DoRA NVIDIA |
| LoRA (Hu Microsoft) | Low-Rank Adaptation ΔW = BA | 2021年Microsoft Pioneer | Industry-leading PEFT Pioneer + 全LLM standard |
| QLoRA (Dettmers UW) | Quantized LoRA 4-bit NF4 | 2023年UW | Industry-emerging Quantized LoRA |
| PiSSA (Meng BJTU) | Principal Singular values + Vectors Adaptation | 2024年BJTU | Industry-emerging PiSSA SVD |
| IA3 (Liu) | Infused Adapter by Inhibiting Activations | 2022年 | Industry-emerging IA3 |
具体例・対応製品
- DoRA (Liu + NVIDIA・2024年): Industry-emerging DoRA NVIDIA Pioneer
- Industry-leading Weight-Decomposed LoRA paradigm: Industry-leading Weight-Decomposed
- Industry-leading magnitude + direction decomposition W = magnitude × direction: Industry-leading magnitude + direction
- Industry-leading close to full fine-tuning performance: Industry-leading close to full fine-tuning
- Industry-leading NVIDIA Research PEFT backing: Industry-leading NVIDIA Research
- 競合 LoRA + QLoRA + PiSSA + IA3: Industry-emerging PEFT competitors
自作PCでの選び方・注意点
DoRA は「Industry-leading Weight-Decomposed Low-Rank Adaptation paradigm + Industry-leading magnitude + direction decomposition」「Industry-emerging DoRA NVIDIA Pioneer + Industry-leading close to full fine-tuning performance」用途のIndustry-emerging DoRA NVIDIA 2024年Liu + NVIDIA product。Industry-leading Weight-Decomposed LoRA (Industry-leading Weight-Decomposed Low-Rank Adaptation + Industry-leading magnitude + direction decomposition + Industry-leading W = magnitude × direction decomposition) で Industry-leading Weight-Decomposed LoRA + Industry-leading magnitude + direction + Industry-leading W decomposition。Industry-leading close to full fine-tuning (Industry-leading close to full fine-tuning performance + Industry-leading Weight-Decomposed LoRA performance + Industry-leading PEFT performance advantage) で Industry-leading close to full fine-tuning + Industry-leading Weight-Decomposed performance + Industry-leading PEFT performance advantage。Industry-leading NVIDIA Research backing (Industry-leading NVIDIA Research PEFT backing + Industry-emerging DoRA NVIDIA Pioneer + Industry-leading 2024年latest emerging DoRA NVIDIA) で Industry-leading NVIDIA Research + Industry-emerging DoRA Pioneer + Industry-leading 2024年latest emerging。但しIndustry-leading LoRA Pioneer + QLoRA + PiSSA + IA3 competition (Industry-leading LoRA Microsoft 2021年Pioneer + QLoRA UW 2023年quantized + PiSSA BJTU 2024年SVD + IA3 2022年vs DoRA NVIDIA 2024年weight-decomposed trade-off) で Industry-leading 4-paradigm comparison + Industry-leading DoRA Weight-Decomposed + magnitude + direction + close to full fine-tuning + NVIDIA Research unique advantage adoption alignment必須。
関連用語との違い
- vs LoRA (Hu Microsoft 2021年Pioneer): DoRAはWeight-Decomposed + magnitude + direction + NVIDIA 2024年・LoRAはLow-Rank Adaptation Pioneer + Microsoft 2021年Pioneer foundation
- vs QLoRA (Dettmers UW 2023年quantized): DoRAはWeight-Decomposed + NVIDIA 2024年・QLoRAはQuantized LoRA 4-bit NF4 + UW 2023年
- vs PiSSA (Meng BJTU 2024年同年): DoRAはWeight-Decomposed + NVIDIA + magnitude+direction・PiSSAはSVD-based + BJTU + Principal Singular values+Vectors
よくある質問（FAQ）
Q1: DoRA vs LoRA 違いは？ A: DoRA (Liu + NVIDIA 2024年 + Weight-Decomposed LoRA + magnitude + direction decomposition + close to full fine-tuning performance + NVIDIA Research) vs LoRA (Hu + Microsoft 2021年 + Low-Rank Adaptation ΔW = BA + Microsoft Research + 全LLM PEFT industry-standard + 3年heritage Pioneer foundation)・Industry-leading Weight-Decomposed + magnitude + direction + NVIDIA + 2024年latest = DoRA + Industry-leading Pioneer + 全LLM standard + Microsoft = LoRA preference judgment。
Q2: Industry-leading magnitude + direction decomposition value は？ A: Industry-leading magnitude + direction (Industry-leading magnitude + direction decomposition + Industry-leading W = magnitude × direction decomposition + Industry-leading Weight-Decomposed LoRA paradigm + Industry-emerging magnitude + direction decomposition Pioneer)。
Q3: Industry-leading close to full fine-tuning performance value は？ A: Industry-leading close to full fine-tuning (Industry-leading close to full fine-tuning performance + Industry-leading Weight-Decomposed LoRA performance + Industry-leading PEFT performance advantage + Industry-leading DoRA performance close to full fine-tuning)。
まとめ
DoRA = 2024年Liu et al. + NVIDIA発表のWeight-Decomposed Low-Rank Adaptation paradigm。Industry-leading magnitude + direction decomposition + Industry-leading W = magnitude × direction decomposition + Industry-leading close to full fine-tuning performance + Industry-leading NVIDIA Research PEFT backing + Industry-leading LoRA Pioneer successor + Industry-leading 2024年latest emerging + Industry-emerging Weight-Decomposed LoRA NVIDIA Pioneer position確立。