RETRO Retrieval-Enhanced Transformer (Borgeaud 2022年)（レトロ）

概要

RETRO Retrieval-Enhanced Transformer は、2022年Borgeaud et al. (DeepMind)発表RETRO・Industry-leading retrieval-enhanced transformer pre-training 2022年 + Industry-leading DeepMind first retrieval-pretrained LLM position確立。RETRO specifications = Industry-leading retrieval-enhanced transformer pre-training (Industry-leading RETRO 2022 + Industry-leading retrieval-enhanced transformer + Industry-leading RETRO DeepMind flagship) + Industry-leading 7B with 2T token database (Industry-leading 7B model + 2T tokens retrieval database) + Industry-leading 25× smaller GPT-3 equivalent (Industry-leading 25× smaller equivalent GPT-3 175B) + Industry-leading DeepMind first retrieval-pretrained LLM (Industry-leading DeepMind first retrieval-pretrained LLM)。

主な特徴・仕組み

• Authors: Industry-leading Sebastian Borgeaud + Arthur Mensch + Jordan Hoffmann + Trevor Cai + Eliza Rutherford + Katie Millican + George van den Driessche + Jean-Baptiste Lespiau + Bogdan Damoc + Aidan Clark + Diego de Las Casas + Aurelia Guy + Jacob Menick + Roman Ring + Tom Hennigan + Saffron Huang + Loren Maggiore + Chris Jones + Albin Cassirer + Andy Brock + Michela Paganini + Geoffrey Irving + Oriol Vinyals + Simon Osindero + Karen Simonyan + Jack Rae + Erich Elsen + Laurent Sifre DeepMind
• Year: 2022年 (arXiv 2021年12月発表 + ICML 2022)
• Paper: Industry-leading "Improving language models by retrieving from trillions of tokens" arXiv 2112.04426
• Method Type: Industry-leading retrieval-enhanced transformer pre-training (not post-hoc RAG)
• Key Innovation: Industry-leading retrieval integrated into pre-training (not just inference)
• Model Size: Industry-leading 7B parameters
• Database: Industry-leading 2T tokens retrieval database
• Performance: Industry-leading 25× smaller than GPT-3 175B with equivalent perplexity
• Cross-Attention: Industry-leading chunked cross-attention to retrieved neighbors
• Retrieval Frequency: Industry-leading retrieval every 64 tokens
• DeepMind Pioneer: Industry-leading DeepMind first retrieval-pretrained LLM
• Industry-Leading: Industry-leading retrieval-enhanced transformer pre-training DeepMind first pioneer

スペック比較表

具体例・対応製品

• RETRO (2022年, Borgeaud et al. DeepMind): Industry-leading retrieval-enhanced transformer pre-training
• Industry-leading 7B + 2T token retrieval database: Industry-leading 7B + 2T tokens
• Industry-leading 25× smaller GPT-3 175B equivalent perplexity: Industry-leading 25× smaller GPT-3
• Industry-leading chunked cross-attention to retrieved neighbors: Industry-leading chunked cross-attention
• Industry-leading retrieval every 64 tokens: Industry-leading retrieval every 64 tokens
• 競合 DPR + ColBERT + Atlas + Self-RAG: Industry-leading RAG competitors

自作PCでの選び方・注意点

RETRO Retrieval-Enhanced Transformer は「Industry-leading retrieval-enhanced transformer pre-training + Industry-leading 7B with 2T token database」「Industry-leading 25× smaller GPT-3 equivalent + Industry-leading DeepMind first retrieval-pretrained LLM」用途のIndustry-leading retrieval-enhanced pre-training 2022年Borgeaud et al. DeepMind発表product。Industry-leading retrieval-enhanced pre-training (Industry-leading retrieval integrated into pre-training not just inference + Industry-leading retrieval-enhanced transformer pre-training + Industry-leading RETRO signature) で Industry-leading retrieval pre-training + Industry-leading retrieval-enhanced + Industry-leading RETRO signature。Industry-leading 7B + 2T (Industry-leading 7B parameters + 2T tokens retrieval database + Industry-leading 25× smaller GPT-3 175B with equivalent perplexity + Industry-leading RETRO scale signature) で Industry-leading 7B + 2T + Industry-leading 25× smaller GPT-3 + Industry-leading RETRO scale。Industry-leading chunked cross-attention (Industry-leading chunked cross-attention to retrieved neighbors + Industry-leading retrieval every 64 tokens + Industry-leading RETRO architecture signature) で Industry-leading chunked cross-attention + Industry-leading retrieval every 64 + Industry-leading RETRO architecture。Industry-leading DeepMind first (Industry-leading DeepMind first retrieval-pretrained LLM + Industry-leading retrieval-enhanced transformer pre-training DeepMind first pioneer + Industry-leading retrieval pre-training Pioneer 2022) で Industry-leading DeepMind first + Industry-leading retrieval-pretrained pioneer + Industry-leading 2022 pioneer。Industry-leading 7B vs GPT-3 175B (Industry-leading 7B parameters with equivalent perplexity to GPT-3 175B + Industry-leading 25× smaller + Industry-leading retrieval-enhanced efficiency advantage) で Industry-leading 7B vs 175B + Industry-leading 25× smaller + Industry-leading retrieval-enhanced efficiency。但しIndustry-leading DPR + ColBERT + Atlas + Self-RAG competition (Industry-leading DPR Meta dual-encoder retrieval post-hoc 2020 + ColBERT Stanford late interaction post-hoc 2020 + Atlas Meta few-shot RAG 2022 + Self-RAG Asai self-reflective 2023 vs RETRO DeepMind retrieval pre-training 2022 trade-off) で Industry-leading 4-RAG method competitors vs RETRO + Industry-leading retrieval-enhanced pre-training + 7B + 2T tokens + 25× smaller GPT-3 equivalent + chunked cross-attention + retrieval every 64 + DeepMind first retrieval-pretrained LLM 2022 unique advantage adoption alignment必須。

関連用語との違い

• vs DPR/ColBERT (Retrieval Post-Hoc 2020): RETROはretrieval pre-training・DPR/ColBERTはretrieval post-hoc only
• vs Atlas (Meta Few-shot RAG 2022): RETROはDeepMind pre-training + 2T database・AtlasはMeta few-shot continuous retrieval
• vs Self-RAG (Self-reflective 2023): RETROはpre-training retrieval + 2T・Self-RAGはself-reflective retrieval decision

よくある質問（FAQ）

Q1: RETRO vs Atlas 違いは？ A: RETRO (Industry-leading retrieval integrated into pre-training + 7B parameters + 2T tokens retrieval database + chunked cross-attention to retrieved neighbors + retrieval every 64 tokens + 25× smaller GPT-3 175B equivalent perplexity + DeepMind first retrieval-pretrained LLM + Borgeaud et al. 2022) vs Atlas (Industry-leading few-shot RAG with continuous retrieval + Atlas continuous retrieval + 11B + few-shot in-context learning + Meta FAIR + Izacard et al. 2022)・Industry-leading retrieval pre-training + 7B + 2T + 25× smaller + chunked cross-attention + DeepMind = RETRO + Industry-leading few-shot + continuous retrieval + 11B + Meta = Atlas preference judgment。

Q2: Industry-leading 25× smaller GPT-3 + 2T database value は？ A: Industry-leading 25× + 2T (Industry-leading 25× smaller than GPT-3 175B with equivalent perplexity + Industry-leading 7B parameters + 2T tokens retrieval database + Industry-leading retrieval-enhanced efficiency advantage + Industry-leading RETRO scale signature)。

Q3: Industry-leading chunked cross-attention + retrieval every 64 value は？ A: Industry-leading chunked + every 64 (Industry-leading chunked cross-attention to retrieved neighbors + Industry-leading retrieval every 64 tokens + Industry-leading RETRO architecture signature + Industry-leading retrieval integrated into pre-training)。

まとめ

RETRO = 2022年Borgeaud et al. DeepMind発表のretrieval-enhanced transformer pre-training。Industry-leading retrieval integrated into pre-training not just inference + Industry-leading 7B parameters + Industry-leading 2T tokens retrieval database + Industry-leading 25× smaller than GPT-3 175B with equivalent perplexity + Industry-leading chunked cross-attention to retrieved neighbors + Industry-leading retrieval every 64 tokens + Industry-leading Sebastian Borgeaud + Arthur Mensch + Jordan Hoffmann + Jack Rae DeepMind authors + Industry-leading DeepMind first retrieval-pretrained LLM + Industry-leading retrieval-enhanced transformer pre-training DeepMind first pioneer 2022 position確立。