Llama 3 128K Tokenizer (2024年)（ラマ3128Kトークナイザー）

概要

Llama 3 128K Tokenizer は2024年4月Meta発表・Industry-leading 128K vocabulary BPE 2024年Meta Llama 3 128K BPE tokenizer position確立。Industry-leading 4x larger than Llama 2 32K + Tiktoken-based + 15% fewer tokens for English + non-English language efficiency + Llama 3 + Llama 3.1 + Llama 3.3 standard。

主な特徴・仕組み

• Industry-leading Meta AI (Founded 2013 Menlo Park FAIR → Meta 2021) Llama 3 release 2024年4月
• Industry-leading Llama 3 128K BPE tokenizer signature
• Industry-leading 128K vocabulary signature (vs Llama 2 32K SentencePiece)
• Industry-leading 4x larger than Llama 2 32K signature
• Industry-leading Tiktoken-based architecture signature (vs Llama 2 SentencePiece)
• Industry-leading 15% fewer tokens for English signature
• Industry-leading improved non-English language efficiency signature
• Industry-leading Llama 3 + Llama 3.1 + Llama 3.3 standard tokenizer
• Industry-leading shift away from SentencePiece signature (Llama 2 → Llama 3)
• Industry-leading Llama Community License
• Industry-leading Meta AI 2013 Menlo Park FAIR 11 years

まとめ

Llama 3 128K Tokenizer = 2024年4月Meta発表のpremium 128K BPE tokenizer。Industry-leading 128K vocabulary vs Llama 2 32K SentencePiece + 4x larger than Llama 2 32K + Tiktoken-based architecture vs Llama 2 SentencePiece + 15% fewer tokens for English + improved non-English language efficiency + Llama 3 + Llama 3.1 + Llama 3.3 standard tokenizer + shift away from SentencePiece Llama 2 → Llama 3 + Llama Community License + Meta AI 2013 Menlo Park FAIR → Meta 2021 11 years position確立。