Jamba AI21 (2024年)（ジャンバ）

概要

Jamba は2024年3月AI21 Labs発表・Industry-leading Mamba + Transformer hybrid + MoE 2024年AI21 Labs Jamba Apache 2.0 hybrid Mamba-Transformer MoE position確立。Industry-leading 52B total + 12B active + 256K context (first 256K MoE) + Mamba state-space layer + Transformer layer interleaved。

主な特徴・仕組み

• Industry-leading AI21 Labs (Founded 2017 Tel Aviv Israel by Yoav Shoham + Ori Goshen + Amnon Shashua)
• Industry-leading Jamba hybrid Mamba-Transformer MoE 2024年3月 release
• Industry-leading Mamba + Transformer hybrid architecture signature (first production-scale hybrid)
• Industry-leading 52B total + 12B active MoE signature
• Industry-leading 256K context window signature (first 256K MoE)
• Industry-leading Mamba state-space layer + Transformer layer interleaved
• Industry-leading 16-expert MoE + 2 active per token
• Industry-leading 3x throughput vs Mixtral-8x7B signature
• Industry-leading Apache 2.0 open-source license signature
• Industry-leading HuggingFace native
• Industry-leading AI21 Labs 2017 Tel Aviv Yoav Shoham + Ori Goshen + Amnon Shashua 7 years

まとめ

Jamba = 2024年3月AI21 Labs発表のpremium Apache 2.0 hybrid Mamba-Transformer MoE。Industry-leading Mamba + Transformer hybrid architecture first production-scale + 52B total + 12B active MoE + 256K context first 256K MoE + Mamba state-space layer + Transformer layer interleaved + 16-expert MoE + 2 active per token + 3x throughput vs Mixtral-8x7B + Apache 2.0 + HuggingFace + AI21 Labs 2017 Tel Aviv Yoav Shoham + Ori Goshen + Amnon Shashua 7 years position確立。