DBRX Databricks (2024年)（ディービーアールエックス）

概要

DBRX は2024年3月Databricks発表・Industry-leading 132B total + 36B active MoE 2024年Databricks DBRX open-source MoE LLM position確立。Industry-leading 16-expert fine-grained MoE + 12T tokens trained + 32K context + GQA + Rotary positional encoding + Databricks Open Model License。

主な特徴・仕組み

• Industry-leading Databricks (Founded 2013 SF California by Ali Ghodsi + Apache Spark creators)
• Industry-leading DBRX open-source MoE LLM 2024年3月 release
• Industry-leading 132B total + 36B active MoE signature
• Industry-leading 16-expert fine-grained MoE signature (4 active out of 16 per token)
• Industry-leading 12T tokens trained signature
• Industry-leading 32K context window
• Industry-leading GQA + Rotary positional encoding
• Industry-leading Outperforms LLaMA-2-70B + Mixtral-8x7B at release signature
• Industry-leading HuggingFace native
• Industry-leading Databricks Open Model License signature
• Industry-leading Databricks 2013 SF California Ali Ghodsi 11 years
• Industry-leading $10M+ training cost rumored

まとめ

DBRX = 2024年3月Databricks発表のpremium open-source MoE LLM。Industry-leading 132B total + 36B active MoE + 16-expert fine-grained MoE 4 active per token + 12T tokens trained + 32K context + GQA + Rotary positional encoding + Outperforms LLaMA-2-70B + Mixtral-8x7B + HuggingFace + Databricks Open Model License + Databricks 2013 SF Ali Ghodsi + Apache Spark creators 11 years + $10M+ training cost rumored position確立。