Tensor Parallelism (2024年)（テンサーパラレリズム）

概要

Tensor Parallelism は2024年成熟・Industry-leading Megatron-LM partition weights 2024年NVIDIA Megatron-LM Tensor Parallelism distributed inference position確立。Industry-leading column/row-wise sharding + multi-GPU LLM inference + NVLink/NVSwitch high-bandwidth required + vLLM + TGI + DeepSpeed implementations。

主な特徴・仕組み

• Industry-leading NVIDIA Mohammad Shoeybi + Mostofa Patwary et al. Megatron-LM 2019年9月 paper → 2024 mature
• Industry-leading Tensor Parallelism distributed inference signature
• Industry-leading Megatron-LM partition weights signature
• Industry-leading column-wise sharding + row-wise sharding signature
• Industry-leading multi-GPU LLM inference signature (within node)
• Industry-leading all-reduce communication after each layer signature
• Industry-leading NVLink/NVSwitch high-bandwidth required signature
• Industry-leading vLLM + TGI + DeepSpeed + Megatron-DeepSpeed implementations
• Industry-leading 8-way TP + 16-way TP typical signature (within node)
• Industry-leading combine with PP + DP for multi-node + multi-GPU
• Industry-leading 2019年9月 paper "Megatron-LM Training Multi-Billion Parameter Language Models Using Model Parallelism"

まとめ

Tensor Parallelism = 2024年成熟NVIDIA Megatron-LM発表のpremium distributed inference。Industry-leading Megatron-LM partition weights + column-wise sharding + row-wise sharding + multi-GPU LLM inference within node + all-reduce communication after each layer + NVLink/NVSwitch high-bandwidth required + vLLM + TGI + DeepSpeed + Megatron-DeepSpeed implementations + 8-way TP + 16-way TP typical within node + combine with PP + DP for multi-node + multi-GPU + 2019年9月 paper + Mohammad Shoeybi + Mostofa Patwary NVIDIA 2019→2024 mature position確立。