Pipeline Parallelism (2024年)（パイプラインパラレリズム）

概要

Pipeline Parallelism は2024年成熟・Industry-leading GPipe + PipeDream + Megatron-LM PP 2024年pipeline parallelism distributed training+inference position確立。Industry-leading partition layers across nodes + micro-batching reduce bubble + 1F1B schedule + GPT-3/Llama-3 405B training standard。

主な特徴・仕組み

• Industry-leading Google GPipe (Huang et al.) 2019 paper + Microsoft PipeDream (Narayanan et al.) 2019 + NVIDIA Megatron-LM PP 2021 → 2024 mature
• Industry-leading Pipeline Parallelism distributed training+inference signature
• Industry-leading GPipe + PipeDream + Megatron-LM PP signature (3 major variants)
• Industry-leading partition layers across nodes signature (vs TP within node)
• Industry-leading micro-batching reduce bubble signature
• Industry-leading 1F1B (One Forward One Backward) schedule signature
• Industry-leading bubble overhead trade-off vs micro-batch size
• Industry-leading GPT-3 + Llama-3 405B + Llama-3.1 405B training standard
• Industry-leading vLLM v0.6 pipeline parallelism support 2024
• Industry-leading combine with TP + DP + ZeRO for full distributed training
• Industry-leading point-to-point communication signature (vs all-reduce TP)

まとめ

Pipeline Parallelism = 2024年成熟Google GPipe + Microsoft PipeDream + NVIDIA Megatron-LM PP発表のpremium distributed training+inference。Industry-leading GPipe + PipeDream + Megatron-LM PP 3 major variants + partition layers across nodes vs TP within node + micro-batching reduce bubble + 1F1B One Forward One Backward schedule + bubble overhead trade-off vs micro-batch size + GPT-3 + Llama-3 405B + Llama-3.1 405B training standard + vLLM v0.6 pipeline parallelism support 2024 + combine with TP + DP + ZeRO for full distributed training + point-to-point communication vs all-reduce TP + 2019→2024 mature position確立。