Expert Parallelism

Expert parallelism is a distributed computing strategy that places the experts of an MoE model across multiple GPUs and runs them in parallel.

1 articleLast mentioned

Expert parallelism is a parallelization strategy used to train or serve large language models with a Mixture of Experts (MoE) architecture across multiple GPUs. The experts (specialized sub-networks) in each layer are placed on different GPUs, and each token is routed to the GPU holding the expert selected by the router, processed there, and gathered back.

It is used alongside data parallelism, which replicates the whole model, tensor parallelism, which splits the matrices within a layer, and pipeline parallelism, which splits layers across devices. Because each GPU only needs to hold a subset of experts, it reduces memory requirements for MoE models whose total parameter count is large. The all-to-all communication between GPUs and load imbalance across experts are the main factors that determine its performance.

This entry is based on AIPOST articles and widely known facts. If something is wrong, please send us a correction request.

Articles covering this entry

  • DeepSeek Opens DeepInfer Serving Engine Alongside MIT-Licensed V4.1-FlashAI News ·

    DeepEP, kernels for expert parallelism

    Read article

Found a mistake or missing an entry?

Wiki entries are based on AIPOST articles and widely known facts. If an entry is wrong, send us a correction request and we will check and fix it. Names from new articles are added to the wiki after the article is published.