Expert parallelism is a parallelization strategy used to train or serve large language models with a Mixture of Experts (MoE) architecture across multiple GPUs. The experts (specialized sub-networks) in each layer are placed on different GPUs, and each token is routed to the GPU holding the expert selected by the router, processed there, and gathered back.
It is used alongside data parallelism, which replicates the whole model, tensor parallelism, which splits the matrices within a layer, and pipeline parallelism, which splits layers across devices. Because each GPU only needs to hold a subset of experts, it reduces memory requirements for MoE models whose total parameter count is large. The all-to-all communication between GPUs and load imbalance across experts are the main factors that determine its performance.