sharding
Sharding splits an AI model or dataset too large for one device into pieces that are stored and processed across many devices.
Sharding is a distribution technique that divides something large into pieces called shards and spreads them across multiple machines. In AI, model sharding refers to splitting a model's weights across several devices when the model is too large to fit in the memory of a single GPU or machine. Each device computes its portion and exchanges intermediate results with the others, so that together they train or run the model as one.
It is widely used for model parallelism across multiple GPUs, and because data moves frequently between devices, interconnect speed strongly affects performance. In databases, sharding means horizontal partitioning, in which rows of data are split across multiple servers.
This entry is based on AIPOST articles and widely known facts. If something is wrong, please send us a correction request.