autograd

Automatic differentiation: the mechanism in deep learning frameworks that records computations and automatically derives the gradient of a loss with respect to every parameter.

1 articleLast mentioned

Autograd is short for automatic differentiation, a technique and its implementation that records the operations a program performs and then traverses that record to compute the derivative of an output with respect to each input. In deep learning frameworks it refers to the engine that stores every operation of the forward pass as a computation graph and obtains the gradient of the loss with respect to all parameters at once through backpropagation. Representative implementations include PyTorch's torch.autograd module, JAX's grad transformation, and TensorFlow's GradientTape.

Unlike symbolic differentiation, where derivatives are worked out by hand, or numerical differentiation, which approximates with small finite differences, automatic differentiation applies the chain rule operation by operation to obtain exact derivatives efficiently. This is what allows models with billions of parameters to be trained by gradient descent without anyone writing derivative formulas.

This entry is based on AIPOST articles and widely known facts. If something is wrong, please send us a correction request.

Articles covering this entry

  • Pre-Training, Fine-Tuning and RL: Where Frontier LLMs Get Their QualityAI Performance ·

    With billions of parameters, that is impossible, which is where automatic differentiation frameworks such as PyTorch, JAX and TensorFlow come in.

    Read article

Found a mistake or missing an entry?

Wiki entries are based on AIPOST articles and widely known facts. If an entry is wrong, send us a correction request and we will check and fix it. Names from new articles are added to the wiki after the article is published.