gradient descent

Optimization algorithm that minimizes a loss function by repeatedly moving parameters a small step in the direction opposite to the gradient.

2 articlesLast mentioned

Gradient descent is an iterative optimization algorithm for finding the minimum of a differentiable function. At the current parameters it computes the gradient of the loss function and then moves the parameters in the direction opposite to the steepest increase, by an amount proportional to the learning rate, repeating until the loss stops decreasing. In neural network training, the gradients obtained by backpropagation are passed to this algorithm to update the weights.

In practice, stochastic gradient descent (SGD), which estimates the gradient from a small mini-batch rather than the full dataset, is the common form, and optimizers such as momentum and Adam that adjust the direction and step size are all built on gradient descent.

This entry is based on AIPOST articles and widely known facts. If something is wrong, please send us a correction request.

Articles covering this entry

  • Temperature, Attention and Context Rot: Model Mechanics Behind Agent DesignAI in Practice ·

    Weights start random, so early forward passes produce noise until gradient descent trains them.

    Read article
  • AI Trust Depends on Auditable Reasoning, Not Louder Risk WarningsAI Ethics ·

    …nents to explain its decisions, faces a hard limit: networks trained by gradient descent behave more like complex living organisms than like machines with inspe…

    Read article

Found a mistake or missing an entry?

Wiki entries are based on AIPOST articles and widely known facts. If an entry is wrong, send us a correction request and we will check and fix it. Names from new articles are added to the wiki after the article is published.