learning rate

The scalar in gradient descent that sets how far parameters move along the gradient at each step; a key hyperparameter in neural network training.

1 articleLast mentioned

The learning rate is a positive scalar that determines how far parameters move in the direction opposite to the gradient at each step of gradient descent. The update rule typically takes the form "new parameter = current parameter − learning rate × gradient," so a larger learning rate means a larger move per step. Among the hyperparameters set by hand before training, it is generally regarded as the one with the greatest effect on results.

Too large a value makes the loss oscillate or diverge; too small a value makes convergence very slow or leaves the model stuck at a poor point. For this reason it is commonly combined with warmup, which raises the rate gradually at the start, with decay or cosine schedules that lower it over training, and with optimizers such as Adam that adapt the step size per parameter.

This entry is based on AIPOST articles and widely known facts. If something is wrong, please send us a correction request.

Articles covering this entry

  • Pre-Training, Fine-Tuning and RL: Where Frontier LLMs Get Their QualityAI Performance ·

    The size of each nudge is set by the learning rate.

    Read article

Found a mistake or missing an entry?

Wiki entries are based on AIPOST articles and widely known facts. If an entry is wrong, send us a correction request and we will check and fix it. Names from new articles are added to the wiki after the article is published.