Autoregressive refers to a model structure or generation method in which each next value is predicted conditioned on the outputs produced so far. In language models, the tokens generated up to that point are fed back as input, the model computes a probability distribution over the next token, one token is sampled and appended, and the process repeats to produce text.
Most large language models, including the GPT family, generate text autoregressively. Because tokens are produced one at a time in sequence, computation grows with output length, which is a major factor in generation latency and cost. In statistics, the term refers to autoregressive models that explain the current value of a time series using its past values.