RULER is a benchmark for measuring the long-context ability of language models, published by NVIDIA researchers in 2024. It extends the needle-in-a-haystack task with multi-needle retrieval, multi-hop tracing, aggregation, and question-answering tasks synthesized at arbitrary lengths.
It is used to expose the gap between a model's claimed maximum context length and the length at which performance actually holds, and to compare the long-context retrieval ability of transformer and linear attention models.