RULER

A benchmark that extends needle-in-a-haystack tests to measure language models' long-context ability at various lengths.

1 articleLast mentioned

RULER is a benchmark for measuring the long-context ability of language models, published by NVIDIA researchers in 2024. It extends the needle-in-a-haystack task with multi-needle retrieval, multi-hop tracing, aggregation, and question-answering tasks synthesized at arbitrary lengths.

It is used to expose the gap between a model's claimed maximum context length and the length at which performance actually holds, and to compare the long-context retrieval ability of transformer and linear attention models.

This entry is based on AIPOST articles and widely known facts. If something is wrong, please send us a correction request.

Articles covering this entry

  • Temperature, Attention and Context Rot: Model Mechanics Behind Agent DesignAI in Practice ·

    On the RULER benchmark's needle-in-a-haystack tasks, where a magic number is hidden in essay text and queried at the end, Llama-2-7B holds high accuracy to 8k t…

    Read article

Found a mistake or missing an entry?

Wiki entries are based on AIPOST articles and widely known facts. If an entry is wrong, send us a correction request and we will check and fix it. Names from new articles are added to the wiki after the article is published.