SWE-rebench

A software engineering benchmark from Nebius that evaluates coding agents on continuously collected real GitHub issues.

1 articleLast mentioned

SWE-rebench is a benchmark that evaluates how well LLM-based coding agents resolve real software bugs and feature requests. It was released in 2025 by researchers at the cloud company Nebius. Tasks are built automatically from issues in public GitHub repositories and the pull requests that resolved them, and an agent's code changes are graded with hidden tests.

It follows the same format as the widely used SWE-bench, but instead of fixing a task set once it continuously adds recent issues to reduce contamination, where solutions appear in model training data. Tasks are drawn mainly from Python repositories, and a public leaderboard compares model results.

This entry is based on AIPOST articles and widely known facts. If something is wrong, please send us a correction request.

Articles covering this entry

  • Local Qwen 3.8 27B Coding Test: Four Harnesses Fix Nearly the Same BugsAI Performance ·

    Each combination attacked the same 68 real GitHub issues from the SWE-rebench suite under a 30-minute limit.

    Read article

Found a mistake or missing an entry?

Wiki entries are based on AIPOST articles and widely known facts. If an entry is wrong, send us a correction request and we will check and fix it. Names from new articles are added to the wiki after the article is published.