SWE-bench

SWE-bench is a software engineering benchmark that measures whether AI models can resolve real GitHub issues by generating patches.

4 articlesLast mentioned

SWE-bench is a benchmark that measures the software engineering capabilities of AI models. It was released in 2023 by researchers at Princeton University and is built from real GitHub issues and their corresponding pull requests in widely used open-source Python repositories. A model is given the issue description and the repository code, generates a patch to fix the problem, and the task counts as resolved if the patched code passes predefined unit tests.

Because scoring is based on the fraction of issues resolved, the benchmark does not evaluate code style or maintainability. Variants include SWE-bench Verified, a human-validated subset of 500 tasks released by OpenAI in 2024, as well as SWE-bench Lite and SWE-bench Multimodal, and it is one of the standard references for comparing coding agents and language models.

This entry is based on AIPOST articles and widely known facts. If something is wrong, please send us a correction request.

Articles covering this entry

  • AI Agent Tuning: When to Rewrite Prompts and When to Retrain WeightsAI Performance ·

    Second, data volume matters: OpenThoughts-Agent results on SWE-bench Verified and Terminal-Bench 2.0 improve steadily as the dataset grows from 316 to 100,000 s…

    Read article
  • Anthropic S-1 RSI Disclosure: 2x Research Speedup, Not a Runaway LoopAI News ·

    Coding benchmarks such as SWE-bench, FrontierSWE, Cognition's evaluations, and Terminal Bench 4 are common reference points, but inspecting public trace logs of…

    Read article
  • Prompt Audits and Cache Reuse Cut Claude Costs Up to 72.9%, Anthropic SaysAI in Practice ·

    SWE-bench Verified -14.3% +1.1 points

    Read article
  • ChatGPT for Financial Services: One Shared Model, Hosted Data, Human ReviewAI News ·

    …on's FrontierCode benchmark asks not whether a patch resolves an issue, as SWE-bench does, but whether a human maintainer would actually merge the pull request…

    Read article

Found a mistake or missing an entry?

Wiki entries are based on AIPOST articles and widely known facts. If an entry is wrong, send us a correction request and we will check and fix it. Names from new articles are added to the wiki after the article is published.