SEC-bench is a benchmark for evaluating how well agents built on large language models perform software security tasks. Released in 2025, it is built from vulnerabilities reported in real open-source projects and assesses two tasks: writing proof-of-concept code that reproduces a vulnerability and patching it.
Agents analyze and modify code inside an execution environment and their results are verified automatically, which distinguishes SEC-bench from earlier security evaluations composed of short-answer questions.