GPQA (Graduate-Level Google-Proof Q&A Benchmark) is a question-answering benchmark released in 2023 by researchers at New York University and other institutions. It consists of 448 multiple-choice questions written by domain experts in biology, physics and chemistry. The questions are designed so that experts in the field can answer them while non-experts struggle even with unrestricted access to web search.
GPQA Diamond is a subset of 198 questions validated as the highest in quality, and it is widely used to compare the scientific reasoning of language models.