MMMU

MMMU is a benchmark that measures the understanding and reasoning ability of multimodal models using college-level questions across many disciplines.

2 articlesLast mentioned

MMMU (Massive Multi-discipline Multimodal Understanding and Reasoning) is a benchmark for evaluating the expert-level knowledge and reasoning of multimodal models that process both images and text. It was released in 2023 by researchers from several universities, including the Ohio State University and the University of Waterloo. It consists of about 11,500 questions collected from college exams and textbooks, covering 30 subjects in six disciplines: art and design, business, science, health and medicine, humanities and social science, and technology and engineering. The questions include many kinds of images, such as charts, maps, chemical structures, and musical scores.

MMMU-Pro, released in 2024, is a harder version of MMMU. It filters out questions that can be answered from text alone, expands the number of answer options to as many as ten, and adds a vision-only setting in which the question and options are embedded in an image so the model must read them from the screen. MMMU-Pro Vision refers to this vision-only setting. MMMU is one of the standard benchmarks used to compare major multimodal models such as GPT, Gemini, and Claude.

This entry is based on AIPOST articles and widely known facts. If something is wrong, please send us a correction request.

Articles covering this entry

  • Construction AI Tests Put Claude Ahead on Drawings, Take-offs Still Near 53%AI Performance ·

    Visual reasoning tests such as MMMU-Pro and legal reasoning tests such as Harvey's serve as partial stand-ins for reading drawings and managing contracts, but u…

    Read article
  • GPT-6.1 Terra vs Claude Opus 5.5: Eight Builds Cost $7.64 Against $60.19AI Performance ·

    On the MMMU-Pro Vision benchmark, Terra again anchors the top-left frontier, ahead of GPT-5.6 Terra, Astra and Sol, which suggests strong spatial and visual rea…

    Read article

Found a mistake or missing an entry?

Wiki entries are based on AIPOST articles and widely known facts. If an entry is wrong, send us a correction request and we will check and fix it. Names from new articles are added to the wiki after the article is published.