MMMU (Massive Multi-discipline Multimodal Understanding and Reasoning) is a benchmark for evaluating the expert-level knowledge and reasoning of multimodal models that process both images and text. It was released in 2023 by researchers from several universities, including the Ohio State University and the University of Waterloo. It consists of about 11,500 questions collected from college exams and textbooks, covering 30 subjects in six disciplines: art and design, business, science, health and medicine, humanities and social science, and technology and engineering. The questions include many kinds of images, such as charts, maps, chemical structures, and musical scores.
MMMU-Pro, released in 2024, is a harder version of MMMU. It filters out questions that can be answered from text alone, expands the number of answer options to as many as ten, and adds a vision-only setting in which the question and options are embedded in an image so the model must read them from the screen. MMMU-Pro Vision refers to this vision-only setting. MMMU is one of the standard benchmarks used to compare major multimodal models such as GPT, Gemini, and Claude.