A set of construction-specific benchmarks built by a construction software company tells a different story from the coding and general-knowledge leaderboards that dominate AI launch announcements. Tested on real drawing sets, quantity take-offs and a deliberately messy warehouse bid package, Anthropic's Claude models took the top spots for reading drawings and for construction management judgment, OpenAI's GPT-6 family followed closely, and Google's Gemini 4 Argon trailed well behind on management tasks. The same tests also exposed two findings that matter more than any single ranking: the software wrapper around a model can swing its accuracy by more than 30 percentage points, and no model yet measures quantities reliably enough to price a bid on its own.

Why generic benchmarks fall short for construction

A benchmark is a standardized test that scores and ranks AI models on a fixed set of tasks. The public ones that get cited most often measure general knowledge work, coding agents or software engineering. Visual reasoning tests such as MMMU-Pro and legal reasoning tests such as Harvey's serve as partial stand-ins for reading drawings and managing contracts, but until this suite existed, nothing measured construction workflows directly.

That gap matters because vendor launch charts for releases like Gemini 4 Argon, Claude Opus 5.5 or GPT-6 tend to feature whichever benchmarks put the new model on top. The construction tests described here were designed to sidestep that problem by scoring models on the work a site office actually does: answering questions from drawings, counting and measuring quantities, retrieving facts from project records, and producing estimates, schedules, progress reports, procurement comparisons, contract administration and cost reports.

The suite splits into two kinds of tests:

  • Objective tests have one correct answer, such as how many UPS outlets appear in a room, and are scored right or wrong.
  • Subjective tests ask for a full deliverable, such as a tender estimate, and are graded against a written rubric covering scope coverage, work breakdown, pricing logic and clarity about inclusions and exclusions.

The subjective tests also hide traps. One tender document contains a single sentence stating that there is no parking near the site. A model that reads carefully should add a transport allowance or extra travel time to its estimate.

The harness changes the result

Construction professionals rarely call a model's raw API. They use applications such as ChatGPT Work, Claude Code, Claude Cowork or Gemini connected to Google Drive. These applications, called harnesses, wrap the model with instructions, project context, file management tools and reasoning scaffolding. The benchmark therefore tested 37 raw models and 22 harness-model pairings separately.

Bare glowing cube with a short bar beside a cube mounted in a tool-filled frame with a much taller bar

▲ Accuracy gain from a software harness over a raw API

The gap between the two is large. On the drawing question-and-answer test, the best raw API result came from OpenAI's GPT-6 Astra at 82.1 percent accuracy, at a listed run cost of $24.11. Inside Claude Cowork, Claude Sonnet 5.5 reached 93.3 percent, a gain of 32.8 percentage points over its raw baseline. GPT-5.6 Sol gained 11.2 points inside ChatGPT Work. The practical lesson is that a raw API score tells a non-coding user little about what they will get from the app on their desk.

Drawing Q&A: Claude models cluster at the top

The drawing benchmark asks 169 questions across nine real drawing sets covering structural, electrical, civil and mechanical disciplines. A typical question asks whether distribution board DB-GB on drawing 402 is labeled essential or non-essential on a single-line diagram. Claude Sonnet 5.5 read the bus label correctly as non-essential; GPT-5.6 Sol confused it with the wider system label above and answered essential with high confidence.

Results inside harnesses:

Model and harness Drawing Q&A accuracy
Claude Sonnet 5.5 in Claude Cowork 93.3%
Claude Fable 5.1 in Claude Cowork 92.5%
Claude Opus 5.5 in Claude Cowork 92.5% and 91.8% across runs
GPT-6 Astra, raw API (best raw result) 82.1%

The spread among the three leading Claude models, from 91.8 to 93.3 percent, falls within normal run-to-run variation and should not be read as a real capability difference.

Quantity take-offs: counting works, measuring does not

The take-off benchmark uses two real projects, an electrical package for a performing arts center and a plumbing package for a supermarket, totaling 130 quantity lines across 33 models and 18 harnesses. Counts of fixtures, outlets and fittings had to match exactly; measured runs of pipe and cable were accepted within 5 percent of in-house verified quantities.

Model and harness Take-off accuracy
Claude Opus 5.5 in Claude Cowork 53.1%
Gemini 4 Argon 52.3%
Claude Sonnet 5.5 in Claude Cowork 49.2%
GPT-6 Astra and GPT-6.1 Sol in ChatGPT Work 46.2%
Claude Fable 5.1 in Claude Cowork 43.9%
Claude Opus 5 in Claude Code 43.1%

Models count discrete symbols on clean vector drawings well. In a flexi-office example, Claude Opus 5.5 found all six UPS outlets while GPT-6 Astra found four. Where they stumble is scale: pipe routes, cable runs and complex dimensional geometry. The jump from Claude Opus 5 at 43.1 percent to Opus 5.5 at 53.1 percent happened within a couple of months, which suggests fast progress, but 53.1 percent is far below what anyone could safely price a bid on. The suite's authors expect foundation models may largely close this gap within about six months, and on that basis they advise against paying for dedicated AI take-off software today. That timing is a forecast, not a result.

Record retrieval is effectively solved

A third objective test asked models to find facts across 286 messy project files using 146 questions about contracts, registers, site records and correspondence, for example how many requests for information remain open and which register proves it. Gemini 3.7 Flash, GPT-5.6 Sol and Claude Opus 5 all scored 97.7 percent, Claude Sonnet 5 scored 97.5 percent, and even the compact GPT-5 Nano reached 95.2 percent. Older and open-weight models lagged, with Mistral Large 2512 at 87.7 percent and Mistral Medium 3.1 at 83.9 percent. Because recent frontier models now reach 100 percent, this benchmark has been retired from the public leaderboard.

Construction management: judgment under messy conditions

The subjective benchmark covers six workflows: estimating, programming, progress tracking, procurement, contract administration and cost reporting. Each scenario mimics a colleague's request backed by conflicting documents. The dataset holds 612 documents, and seven models were compared.

Thick bid package with cost schedule under a magnifier, eight-door quote beside sixteen openings, crane circled

▲ Seeded errors in the warehouse tender package

For the tender task, models received a 31-file bid package for a warehouse with an attached office. It included architectural and structural drawings, scopes of work, contract conditions, addenda emails, seven supplier quotes with exclusions and an internal cost library. Each model had to produce an Excel estimate, a PDF tender submission and an assumptions register. Seeded errors included a concrete rate typed as $39.60 per cubic meter instead of $396 per cubic meter, a door quote covering supply only for eight doors when sixteen installed doors were specified, and drawing registers referencing sheets that were not in the package.

Grading used three blinded AI review sessions with model names hidden, taking the middle grade, for up to 80 points, plus 20 points of automated checks.

Specific tasks showed where models separate:

  • Procurement: a steel subcontractor quoted one crew over 16 weeks with craneage included, but the fast-track schedule required 10 weeks. That meant two crews, an extra $140,000, and a separate mobile crane hire of roughly $434,000 that the revised rate omitted. Claude Opus 5.5 caught the missing craneage, added the $433,819 crane cost to its comparison and scored 3.5 of 4 on budget. Claude Haiku 5.5 accepted the compressed dates without the crane cost and scored 1 of 4.
  • Estimating: drawings showed a 13,270 square meter warehouse slab overlapping 1,793 square meters of office foundations. Opus 5.5 recognized the overlap and reduced the slab to 11,670 square meters; Haiku 5.5 double-counted the shared area.
  • Programming: Claude Sonnet 5.5 projected a completion date of 3 November 2027 based strictly on ordered resources and presented acceleration as an unapproved option. Opus 5.5 adopted an unapproved accelerated date of 13 September 2027. The larger model does not always win.
  • Progress tracking: Haiku 5.5 accepted a site diary pour date of 23 February, while Sonnet 5.5 checked delivery dockets and test registers to confirm 24 February.
  • Payment claims: Opus 5.5 recommended a provisional 57 percent assessment after verification, where Haiku 5.5 recommended an unverified 70 percent.

Overall scores out of 100, which are quality grades to be read relatively rather than as percentages correct:

Model and harness Score
Claude Opus 5.5 in Claude Cowork 65.9
Claude Fable 5.1 in Claude Cowork 63.4
Claude Sonnet 5.5 in Claude Cowork 63.2
GPT-6.1 Sol in ChatGPT Work 60.5
GPT-6 Astra in ChatGPT Work 56.8
Gemini 4 Argon in the Gemini app 44.1
Claude Haiku 5.5 in Claude Cowork 35.2

Claude Fable 5.1 landing below Opus 5.5 on several management tasks ran against expectations.

What this means and what to do

The ranking from industry-specific tests differs from what general leaderboards suggest, and the method is the transferable part. For construction teams, the practical takeaways are:

  • Choose between Claude and ChatGPT. Claude Opus 5.5 currently shows the deepest reasoning over complex document folders, while ChatGPT provides roughly twice the usage volume per subscription dollar. Gemini and current open-source models appear uncompetitive for management workflows.
  • Use AI for drawing questions and record retrieval with confidence, but treat every measured quantity as a draft to be checked.
  • Do not buy specialized take-off AI tools yet, given the roughly 53 percent ceiling.
  • Prompt explicitly: ask models to check whether acceleration quotes include cranes and hoists, to look for spatial overlaps in slab and foundation quantities, to prefer confirmed subcontracts over unapproved accelerated schedules, and to cross-reference site diaries with delivery dockets and test registers.

For readers in any other field, the same approach applies. Test the actual app you will use, not the raw API. Build tasks from your own messy documents, plant realistic traps, and rerun close results before trusting a 1 or 2 point margin. Because Anthropic and OpenAI leapfrog each other with each release, settling on one ecosystem and integrating it into daily work is likely to serve better than chasing every new leaderboard.