DeepSeek V4.1 Flash costs roughly one tenth of what rival APIs charge, and its public benchmark scores sit close to Claude Opus 5 and GPT-5.6 Sol. But when the model was run against real construction work, the gap to the current frontier models was wide. On a set of 169 verified questions about tender drawings it scored 62.7%, against 93.3% for Claude Sonnet 5.5. On a plumbing quantity takeoff it reached 20.7%, against 42.7% for Claude Opus 5.5. And in a cost-reporting exercise it forecast a $1.03 million profit on a job that was actually heading for a $250,000 loss. The results suggest that, for now, the savings on tokens do not cover the risk of acting on the output.

Why real tasks instead of leaderboard scores

DeepSeek is an open-weight model family from China, meaning its weights can be downloaded from hubs such as Hugging Face and run on private hardware. The V4.1 Flash release notes claim performance that rivals Claude Opus 5 and GPT-5.6 Sol on standard tests such as Terminal-Bench 3.0, DeepSeek-SWE 1.1, CyberGym and Automation-Bench. Those are useful signals, but they do not tell a contractor whether a model can count floor drains on a plumbing sheet or spot a trick clause in a tender.

The evaluation described here therefore used a construction-specific benchmark instead. It has three parts: an objective drawing test built from 9 real tender drawing sets and 169 questions with checked answers; a quantity takeoff test built on an 11-sheet Whole Foods plumbing set with 82 scored item quantities; and a subjective construction management test covering six job workflows scored against a rubric. The same benchmark database holds results for 38 raw models and 22 harnesses, so DeepSeek's numbers can be placed next to Claude Sonnet 5.5, Claude Opus 5.5, GPT-6 Astra and GPT-6.1 Sol.

The setup: model, harness and price

DeepSeek was tested inside DeepSeek Harness, its free desktop application. A harness is the software environment that wraps a model into an agent: it manages chat sessions, file access, tools and automation, in the way Claude Code or ChatGPT Codex do. DeepSeek Harness is distributed as a free download from DeepSeek's official website and runs on Windows among other operating systems. It is also model-agnostic: in the Models tab of its settings you can register a DeepSeek API key, or keys from Anthropic, OpenAI or a routing gateway such as OpenRouter, and switch between them.

A few setup details mattered for the construction tests:

  • Model choice. DeepSeek V4.1 Flash was selected over DeepSeek V4-Pro because V4-Pro lacks native visual understanding and cannot read plan drawings. Unlike Claude or ChatGPT, where the core models share similar multimodal features, DeepSeek variants differ in what they can do, so the specific model's capabilities should be checked before processing technical documents.
  • Workspaces. Each workspace maps to a local folder chosen through a standard file dialog, and sessions and artifacts are grouped under it in the sidebar. The tests were run from a desktop folder, mirroring the desktop co-work setups used for Claude and ChatGPT.
  • Permissions. A prompt-bar menu offers Read Only, Workspace Write and Full access. Workspace Write was used so the agent could edit project files without touching the wider system.
  • Plugins. Built-in plugins control web search, shell execution limits, agent loop dispatch and a Subagent feature that lets a main chat spawn child agents, with sliders for recursion depth and parallel limits. Keeping those limits tight helps avoid runaway token spend.

Running the open weights locally was ruled out as impractical. On a standard laptop, inference is too slow for daily work unless tens of thousands of dollars go into specialized accelerator hardware.

Pay-per-use versus a flat subscription

The headline attraction is price. DeepSeek V4.1 Flash lists at $0.30 per million input tokens on a cache miss and $1.20 per million output tokens in standard hours, about one tenth of competing list prices. Cache hits bring input down further, to $0.003 per million tokens off-peak and $0.015 on-peak, and off-peak output drops to $0.60 per million tokens.

There is no flat consumer subscription, though. Every call is metered. By contrast, a $200 per month Claude or ChatGPT plan delivers subsidized token volume that, if fully used, could be worth an estimated $10,000 in raw API compute. For heavy users a flat plan may therefore be the better value, and the repeated $10 credit top-ups that metered billing requires create their own friction by keeping the running cost constantly in view. Open-weight competition still matters here: it pressures proprietary vendors to hold frontier pricing down.

Reading drawings: strong on text, weak on measurement

The drawing test covers architectural, structural, electrical, mechanical, plumbing and civil plans. DeepSeek's overall result and its breakdown by task type are shown below alongside the frontier models.

Model Overall drawing score
Claude Sonnet 5.5 93.3%
Claude Opus 5.5 92.5%
GPT-6 Astra 86.6%
GPT-6.1 Sol 85.8%
DeepSeek V4.1 Flash 62.7%
Task type DeepSeek V4.1 Flash Best frontier result
Reading written information 82.8% 96.5% (Claude Sonnet 5.5 and Opus 5.5)
Counting items 55.9% 91.2% (Claude Sonnet 5.5)
Measuring from the drawing 9.1% 81.8% (Claude Sonnet 5.5 and GPT-6 models)

The pattern is clear. When the answer is printed on the sheet as text, DeepSeek does reasonably well. When the answer requires counting symbols it slips, and when it requires scaling a physical dimension off the drawing it almost always fails. Two specific misses illustrate the behavior. On an electrical drawing numbered 402, asked whether the DB-GB distribution board was essential or non-essential, DeepSeek replied that the material provided did not state this; the checked answer was non-essential. On drawing 201 of an office plan it declined to count first-floor small power outlets, where the verified figure was 6.

The test run also hit stability problems. DeepSeek halted mid-prompt several times and had to be restarted manually, something that rarely happens with Claude or ChatGPT on the same workloads.

Plumbing plan with floor drain symbols on two pipe networks, beside two trays of tokens sorted in the wrong proportion

▲ Drains counted but assigned to the wrong system

Plumbing takeoff: the right total, the wrong system

A quantity takeoff is the process of counting and measuring every item on a drawing set so it can be priced. On the 11-sheet plumbing set with 82 scored quantities, the results were:

Model Takeoff accuracy
Claude Opus 5.5 42.7%
GPT-6 Astra 37.8%
GPT-6.1 Sol 35.4%
Claude Sonnet 5.5 34.2%
DeepSeek V4.1 Flash 20.7%

No model is close to production-ready here; even the top score is under half. But DeepSeek's failure mode is instructive. On sheet P100, the task was to count HD-1 hub drains and assign them to the sanitary waste or grease waste system. The verified answer was 64 sanitary and 2 grease. DeepSeek reported 6 sanitary and 55 grease. The total was roughly right; the allocation was almost perfectly inverted. In a priced estimate that kind of error would not show up as an obvious gap; it would show up as the wrong pipework, fittings and connections being priced for the wrong system. On this evidence DeepSeek does not appear suitable for automated takeoff workflows.

Six construction management tasks

The subjective benchmark asks the model to perform six realistic workflows and scores each out of 100: preparing a tender estimate, building a baseline construction programme at award, planning progress and recovery, managing procurement, administering contracts and producing cost reports. The tender documents include planted traps, such as a clause banning car parking near the site, to see whether the estimator adds a transport allowance for workers.

Task DeepSeek V4.1 Flash Best competing score
Tender estimate 32.2 67.8 (Claude Opus 5.5)
Programme at award 30.3 67.2 (Claude Sonnet 5.5)
Progress and recovery 25.6 66.6 (GPT-6.1 Sol)
Procurement 36.8 58.4 (Claude Opus 5.5)
Contract administration 27.6 78.8 (Claude Opus 5.5)
Cost reporting 24.0 71.8 (Claude Opus 5.5)

DeepSeek finished last in every category. The best average across the six tasks in the database was 65.9%, held by Claude Opus 5.5. Two outputs stand out. Asked for a construction schedule, DeepSeek produced 29 tasks with zero dependency links, just a column of disconnected dates in a spreadsheet, even though activities such as steel erection, roof sheeting and SFRC slab pours have to follow one another. And in the cost report, it forecast a profit of $1.03 million at a 6.6% margin on a job whose ground-truth assessment showed a $250,000 forecast loss.

That last result frames the cost argument. Saving $20 in tokens on a job that costs $10 to run instead of $30 is irrelevant if the report that comes out of it points the project in the wrong financial direction.

Creator mode and the takeoff plugin that did not materialize

DeepSeek Harness has a second mode beyond Standard called Creator mode, toggled in the workspace header. It exposes the harness configuration and lets you instruct the agent, in plain chat, to write and install custom UI plugins. It is comparable in intent to Claude Mods, the recently introduced feature that lets users alter Claude's workspace and add extensions; DeepSeek calls the equivalent components plugins.

To test it fairly, a construction takeoff plugin was attempted: an interactive interface where a user could see takeoff item overlays drawn on top of the sheets and adjust them. After two hours of prompting and iteration, the result was a basic tabular schedule with item descriptions, units, quantities, rates and amounts, a CSV import, a manual waste percentage field and a Run take-off button. It had no visual sheet overlay at all and did not meet the design brief. The attempt was abandoned. Creator mode does not appear beginner-friendly, and building sophisticated visual tools through text prompts proved frustrating.

Two monitors: a drawing with colored overlay markers on the left, a plain empty spreadsheet grid on the right

▲ Intended visual takeoff tool versus the table produced

A skeptical take on harness engineering

The experiment feeds into a broader judgment about the current enthusiasm for "harness engineering," the idea that end users should be re-architecting the agent environments they work in. Configurable knobs such as subagent recursion depth, parallel subagent limits, agent loop dispatching and shell constraints are interesting, but the view here is that practitioners get better results from a conventional setup: well-defined skills, standard plugins, clearly structured project folders and verified test datasets. Platform architecture may be best left to frontier labs such as Anthropic and OpenAI, while construction professionals spend their time on workflows, testing and the quality of their data.

One caveat deserves weight. DeepSeek Harness is free and new, and comparing it head-on with mature paid products is somewhat harsh. The price advantage is real, and open-weight competition benefits everyone by restraining frontier pricing.

What construction professionals should take from this

  • Judge models on your own tasks. Public benchmark parity did not translate into construction competence. A small set of verified questions from your own drawings is a better selection tool than a leaderboard.
  • Treat text extraction and spatial work differently. DeepSeek was usable at 82.8% for reading printed information but scored 9.1% on measurement. If a workflow depends on scaling or counting, the low-cost option currently does not hold up.
  • Check system allocation, not just totals. The hub drain example shows a count can be nearly right in aggregate and almost entirely wrong by system. Verify every takeoff line against the drawing before pricing it.
  • Open every generated schedule and look for links. A programme with dates but no predecessors or successors is not a programme.
  • Weigh token cost against decision risk. A cheaper run is not a saving if the cost report it produces inverts a loss into a profit.
  • Spend time on workflows, not harness rebuilds. Pick the model variant with the capabilities you need, set Workspace Write permissions, cap subagent depth and concurrency, and organize project folders cleanly. Leave plugin and interface construction to the platform builders unless you have a clear, tested reason not to.

DeepSeek V4.1 Flash is cheap, open and surrounded by a flexible free harness. For construction drawing interpretation, quantity takeoff and management reporting, though, the current frontier models from Anthropic and OpenAI remain substantially more accurate, and none of them, including the leaders, has yet earned unsupervised trust on takeoff.