Swap the coding harness but keep the model, and the bug-fixing score barely moves. That is the main finding from a controlled test in which a single local model, Qwen 3.8 27B, was wired into four different coding agents: Claude Code, OpenCode, Pi and the new DeepSeek Harness. Each combination attacked the same 68 real GitHub issues from the SWE-rebench suite under a 30-minute limit. Claude Code fixed 27, OpenCode and DeepSeek Harness fixed 24 each, and Pi fixed 22. The spread is small enough that it may come down to variance rather than any real difference in agent logic. What did differ, sharply, was how much work it took to get each harness running correctly against a local server, and how efficiently each one used the GPU once it was running.

What was tested and how

A harness is the software layer that turns a language model into an autonomous agent. It supplies the system prompt, the tools for editing files and running shell commands, and the loop that keeps the model working until the task is done. The model stays the same; the harness decides what the model sees and what it can do.

The setup used Unsloth's Qwen 3.8 27B UD Q4_K_XL GGUF file served through llama.cpp's llama-server. Versions were pinned: Claude Code 2.1.300, OpenCode 1.14.43, Pi 1.0.4, DeepSeek Harness 0.2.0-rc.2 and llama.cpp build b1-8ed274e. The llama-server configuration was also fixed in advance: a 200,000-token context, flash attention on, a single slot (-np 1), q4_0 quantized key and value caches, batch sizes of 4096 and 512, draft MTP speculative decoding with a maximum of two draft tokens, and the --jinja flag for chat templating.

Every harness received the same input: the issue description from SWE-rebench plus a short instruction to fix the issue without human help. Tasks came from Python repositories including pandas, scikit-learn, loguru and sqlfluff, and were graded automatically against hidden test suites the agent never saw.

Running 68 tasks across four harnesses means 272 runs, each allowed up to 30 minutes. Doing that on a single RTX 3090 Ti was impractical, so the runs executed concurrently on a rented cloud machine with eight NVIDIA RTX A6000 GPUs, priced at around $2.19 per hour. The original plan called for 156 bugs; compute cost cut it to 68.

The results, and why they look like a tie

Harness Bugs fixed (of 68)
Claude Code 27
OpenCode 24
DeepSeek Harness 24
Pi 22

The overlap tells more than the totals. Comparing Claude Code and DeepSeek Harness bug by bug, both fixed the same 22 issues and both failed the same 39. Claude Code picked up five bugs DeepSeek Harness missed; DeepSeek Harness picked up two that Claude Code missed. A pairwise gap analysis with pre-registered decision rules ruled out large differences, defined as more than 14 of the 68 bugs, in five of the six harness pairings. Small differences remained statistically inconclusive.

Puzzle pieces sorted into five piles, with the two tallest piles at the ends and four arms pointing at them

▲ Bugs solved by none versus all four harnesses

Across all four harnesses, 33 bugs were solved by none and 16 were solved by all. Only seven bugs were solved by exactly one harness, ten by two, and two by three. In other words, 49 of the 68 bugs produced the same outcome regardless of which agent was driving. The difficulty of the individual bug appears to decide success far more than the harness does.

Two caveats attach to these numbers. Qwen 3.8 was released after many of the benchmark issues were created, so its training data may contain some of the solutions, which could compress differences between harnesses. And Claude Code is built around Anthropic's own Claude models, so its behavior with an open-weight Qwen model may not reflect how it performs with Claude.

Four setup bugs that had to be fixed first

Before any benchmark could run, four problems with local execution needed work. These, more than the scores, are what a developer building a local setup is likely to hit.

Claude Code could not start

Pointed directly at llama-server, Claude Code crashed immediately with a Jinja template exception: "System message must be at the beginning." Newer builds of Claude Code inject system messages mid-conversation to update environment details and token usage. Qwen 3.8's default chat template only accepts system messages at the top of the context.

The fix was a small Python proxy between Claude Code and llama-server that intercepts late system messages and rewrites them as user messages wrapped in <system-reminder> tags. The obvious alternative, folding those messages into the initial system prompt, was deliberately avoided. Changing the top of the conversation invalidates the server's KV cache prefix on every turn. The KV cache is the GPU's stored record of already-processed tokens; if the beginning of the prompt changes, none of it can be reused and the whole history must be processed again.

Claude Code's web search depends on Anthropic's cloud endpoints and does not work with a local model, but ordinary code search works through bash command fallbacks.

Every harness picks its own thinking level

Each harness quietly sets its own reasoning and sampling parameters on outgoing requests. Pi asked for medium reasoning effort, DeepSeek Harness for high, Claude Code for adaptive thinking at high effort, and OpenCode hardcoded top_p to 1.0, overriding Qwen's recommended 0.9. Older llama-server builds ignored reasoning effort tags; newer builds pass them through, so the model would have behaved differently under each tool. To make the comparison about agent logic only, the proxy was extended to strip temperature, top_p, top_k, reasoning, reasoning_effort, thinking and max_completion_tokens from every request, leaving the server's own configuration in charge.

One slot, one cache

With -np 1, llama-server keeps exactly one conversation in cache. Several harnesses make background side calls, for example asking the model to generate a title for the session. On a single-slot server, that side call evicts the main coding conversation, and the next turn reprocesses the entire history from scratch. A simple test showed the effect: 370 prompt tokens processed on the first call, 4 on a cached repeat, and back to 370 once a different prompt had taken the slot. Side calls were disabled with flags such as CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 and DISABLE_AUTOUPDATER=1 for Claude Code, and a static --title for OpenCode. On the multi-GPU rental, llama-server's RAM prompt cache also consumed up to 8 GB per instance and caused out-of-memory errors, so it was turned off with --cache-ram 0.

OpenCode's search tools

The test containers had no internet access. OpenCode tried to download a ripgrep release from GitHub and failed 107 times across the runs before falling back to shell grep. The fallback cost one to two extra model calls per run, and it meant OpenCode completed the entire benchmark with its dedicated search tools broken. Preinstalling ripgrep avoids this.

Context limits also vary. OpenCode, Pi and DeepSeek Harness read the server's context size and compact the conversation before it overflows. Claude Code assumes 200,000 tokens for an unrecognized model, which happened to match this server. Anyone running a smaller context must set the maximum context tokens environment variable by hand.

Token efficiency: where the harnesses really differ

Before touching a single bug, each harness sends its instructions and tool definitions to the GPU. Pi's initial prompt was the leanest at 1,595 tokens, followed by DeepSeek Harness at 2,948, Claude Code at 3,662 and OpenCode at 8,665.

Over whole runs, the gap widened:

Harness Avg. prompt tokens processed per bug
Pi 44,000
Claude Code 53,000
OpenCode 60,000
DeepSeek Harness 86,000

DeepSeek Harness processed about 1.7 times as many prompt tokens as Pi, and more than Pi on 54 of the 68 bugs. The cause is a feature, not a bug: middle pruning. When a tool returns a long file read, DeepSeek Harness trims the middle of that earlier result to save context. In one sqlfluff run, a read of errors.py shrank from 8,257 to 5,159 characters with a "tool result middle pruned" marker. But editing a past message changes the conversation prefix, which breaks the server's KV cache match and forces full reprocessing.

A paper tape with a cut-out spliced section rewinding to the start inside a reading machine as a clock ticks

▲ Context pruning forcing full prompt reprocessing

Cache invalidation hit 21 of the 68 DeepSeek Harness runs and accounted for roughly one-third of all the tokens it processed.

On fast GPUs this mattered less than it sounds. About 97 percent of prompt tokens across the runs came straight from the cache, so prompt processing took only 1.5 to 2.3 minutes per bug, while generating replies took 15.2 to 15.4 minutes. On hardware with slower prompt processing, such as Apple Mac systems running without cache reuse, the same cache-busting behavior would likely inflate wall-clock time considerably. Speed comparisons across such different architectures should be read cautiously.

The time limit changes the winner

Simulating earlier cutoffs on the same runs reshuffled the leaderboard. At 20 minutes, OpenCode led with 22 fixes against 17 each for Claude Code and Pi and 16 for DeepSeek Harness. At 28 minutes, Claude Code, OpenCode and DeepSeek Harness were tied at 23. Claude Code's final lead arrived only in the last two minutes of the 30-minute window. These cutoffs are suggestive rather than conclusive, since runtimes shift with hardware.

One Pi run on sqlfluff issue #7617 shows how brittle a hard timeout can be. The model fixed the bug and confirmed all four tests passing at 29:57, then was killed at 30:00 while writing its summary. The hidden tests passed on the resulting diff, but the run was scored as a failure. Harnesses tend to spend their final four to five minutes rerunning verification after the correct edit is already in place, which is a reason to inspect the git diff during long runs rather than waiting for the agent to declare itself finished.

Choosing a harness for a local model

Based on this test, each harness has a distinct profile. Claude Code produced the highest raw pass count by a narrow and possibly noisy margin. Pi was the most token-efficient, sending the lightest load to the GPU. DeepSeek Harness was the only harness whose built-in search tools worked in this offline setup, at the cost of the heaviest token volume through cache invalidation. OpenCode ran without working search because ripgrep was unavailable. Qwen Code was not tested and is a candidate for a future comparison.

None of this proves that any harness is better than another. Fair agent evaluations carry hidden confounds, and every harness bakes in assumptions designed around cloud APIs rather than a local llama-server. The practical conclusion is that configuration beats selection. A setup checklist for pairing any harness with a local model:

  1. Put a proxy in front of the server that demotes late system messages to user messages, without touching the top of the prompt.
  2. Strip client-side sampling and reasoning parameters so the server's configuration applies to every tool.
  3. Disable side calls such as session titling so a single-slot server keeps the main conversation cached.
  4. Match the harness context limit to the inference server's actual limit.
  5. Preinstall ripgrep in the environment, especially when running offline.

Developers assembling a local AI coding environment can take this as the order of operations: get the plumbing right first, measure token use and cache behavior on their own hardware, and only then worry about which harness is nominally on top. On the evidence here, the bug in front of the agent will decide the outcome more often than the agent itself.