Frontier language models now write game and web application code that looks clean on inspection and renders attractive screenshots, which makes the traditional way of grading that code almost useless. Arena, the organization behind Code Arena and Agent Arena, has published research on a different approach it calls Interaction-as-a-Judge: a computer-use agent, an AI system that clicks, types and navigates software the way a person would, actually plays the generated game or uses the generated website, and a judge model then evaluates the recorded session instead of the raw code. In Arena's experiments, this raised agreement with human preferences for every judge tested, caught far more broken features than agents that only read code, improved inference-time repair loops by up to 20%, and made it possible to train a code generator with reinforcement learning without ever checking the source.
Why reading the code no longer works
The problem is easiest to see with a real prompt from Code Arena, where users ask two competing models to build the same thing and vote on the winner. One user asked for a Minecraft clone. The request implies a handful of functional requirements: moving around, rotating the camera, breaking blocks and placing items.
Side by side, the two models' JavaScript and WebGL code looked nearly interchangeable. Both implemented the same yaw and pitch math for camera rotation and similar mining-progress checks. Neither contained obvious syntax or logic mistakes. Static screenshots did not help either: both rendered convincing voxel landscapes with trees, water and sky.
Only playing the two games exposed the difference. In the first version, movement worked, zombies could be attacked, flowers could be placed and blocks broke on click. In the second, movement worked at first, but rotating the camera produced a disorienting third-person view with no character model, and clicking on blocks did nothing. The human vote on Code Arena matched what a short play session revealed, and nothing in the code or screenshots would have predicted it.
This is the core argument behind the research: when generated code is uniformly competent on the surface, the bugs live in runtime state and interaction loops that a reader cannot simulate. Humans find them by playing, but paying people to play every generated application removes the cost advantage that made automated LLM judging attractive in the first place.
How the pipeline works
Interaction-as-a-Judge keeps the automated judge but inserts an agent between the artifact and the verdict.

▲ The Interaction-as-a-Judge evaluation pipeline
- The generated code is built into a running application in an isolated sandbox.
- A computer-use agent interacts with it, seeing only screenshots and issuing primitive inputs such as mouse clicks at pixel coordinates, drags, key presses and waits. The team chose pixel-level actions over DOM or accessibility-tree inspection for simplicity and because they work on any application.
- The session is recorded as an interaction trace: the video frames, the sequence of actions and the agent's step-by-step reasoning text.
- A judge consumes the trace, optionally alongside the code, and outputs either a pairwise preference or a pass/fail verdict on a specific requirement.
Every run is capped at 30 interaction steps. For real-time games, a custom harness pauses the game between actions and advances it by 500 milliseconds after each one, so slower models are not penalized for inference latency.
Two kinds of judges were tested. The first is an off-the-shelf LLM prompted to compare two traces; to counter the well-documented tendency of LLM judges to favor whichever option appears first, every comparison is run in both AB and BA order and the scores are averaged. The second is a trained reward model built on Qwen-3.5-9B, with a vision encoder for the frames, text embeddings for code, actions and reasoning, and a linear head that produces a scalar score. Pairs are trained with a Bradley-Terry loss, a standard formulation for learning from "A beat B" comparisons. The vision encoder was kept frozen because unfreezing it added instability without measurable gains.
Three ways to drive the agent
Interaction quality costs money, so the research compares three paradigms.
| Method | How it works | Cost | Weakness |
|---|---|---|---|
| Iterative | Observe screen, reason, act, repeat on every step | Highest; a model call per step | Expensive on long traces |
| One-shot | Read code and first screenshot, plan all actions up front, execute linearly | Low; one model call | Cannot recover from an early miss |
| Heuristic | Random or rule-based clicks on interactive elements, no model | Zero inference cost | Breaks down in games with continuous action spaces |
Test runs on a mock food-delivery app called Zazu make the trade-offs concrete. An iterative GPT-5.6 Luna agent logged in, browsed categories, picked dishes, filled in checkout details and completed an order. A one-shot agent missed a button early in its plan, ended up on an empty cart page, and spent the rest of its budget idling through pre-scheduled waits. A heuristic agent clicked randomly, accidentally switched the interface to Spanish and wandered through account forms without ever reaching the main ordering flow. Even so, that random run surfaced far more of the application's dynamic states than a single screenshot could, which is why heuristic interaction still beats static baselines on ordinary websites.
Two new benchmarks
The work introduces two datasets.
ArenaPreferences covers subjective quality: which app looks more polished or feels more engaging. It contains 6,858 pairwise battles collected from real Code Arena users starting in July, each with the prompt, both generated apps, the human vote and automated interaction traces. The seven categories are Gaming (1,288), Brand & Marketing (1,246), Content Creation Tools (1,020), Data & Analytics (949), Simulations (904), Reference-Based Design (753) and Consumer Product (698).
ArenaCorrectness covers objective requirements. It holds 412 requirements across 222 game versions mined from 67 real developer sessions on Agent Arena, an in-browser sandbox where developers iteratively prompt agents to build and modify projects. Because developers do not label correctness, the team mined their complaints. In one Geometry Dash-style platformer, a user reported that the second platform always killed the player; an LLM turned that into the requirement "the second platform must not kill the player," human annotators verified it, and the broken version was paired with the later fixed version where the spike had been removed. Since people mostly comment when something fails, 323 requirements describe broken states and 89 describe working ones. The dataset supports two tasks: requirement verification (given a requirement, does this version pass?) and requirement discovery (find the broken requirements without being told what to look for).
Results: preference judging
Across 6,858 battles with 30 actions per run, richer interaction consistently improved agreement with human votes.
| Judge | Code only | Code + screenshot | Heuristic | One-shot | Iterative |
|---|---|---|---|---|---|
| Gemini 3 Flash | 65.6% | 67.6% | 68.4% | 68.3% | 69.1% |
| GPT-5.6 Luna | 68.3% | 66.6% | 67.2% | 67.0% | 67.9% |
Gemini 3 Flash follows the expected pattern. GPT-5.6 Luna shows an anomaly the research describes as one of its strangest findings: adding a screenshot to the code lowered accuracy, and interaction traces only partly recovered it. The interpretation offered is that frontier models from OpenAI and Anthropic are trained with heavy emphasis on code, so mapping a mental model of the code onto a rendered visual state remains a weak point. The caveat is that this code-first advantage appeared only on preference judging, not on correctness.
The trained Qwen-3.5-9B reward model beat both LLM judges. On full codebases, often exceeding 100,000 characters per app, it moved from 69.7% with code alone to 70.8% with iterative traces; in reward-model benchmarks, a one-point absolute gain is considered a large margin. When the code context was cut down to just App.tsx, the main React component file, code-only accuracy dropped to 68.2% while the interaction-based score held at 70.2%.
The most consequential result is the no-code setting. A reward model given only iterative interaction traces and no source at all reached 69.0%, above every frontier LLM judge working from code and screenshots and close to the 69.7% full-code baseline. That opens evaluation and training for software whose source is unavailable, with commercial games like Call of Duty cited as the kind of target this unlocks.
Two more findings matter for practitioners. First, a reward model trained on cheap heuristic traces and then applied to iterative traces at inference time still improved, from 67.9% to 68.9% overall and from 70.8% to 73.2% in gaming, with reference-based design the only category to regress (70.1% to 68.4%). Second, the gains concentrate where interactivity is high: with Gemini 3 Flash, iterative interaction beat code-plus-screenshot by 2.8 points on simulations and 2.7 on games, but by -0.3 on data applications.
Results: finding broken features
On ArenaCorrectness, the comparison was against a code agent that navigates the repository statically without running anything. Both approaches verified working requirements about equally well (87.6% for Luna, 86.5% for Terra, and 91.0% versus 92.1% for Sol). The gap appeared on broken requirements.
| Model | Code Agent-as-a-Judge | Interaction-as-a-Judge |
|---|---|---|
| GPT-5.6 Luna | 51.6% | 65.2% |
| GPT-5.6 Terra | 59.6% | 68.0% |
| GPT-5.6 Sol | 63.0% | 71.1% |

▲ Broken and fixed versions of a platformer requirement
The spike example explains why. Whether a hazard placed just past a bounce pad is actually unavoidable depends on physics at runtime; reading the level data rarely settles it, while a single play attempt does. Accuracy also scaled with model capability across all three tiers, which the research takes as a sign that later generations should verify better still.
Requirement discovery stayed hard. Asked to diagnose a broken game with no checklist and generate candidate requirements, recall peaked at 28% for Sol with interaction (25% with the code agent), with Luna at 17% versus 14% and Terra at 15% versus 17%. The research presents this as an open problem.
Using the judge to build better software
Beyond grading, the team tested whether interaction feedback improves generation.
In an inference-time repair loop, GPT-5.6 Luna acted as both the repair agent and its own verifier, with the stronger GPT-5.6 Sol serving as the external referee. Over five iterations, the interaction-verified loop fixed 63.5% of requirements on the first pass and 75.5% by the fifth, against 49.8% rising to 55.4% for the code-agent loop when Sol judged by interaction. When Sol judged by code instead, the numbers were 69.7% to 77.4% versus 57.9% to 63.8%, so the lead held regardless of how the referee measured.
The loop also curbed over-claiming. Luna believed it had fixed 81.7% of issues under interaction self-verification and 75.9% under code self-verification. Against Sol's interaction check, the code-agent loop over-claimed by 20.4 points while the interaction loop over-claimed by 6.2; against Sol's code check, the gaps were 12.1 and 4.3 points. Testing real behavior appears to make it much harder for a model to convince itself a bug is gone.
For training-time use, the team ran reinforcement learning on a Qwen 2.5 9B generator using DAPO (Direct Alignment and Preference Optimization) updates. Each rollout was built into a website, exercised by a heuristic agent (full visual agents were too slow and costly at training scale, and the experiments were limited to non-game websites), scored by the interaction reward model, and fed back as a policy update. Because the small generator often produced unbuildable code outside the reward model's training distribution, a simple negative build penalty was added rather than retraining the reward model.
Over 35 training steps, both a code reward model and the interaction reward model learned steadily, and gains transferred in both directions. The structural outcomes differed sharply:
- The code reward model showed strong length bias; generated apps bloated past 60,000 characters by step 35.
- The interaction reward model drove mean code length from about 45,000 characters to under 25,000, producing 35.7% less code overall.
- Build failure rates fell 7.2 percentage points relative to the code reward model.
- On Arena's internal auto-rubric evaluation, the interaction-trained policy gained 6.72 points of win rate over the base model, compared with 6.14 for the code-trained policy, which Arena's calibration equates to roughly a 60 to 70 point leaderboard jump.
What to take from this
The case the research makes is straightforward: when generated code looks uniformly good, the only reliable way to tell working software from broken software is to run it and use it, and computer-use agents are now capable enough to do that at scale. The evidence from this single body of experiments supports several practical moves for teams that evaluate or train coding agents:
- Stop relying on code review and screenshot diffs to grade interactive applications; run them in a sandbox and record the session.
- Use iterative agents where budget allows, but collect cheap heuristic traces for training, since reward models trained on weak exploration transferred to stronger agents at inference time.
- Restrict random exploration to interactive elements such as buttons and inputs, and add a pause-and-advance harness for real-time games.
- Mine user bug complaints from development sessions into paired broken and fixed test cases to build correctness benchmarks without manual labeling.
- Prefer interaction-based verification inside self-repair loops to limit false claims of success.
- Expect the approach to improve on its own as computer-use agents get better, and expect the biggest payoff in games and simulations rather than static data pages.
The limits are equally clear. Interaction adds latency that becomes expensive across tens of thousands of battles, full visual agents remain too slow for large RL rollouts, and discovering unknown bugs without a checklist still tops out at 28% recall. Even so, decoupling the reward signal from source code is the part most likely to matter long term, because it lets models learn from software that nobody can hand them the code for.


