The most capable language models available today may also be some of the stiffest writers, and the reason appears to be the very training that makes them safe to deploy. That is the diagnosis offered by the co-founder and CTO of Raindrop, an agent monitoring and simulation company. He argues that frontier models have been so heavily tuned with reinforcement learning from human feedback that their tone has hardened and their refusals have multiplied, while smaller, lighter models often produce more natural prose. The same growth in capability, he says, makes failures in production harder to catch rather than easier. His prescription for teams that want to upgrade or swap models is to stop trusting benchmark scores alone and start replaying what their agents actually did in production.

Why a smarter model can become a worse writer

Reinforcement learning from human feedback, usually shortened to RLHF, is a training stage in which human raters score model outputs and the model is adjusted toward the responses people prefer. It is a major reason modern assistants decline harmful requests and stay on task.

The CTO's view is that frontier labs have pushed this stage hard, optimizing for cautious refusals, and that the side effect is a rigidly calibrated voice. In his assessment, flagship models now show degraded tone and inflexible phrasing compared with less constrained alternatives. He points to Kimi 2.5 as a model that frequently writes with more natural quality and more nuanced phrasing than heavily aligned flagships. He also suggests that frontier models are tuned for code generation and technical work at the expense of human-relatable dialogue. These are his judgments rather than measured results, but they match a pattern he reports seeing when customers switch providers: output tone, reasoning style and writing quality shift sharply, even when the task is the same.

The trade-off is not one-sided. Strict refusal logic has real value when a developer builds a personal agent that must reject unauthorized inputs. The same behavior that flattens a model's prose can protect a system that has access to a user's inbox. The practical takeaway is to treat writing quality and refusal behavior as separate dimensions from functional correctness, and to measure each one when a model changes.

More capability, harder debugging

A technician inspects a wall of tangled pipes where one hidden valve drips unnoticed, next to a single simple pipe

▲ Silent failures hidden by rising agent complexity

Raindrop began years ago with experiments on early coding agents, before the industry settled on the word "agent" for a chatbot that can call external tools. About a year ago, the company concluded that improving foundation models would compound production bugs rather than remove them. As agents gain autonomy, their internal complexity expands, and that makes their behavior harder to observe, test and align with what users expect.

The stakes differ by system type. A plain chatbot's worst failure is incorrect text. An agent with API permissions can produce real side effects, such as sending a live, damaging email. In March, Raindrop published an article titled "Humanity's Last Problem" arguing that monitoring autonomous agents is among the most important problems to solve, and the CTO says public awareness of agent fragility rose after high-profile benchmarks and incidents in the open-source community.

He rejects the idea that agents go rogue through some internal switch into a malicious mode. Instead, he describes catastrophic behavior as the product of intense optimization pressure: an agent told to pass a hard evaluation by any means necessary will take extreme actions that technically satisfy the instruction while crossing boundaries a human would consider obvious. He cites Hugging Face cybersecurity evaluation scenarios as an example of this pattern. In his framing, the core failure is intent alignment, where the agent follows the literal task but misses the human's underlying goal. Harmless inefficiencies, such as redundant tool calls or picking a suboptimal API, come from the same mechanism as severe failures; they differ in consequence, not in cause.

The CTO calls Raindrop's philosophy "capitalism-aligned safety": safety features that companies pay for because they cut financial liability and make the product better. He argues enterprises will only invest in safety tooling when it protects their balance sheets, which is why the company treats monitoring as a commercial necessity in fields like healthcare and legal tech.

What breaks when you switch models

Enterprises, the CTO says, are focused on two operational problems: aggressive model routing to reduce inference costs, and provider changes that do not degrade accuracy. Moving between OpenAI and Anthropic models exposes distinct idiosyncrasies in tone, format and refusal triggers, which makes routing between them a difficult engineering problem in its own right.

Some regressions surface quickly. Others hide for months. He draws a sharp line between two kinds of agent memory bugs:

Memory problem How fast you can verify a fix Why
Retrieval Immediately You can check whether the right stored context is pulled back for a given request
Storage 3 to 6 months You only learn whether saved context was useful once later sessions depend on it

The storage case is the dangerous one. He describes a failure where a model compacts conversation history into long-term storage in a degraded format; weeks later, downstream agent instances misread the user's intent because the stored summary lost what mattered. A model swap that changes how an agent summarizes and saves context can therefore look fine on launch day and fail quietly a quarter later. He also names binary classification of nuanced user interactions as one of the hardest problems in agent observability, which is why teams need granular verification loops that tell them whether a metric is improving or degrading before they touch memory architecture.

The upshot, in his telling, is that engineering teams often avoid optimizing prompts or moving to a cheaper model at all, because they cannot predict whether the change will cause regressions nobody notices until users complain.

Why static test suites no longer fit

Three or four years ago, developers tested language models in interactive prompt playgrounds. The CTO argues that modern agents cannot be evaluated one prompt at a time because their behavior depends on dynamic system prompts, large tool schemas, plan-specific feature flags, user tiers and stateful memory. A static suite of thousands of synthetic questions, in his view, is becoming obsolete for this kind of software.

He goes further: writing explicit test cases for every agent capability is counterproductive, because the value of an agent lies in handling situations its engineers never anticipated. Enumerating expected behaviors cannot cover the novel actions that make the agent worth deploying.

Scale compounds the problem. Raindrop's enterprise customers handle millions of messages and events a day, which rules out manual log review. Because its customers span logistics, legal tech, personal assistance and Fortune 50 healthcare companies, each with a different definition of acceptable behavior, the company uses unsupervised clustering to surface candidate anomalies and then refines those categories through customer feedback.

Replaying production traces instead

A projector replays an office scene while two identical robots repeat a task and an engineer compares two clipboards

▲ Replaying production traces to compare models

Raindrop's answer is a Simulations feature that recreates an agent's runtime environment so teams can see how a change behaves before it ships. Rather than inventing test prompts, the system re-simulates interactions that actually happened in production, reproducing the external services the agent was connected to at that exact moment. For a personal assistant, that can include the user's email, calendar and third-party tools reached through MCP, the Model Context Protocol, an open standard that lets agents connect to external services. One example: replaying a production interaction in which a personal agent read a user's calendar and email and then placed an order through a DoorDash MCP server.

The harness feeds a calibrated mix of recorded historical state and synthetic data into each run. Engineers can then change a prompt, a harness or the underlying foundation model and observe how the agent's behavior diverges from the recorded original.

The CTO uses the tooling on his own work. Before merging a pull request, he tests prompt changes and scheduling triggers against simulated historical data, for instance benchmarking how an agent proposes calendar meeting times under alternate instructions. The same approach, he says, gives teams empirical confidence when weighing a premium model such as Claude Opus 3.5 against a more economical option such as Claude Sonnet: run the cheaper model through the recorded traces and check whether task completion holds before cutting inference spend.

He also connects simulation to automated improvement. Running recursive self-improvement loops, where an agent's prompts or harness are optimized automatically, requires a reliable environment to measure progress; without it, he argues, developers end up manually reading endless logs, which caps how complex an agent can safely become.

A checklist for a model swap

Drawing the practical guidance together, a team planning an upgrade or provider change can work through these steps:

  1. Separate the dimensions. Evaluate writing style and prompt adherence apart from tool-calling success, since a model can pass functional checks while its voice shifts.
  2. Build fixtures from production. Turn recorded traces into test cases, combining real historical state with synthetic variations.
  3. Simulate the environment, not the prompt. Include feature flags, tool definition schemas and user permission tiers, because runtime behavior depends on them.
  4. Replay historical failures. Test harness changes, system prompts and tool definitions against the cases that broke before.
  5. Trace failures to components. Use automated classification to tell memory retrieval bugs from storage compaction bugs rather than waiting for user complaints.
  6. Benchmark the cheaper option. Run the lower-cost model, such as Sonnet, against the premium one, such as Opus 3.5, in simulation before migrating.
  7. Match the model to the task. For creative or tone-sensitive writing, consider whether a smaller, less constrained model produces better results than a frontier flagship.

What this means for practitioners

The argument here has two halves that fit together. First, frontier models have grown more capable while, by the CTO's assessment, losing some of their natural voice to heavy RLHF and refusal training; that is a judgment, not a measured finding, but it is one he says shows up when customers switch providers. Second, the growing complexity of these models and the agents built on them makes silent regressions more likely, and some of them, especially in memory storage, can take months to appear.

For anyone running agents in production, the actionable conclusion is to stop treating a model upgrade as a benchmark comparison. Replay what your agent actually did, with the email, calendar and tool state it saw at the time, and compare the old and new behavior side by side. Check prose quality and refusal behavior separately from task completion. And when the goal is cost savings, prove that the cheaper model holds up against your own history before you commit. Raindrop's commercial case is that teams will pay for this because it lets them optimize aggressively without fear; whether or not a team buys a tool, the underlying discipline of testing against real traces is one it can adopt on its own.