Part 5 of 6 in a series about Harness Engineering. Previous articles:
Part 1: Harness Engineering
Part 2: The Model Should Never Touch the Tool
Part 3: Never Let an Agent Grade Its Own Homework
Part 4: What the Agent Forgot
Return to the incident that opened this series. For roughly six weeks in early 2026, engineers using Claude Code were convinced the model had gotten worse. They said so, repeatedly and publicly. The prevailing response was that they were imagining it — that this was the familiar pattern of users acclimating to a tool and mistaking their own rising expectations for a decline in quality.
They were not imagining it. When Anthropic published its postmortem in April, it identified three real changes in the layer around the model, and the model weights were never touched at all. The behavior of the system had changed, measurably, and the people closest to the telemetry took six weeks to see what the users saw immediately.
That gap is the subject of this article. If a company with that level of engineering depth and direct access to its own infrastructure can take six weeks to detect a harness-level regression, the question for the rest of us is not whether it can happen. It is how long it would take us to notice.
There are two ways an agent's behavior changes without anyone deciding it should.
The first is provider-side. A model gets updated, a default changes, a serving configuration is adjusted — and your outputs shift with no commit on your side. You did not do anything, which is precisely what makes it hard to detect: your instinct is to look at what changed recently in your repository, and the answer is nothing.
The second is inside your own harness, and it is more common. A prompt gets edited to fix one behavior and shifts three others. A retrieval threshold is tuned. A timeout is shortened. A verbosity limit gets added. Each change is small, locally justified, and invisible in aggregate — which is exactly what the Claude Code postmortem describes: three individually reasonable decisions compounding into a visible quality problem.
Nobody made a bad decision. The system got worse anyway. That is what an unobservable system feels like from the inside.
The good news is that this problem is further along than most of the rest of the harness. The OpenTelemetry GenAI semantic conventions, backed by the CNCF, now define a standard schema for AI telemetry: prompts, responses, token usage, tool and agent calls, model and provider metadata. Major observability vendors support them natively. If you already run OpenTelemetry — and most enterprises do — this is an extension of existing practice rather than a new platform to buy.
At minimum, a trace should let you reconstruct any single run end to end:
One honest caveat about the standard: the GenAI conventions cover mechanics — attributes, token usage, latency — but they do not yet cover output evaluation, safety scoring, or quality assessment. Embedding evaluation results into spans is still in discussion in the working group. So the plumbing is standardized and the judgment is not, which means the quality layer is still something you build.
Ask an enterprise AI team how their agents are performing and you will usually get activity metrics. Tokens consumed. Tasks attempted. Sessions run. Requests served. These measure motion, and motion is not the thing anyone is buying.
The metric worth organizing around is cost per accepted change: total cost of everything it took — including the failed attempts, the retries, the verification passes, and the human review time — divided by the work that was actually accepted and shipped.
It is a demanding number, which is the point. It exposes the retry loop that burns forty attempts to land one change. It exposes the agent with an impressive completion rate whose output a human quietly rewrites every time. It makes the trade-off between a cheaper model and a higher rejection rate legible instead of theoretical. And it is the single number that lets a team running the same models as everyone else demonstrate whether their harness is worth what it cost to build.
It also reframes the conversation with finance in a useful way. Token spend is a number that only ever goes up and has no denominator. Cost per accepted change has a denominator, and a denominator turns a cost center into an efficiency argument.
Once you can see cost per step, the largest and most obvious inefficiency becomes hard to ignore. Most harnesses send every subtask to the strongest model available, because that is the default and because nobody wants to be the person who chose the cheap model on the step that broke.
The industry numbers here are consistent enough to plan against. On a typical enterprise workload — summarization, classification, straightforward code edits, structured extraction — the fast tier handles somewhere between half and seventy percent of requests at parity with the strong tier. Published estimates for what a router saves land around thirty-eight percent of inference spend on typical 2026 workloads, and stacking static tiering with prompt caching, reasoning-effort caps, and batch APIs is reported to reach fifty to seventy percent.
The important detail is that most of that saving comes from the boring interventions, not from a sophisticated learned router. Static tiering by task type, caching, and capping reasoning effort where reasoning is not needed will get you most of the way. Build the simple version first.
A workable default assignment:
That last line matters, and it connects directly to the previous article in this series. Routing is how you fund verification. The savings from sending triage to a cheap model are what pay for checking the output with an expensive one, and a harness that gets both decisions right is cheaper and more reliable than one that gets neither.
The failure mode here is not technical. It is that teams instrument thoroughly, build dashboards, and then never look at them, because nothing is asking anyone to.
Traces are only useful if something acts on them. That means alerting on the things that actually indicate trouble — verifier rejection rate moving, cost per accepted change trending the wrong way, latency distribution shifting, a model version changing underneath you — rather than on raw volume, which tells you only that people are using the thing. And it means someone owns the review. A weekly look at rejection reasons is worth more than a real-time dashboard nobody has open.
There is also a privacy dimension that is easy to get wrong in a hurry. Full traces contain prompts and outputs, which means they contain whatever your users and your documents contain. The data guardrails from Part 2 apply to your telemetry pipeline as much as to your agent's output — arguably more, because logs are retained, replicated, and shared with vendors in ways live output is not.
Next, and last in this series: how a harness stops repeating its mistakes, and a practical order of operations for building all of this.
This is the fifth article in a series on harness engineering. Part 4 covered context and memory. Part 6 covers feedback loops and where to start.