The Harness That Learns

Oct 26, 20266 mins read
JJ Segado | VP, AI Solutions, Argano
At Argano, JJ leads the strategy and delivery of AI solutions, combining deep technical expertise with delivery leadership to help clients turn complex business challenges into measurable outcomes. As VP of Delivery, he brings more than 20 years of experience leading technology teams and enterprise programs.

Part 6 of 6 in a series about Harness Engineering.  Previous articles: 
Part 1: Harness Engineering
Part 2: The Model Should Never Touch the Tool
Part 3: Never Let an Agent Grade Its Own Homework
Part 4: What the Agent Forgot
Part 5: You Cannot Manage What You Cannot Trace

Feedback loops, the maturity ladder, and where a harness stops helping 

Most agent deployments run exactly the same way on attempt five hundred as they did on attempt one. They make the same mistake, get corrected the same way, and arrive at the next session with no record that any of it happened. The organization learns. The system does not. 

This is the last layer of the harness and the one fewest teams build, partly because it sounds like it requires machine learning expertise. It does not. The weights never change. What changes is everything around them. 

The loop, in four steps 

The mechanism is unglamorous, which is why it works. 

  1. The agent runs — and produces work. 
  2. The verifier rejects something — with a specific, named reason. 
  3. That reason is written down — as a durable, structured rule — not as a log entry. 
  4. The next run loads it — as an active constraint, before it starts. 

Run that cycle enough times and the accumulated rules start to look like something recognizable: an onboarding document for a new team member, written by the mistakes of everyone who came before them. Entries like never disable a failing test, escalate instead. Flag contradictions between sub-agents rather than silently picking one. Cite a source for every claimed metric. 

Each of those started as a rejection. Now they are preconditions, and the verifier never has to catch that class of error again. 

This is not fine-tuning. The model's weights are untouched. The rules tighten, the routing improves, the context sharpens — until the verifier has less and less left to find. 

The research is ahead of the practice here 

Work on evolving agent contexts has formalized this pattern and measured it. The Agentic Context Engineering framework splits the loop into three roles — a generator that produces the work, a reflector that distills what went wrong into concrete insights, and a curator that integrates those insights into the context — and treats the accumulated result as a playbook rather than a summary. 

The reported results are substantial: roughly ten percent improvement on agent benchmarks and around eight and a half percent on finance tasks against strong baselines, with up to seventeen percent on one agent benchmark, achieved purely from execution feedback with no ground-truth labels. Notably, adaptation latency dropped by an average of eighty-seven percent, because loading a refined playbook is dramatically cheaper than rediscovering the same lessons. 

The part worth internalizing, though, is the failure mode the research identifies. Naive approaches to this — rewrite the accumulated context each cycle, keep it tidy — hit exactly the problems from Part 4 of this series: brevity bias strips the specific domain detail in favor of readable summaries, and context collapse erodes precision a little further with every pass. The lessons that matter most are the ones that summarize worst. 

The fix is structural. Lessons are appended, not rewritten. Each is a discrete, individually addressable entry that can be added, retired, or corrected without disturbing its neighbors. A lesson log is closer to a rules file under version control than to a running narrative. 

What makes a lesson worth keeping 

The obvious way to build this is to write every rejection to the log, and it fails quickly. Within a few hundred runs you have a context full of situational noise, most of it a restatement of a single unusual afternoon. 

Three tests before something becomes a standing rule. Is it general — will it apply again, or is it a fact about one file on one day? Is it actionable — does it tell the agent what to do differently, rather than merely describing what went wrong? And is it still true — because a rule written against last quarter's architecture will be applied with total confidence against this quarter's. 

That last one is the reason this layer needs an owner. Lessons do not expire on their own. A durable memory that nobody prunes becomes an unaudited configuration surface, and the question 'what does this agent currently believe?' should be answerable by reading a file, not by running experiments. 

The order of operations 

Across this series we have covered seven layers, which can read as a demand to build all of them before deploying anything. It is not. Attempting all seven at once is its own failure mode — it front-loads months of infrastructure work before anyone has learned what this particular agent actually gets wrong. 

Add each layer when the pain that justifies it appears. 

  • Day one: tool permissions and a basic verification step. These two prevent the failures that are hardest to recover from and most likely to end up in a public disclosure. Everything else can wait; these cannot. 
  • When sessions start repeating themselves: add memory. The signal is an agent rediscovering the same context every morning. 
  • Before anything touches a customer: add guardrails. The signal is a change in blast radius, not a change in volume. 
  • When you need to prove it works: add observability. The signal is usually someone asking for a number you cannot produce. 
  • When cost gets uncomfortable: add routing. The signal is an invoice that surprised somebody. 
  • When you are tired of the same mistake: add feedback. The signal is a correction you have now given three times. 

Sequenced this way, each layer is justified by a problem the team has actually experienced, which is also the only reliable way to get it funded. 

Where a harness does not help 

A discipline that claims to solve everything solves nothing, so it is worth closing the series on the limits. 

A harness will not fix a vague task definition. Seven well-built layers will reliably, efficiently, and observably produce the wrong thing if nobody agreed on what the right thing was. The most common root cause of a disappointing agent is not a missing layer; it is a specification that never existed. 

Guardrails set once and never revisited become a false sense of security, which is worse than no security, because it stops people from looking. 

Over-harnessing is real. Every approval gate costs latency and human attention, and a harness designed for a regulated bank dropped onto an internal prototyping team will produce a shadow workflow within a fortnight. Match the harness to the blast radius. 

And the one worth remembering longest: the harness is infrastructure, not product. Nobody buys a harness. They buy what the agent produces. The harness matters only because it is the difference between an agent that produces something and one that produces a demo. 

The question, six articles later 

The premise of this series was that model capability has stopped being the interesting variable in enterprise AI. It is broadly available, improving on a schedule nobody controls, and equally accessible to every competitor you have. 

What is not evenly distributed is the engineering discipline to build a reliable system around one. That discipline is ordinary — permissions, verification, state management, instrumentation, cost control, feedback. It is closer to platform engineering than to data science, and most enterprise engineering organizations already have the skills. What they usually lack is the framing that tells them these problems are one problem. 

So the next time an agent initiative stalls and the explanation is that the model was not good enough, the more useful question is which of the seven layers was missing. In our experience the answer is rarely none of them, and it is almost never the model. 

This is the final article in a six-part series on harness engineering. Part 1 introduced the concept; Parts 2 through 5 covered tool orchestration, verification, context and memory, and observability and routing.