Two weeks ago I wrote a rule into a config file: never apply an edit the human has not approved.

It was a good rule. It was clearly worded, it sat near the top of the file, and the agent read it at the start of every session.

One session ran straight through it eleven times in an afternoon and put roughly seventy-six unapproved edits into a client deliverable. Nothing stopped it. No error, no flag, nothing waiting for a yes. Two hours later I went through the file putting a marker on every line the agent had rewritten without asking. Forty-six lines, in a document with a client's name on it.

I had written a good rule into the wrong place.

I rewrote it as a hook: a script the harness runs before it dispatches the tool call, which reads the pending call and returns a deny. The agent can still decide to make the edit. The edit never reaches the file.

The hook has been wrong twice since. Both times I had written the rule too narrowly, and both times I could open the file and point at the line that was wrong. That is the part worth having. The prompt version gave me nothing to point at.

That gap, between a rule you state and a rule you enforce, is what a harness is. It lives in five places:

1. What the agent is allowed to do
2. What it can reach
3. What survives a restart
4. Who checks its work
5. What you pay to show it anything

Each one below comes with code, and with a repo where you can watch that layer fail on your own machine before it fails on you in production.

(If agents themselves are still new to you, start here and come back. Everything below assumes you have already shipped one and watched it misbehave.)

What the Harness Actually Buys You

Let me start with the thing that sounds too simple to be worth saying.

The core of a coding agent loop is six lines:

while agent.turns < max_turns:
    call = agent.next_call(observations)
    if call is None:
        break
    result = execute(call)
    observations.append(result)

That's it. That's the agent. Add error handling and a retry and you are at about twenty lines. Every framework you have evaluated wraps those six, and the interesting part is execute, which is where you decide what the model is allowed to do, what it can reach, what it remembers, who checks it, and what it gets to see.

That is the harness. And right now your feed is full of people telling you it matters more than which model you pick.

I spent a week checking that claim. It does not hold, and what replaces it is more useful.

Where the harness dominates: cost. A June 2026 preprint ran three harnesses (Goose, OpenCode, OpenHands-SDK) across two models on a stratified 50-task subset of Terminal-Bench Pro, the harder variant of the Terminal-Bench agent benchmark. It measured up to a 40x difference in tokens per solved task. Upgrading the model moved the same number by 1.0 to 1.3x.

Where it does not: capability. The same study found pass-rate differences between harnesses of 0 to 8 percentage points, with bootstrap confidence intervals crossing zero on all but the largest gap. The authors call their own accuracy findings descriptive, on n=50.

So the same task, on the same model, at the same success rate, cost forty times more on one harness than on another.

Everything below makes your agent cheaper and harder to surprise. None of it makes a weak model smarter.

Move the Rule Into Code

A rule in a system prompt is a bet that the model remembers it, reads it the way you meant it, and follows it under context pressure. The model wins that bet most of the time. That reliability is the danger, because it is the reason you stop checking.

A rule in code runs on every call, including the ones where the model has forgotten it exists.

There are five places in an agent loop where a rule can be enforced by code. The sections below take them one at a time: what the layer is for, the code that enforces it, and what the same agent does when the layer is missing. You do not need all five on day one. Start with the layer whose failure you can picture happening to you this week.

Layer 1: The Execution Boundary

The question: what is the agent allowed to do?

The boundary is a function that runs between the model's decision and the tool call. Here is the same agent without one. The system prompt says never delete a file without asking, and it has three tool calls queued.

SYSTEM_PROMPT = "IMPORTANT: never delete a file without asking the user first."

def execute(call):
    if call.name == "delete_file":
        FILES.pop(call.args["path"], None)
        return f"deleted {call.args['path']}"

Run it and the agent deletes cache.tmp, which is junk, and then deletes client-deliverable.md, which is exactly what the name says. It never asked. The rule was in the prompt the whole time.

The fix is a function that runs before the tool:

def boundary(rules):
    def wrap(execute):
        def guarded(call):
            for name, deny_if, reason in rules:
                if deny_if(call):
                    return f"DENIED by {name}: {reason}"
            return execute(call)
        return guarded
    return wrap

NEEDS_APPROVAL = [(
    "delete-needs-approval",
    lambda c: c.name == "delete_file" and not c.args.get("approved_by_human"),
    "deletion requires an explicit human approval flag on the call",
)]

Same agent, same script, same prompt.

Nothing gets deleted.

Layer 2: Sandboxing

The question: what can the agent reach?

A sandbox is the set of directories and hosts that the process running your agent can touch at all. You build one by starting from nothing and adding back what the job needs: an allowlist of paths, an allowlist of hosts, and a process that cannot see anything outside them.

The usual version is a deny list that names the paths the agent must not touch:

DENY = ["secrets/"]

def execute(call):
    path = call.args["path"]
    if any(path.startswith(d) for d in DENY):   # checks the spelling
        return "DENIED by deny-list"
    real = os.path.normpath(path)               # the ../ collapses HERE, after the check
    return DISK.get(real, "not found")

Run it and the agent asks twice:

read('secrets/api_key')          -> DENIED by deny-list
read('work/../secrets/api_key')  -> sk-live-DO-NOT-LEAK

The check ran on the string the agent typed. The lookup ran on where that string actually points.

Two lines apart, and both spellings are the same file.

The fix is to resolve the path before comparing anything:

ALLOW_ROOTS = ["work"]

def resolve(path):
    real = os.path.normpath(path)               # resolve FIRST
    if not any(real == r or real.startswith(r + os.sep) for r in ALLOW_ROOTS):
        return None
    return real

Both spellings now arrive as secrets/api_key, neither one sits under work, and both are refused. The allowlist replaced the deny list for the same reason: a deny list only stops the spellings you thought of.

Both versions run in the repo as failures/02_no_sandbox.py and layers/02_sandboxing.py. They are Python functions so you can run them without setting up a container. In production the same check is the process itself, a user or container with no read permission outside work/, and the order of operations is identical.

This is the job Layer 1 cannot do. A check inside your own program only sees the calls it recognises, and the same file can be reached through a shell command, through a symlink, or through a path with .. in it. None of those are the call it was watching for.

Layer 3: Memory Persistence

The question: what survives a context reset?

The most useful thing I changed this year was learning to throw away conversations without hesitation.

Context fills, quality degrades, the agent starts contradicting itself. The instinct is to nurse the session along. The better move is to restart it freely and make sure the things that matter live somewhere a restart cannot touch: the committed code, the config, the harness itself, the extracted notes.

That means deciding, deliberately, where every piece of state your agent produces belongs: (1) in the conversation, where it dies with the session, (2) on disk, where it survives and can be retrieved, or (3) in the harness config, where it shapes every session that follows.

Three destinations, one dispatch:

CONVERSATION, DISK, HARNESS_CONFIG = [], {}, {}

def remember(kind, key, value):
    """'chat' dies with the session, 'disk' survives it,
    'config' shapes every session that follows."""
    {"chat": lambda: CONVERSATION.append(value),
     "disk": lambda: DISK.__setitem__(key, value),
     "config": lambda: HARNESS_CONFIG.__setitem__(key, value)}[kind]()

The dispatch is trivial. The value is that you can no longer store anything without saying out loud which of the three it is.

Most teams never make that call explicitly, which is why their agents feel amnesiac in some ways and stubbornly wrong in others. (Run layers/03_memory_persistence.py: the conversation comes back [] while the decision and the rule are still sitting in disk and config.)

Layer 4: Verification Loops

The question: who checks the work, and what stops them fixing it themselves?

A verification loop is a second pass over the agent's output by something that can report a problem and cannot fix it.

The second half of that sentence is the part people skip. Give a reviewer subagent write access and it will silently repair what it finds. That sounds efficient. It destroys the signal, because now you cannot tell the difference between work that was right and work that was wrong and got patched.

Make the reviewer read-only. Its output is a critique. Acting on it is a separate step with its own record. That separation is what keeps the signal readable.

The second pattern here is the dry run. For anything irreversible, the tool takes a flag, defaults to describing what it would do, and only acts when the flag is explicitly off. The default is the safe one, so forgetting is harmless. (In layers/04_verification_loops.py, apply_fix prints DRY RUN: would rewrite total.py, nothing written and the off-by-one is still there at the end.)

Both rules fit inside one execute:

def execute(call):
    if call.name == "review":
        snapshot = dict(CODE)               # a copy, so the reviewer cannot write
        src = snapshot[call.args["path"]]
        return f"VERDICT: {'off-by-one' if '+ 1' in src else 'looks good'}"
    if call.name == "apply_fix":
        if call.args.get("dry_run", True):  # on by default, turned off on purpose
            return f"DRY RUN: would rewrite {call.args['path']}, nothing written"
        CODE[call.args["path"]] = call.args["new"]
        return f"wrote {call.args['path']}"

The reviewer finds the off-by-one and cannot reach it.

Layer 5: Context Pipelines

The question: what does the model actually see, and what does that cost?

A context pipeline is the route information takes to reach the model's context window, and the decision about how much of it arrives.

The deciding is the expensive part. A token you let into the window is one you pay for on that turn and on every turn after it, which is why the forty-fold gap between harnesses shows up here and not in Layer 1.

The naive harness puts everything in the window: the whole file when the agent asked about one function, the whole search output when it needed one line, and every turn that came before. It works. It also means turn twenty pays again for everything turns one through nineteen read.

The move that fixes it is delegation with distillation. A subagent gets its own context window, explores tens of thousands of tokens, and returns one to two thousand. The main thread never sees the search dump. It sees the answer.

In code, the whole trick is which counter gets incremented:

def subagent_search(query):
    """Its own window. The main thread never pays for this reading."""
    global subagent_tokens
    subagent_tokens += sum(len(v.split()) for v in CORPUS.values())
    return next(f"{n}: {b.split('ANSWER:')[1].strip()}"
                for n, b in CORPUS.items() if "ANSWER:" in b)
def execute(call):
    global main_tokens
    distilled = subagent_search(call.args["q"])
    main_tokens += len(distilled.split())    # the only line that bills you
    return distilled

The demo in the repo is the smallest honest version of that. Same question, same answer, both times. The main thread reads 9,602 tokens in one and 2 in the other.

That is also why the sequential-subagent pattern disappoints people: a chain where each step waits on the last buys you context isolation, and it buys you no parallelism at all. If the steps are genuinely independent, run them that way. If they are not, be honest that you are paying for isolation. (Isolation is often worth the money. Just know that is what is on the invoice.)

What Nobody Can Tell You Yet

The limits, because the confidence around this topic is running well ahead of the evidence under it.

1. Not one controlled study here is peer reviewed. Every harness-isolation result as of August 2026 is an un-refereed arXiv preprint. The most careful experimental one describes itself as preliminary work under review at a workshop. The most-quoted framing paper is a self-labelled position paper that ran no experiment of its own. What this means for you: nothing here has cleared peer review, mine included. Re-run the comparison on your own tasks before you budget against any of it.
2. Harness gains often do not survive new tasks. A July 2026 paper found that automatic harness evolution "does not consistently outperform simple test-time scaling methods and exhibits limited generalization," because the search and the final evaluation share a benchmark. A continual-learning evaluation on Terminal-Bench 2.0 watched GEPA climb to 70.8% on the tasks it was tuned against, then fall to 54.5% across the wider set, below the 56.8% it started from. The optimization made it worse. What this means for you: measure your harness on tasks you did not use to build it. A gain that only appears on the tuning set is a gain you lose in production.
3. A famous fine-tuning result is scored on its own reward signal. Baseten fine-tuned Gemma 3 27B for clinical note generation, taking it from 35% worse than Claude Sonnet 4 to 60% better. Their write-up says plainly that "the eval harness itself is used as the reward signal during training." Legitimate technique, and the result is measured against the thing that trained it. What this means for you: when a vendor reports a fine-tuning win, ask what the eval was during training. If it is the same harness, the number tells you the model learned the eval.
4. Your harness is not portable. A June 2026 study found that applying a harness only after training recovers little of the benefit of training with it in place. Models post-trained on a minimal harness returned "Invalid tool format" in 75.1% of attempts once the tool definitions changed shape underneath them, and one Qwen2.5–7B variant scored 10.8 points below the base model it started from. A harness that works with every model works with every model slightly worse. What this means for you: choose your harness before you post-train. Bolting one on afterwards recovers little of what training with it in place would have given you. If you are not training your own model, expect a mid-project harness swap to cost more than the migration looks like it should.
5. The field has not agreed what the word means. Hugging Face published a glossary in May 2026 splitting harness from scaffold, and noted that Claude Code, Codex and others call the whole thing a harness anyway. Five layers is one decomposition. Under a narrower definition, half the evidence above is measuring something else. Pick a vocabulary, stay consistent inside your team, and treat cross-study comparisons carefully. (This is why the five layers arrive as a checklist for your own setup: a checklist survives a vocabulary change, and a taxonomy argues with one.) What this means for you: before you believe a gap between two agent results, ask which layers each one included.
6. And harness effects appear to shrink as models improve. Harness-Bench reports that stronger model backends show "higher mean scores while exhibiting lower cross-harness variance." The scaffolding may be compensating for weakness that is on its way out. What this means for you: weight your harness work toward cost control, which persists, over accuracy, which the next model release may absorb for free.

The Repo

Ten scripts, at github.com/paoloap-py/agent-harness-guide: each layer with its guard, and the same agent without it. Every one runs on your machine with no API key, because the model is replaced by a scripted stand-in that emits a fixed sequence of tool calls. A difference you can run beats a claim you have to trust.

git clone https://github.com/paoloap-py/agent-harness-guide
cd agent-harness-guide

python3 run_all.py        # all five layers, guard off then on, side by side
python3 test_harness.py   # asserts every difference above, 10 checks

The README carries the five layers as an audit checklist. For each one, the question is the same: is this enforced by code, or by a sentence in a prompt? Any box you cannot tick with a file path is a rule you are hoping for.

Where This Leaves You

My rule was good. It was in the wrong place for two weeks, and the cost of that was fifteen violations, eleven of them in the one afternoon, and a day I would like back.

Moving the rule is what enforced it. The words never changed.

That is the whole idea. You buy the model. You build everything around it, and that half decides what your agent costs and how often it does something you did not sanction. Both are engineering work. Only one of them is yours.

If this taught you something, give it 50 claps. It tells Medium to show it to more engineers before their agent deletes the wrong file.

I break down the engineering behind AI systems like this every week in The AI Engineer, where software engineers get dangerously good at AI engineering. Subscribe free.

Why most AI agents fail in production is the other half of this, for when the breakage is inside the agent and the code around it is fine.

FAQ

What is an agent harness? The code around the model: the tool loop, the checks that run before a tool executes, the sandbox the process lives in, what persists across sessions, what reviews the output, and what context each turn receives. The loop itself is six lines, twenty with error handling. The harness is everything else.

Does the harness really matter more than the model? On cost, yes, by up to 40x in tokens per solved task. On whether the task gets solved, no: measured pass-rate differences between harnesses sit at 0 to 8 percentage points with confidence intervals crossing zero. The Terminal-Bench authors conclude model selection usually matters more for performance.

Where does harness choice cost me the most? Context. Every token you hand the model on every turn is a token you pay for on every turn, which is why Layer 5 is where the money goes and why delegating a search to a subagent changes the bill and not the answer.

Are permission rules a security boundary? No. Your harness check runs on the call the agent made, and the same file is reachable through a shell command, a symlink, or a .. path that check never inspects. The limit that holds is the one the operating system puts on the process: start it with access to nothing and add back what the job needs.

Should I build my own harness or adopt one? Adopt one and commit to it. Switching resets the institutional knowledge in your config files, and every rule you wrote after a real failure has to be earned again.