3,000 Lines vs 512,000: The Tiny Agent Harness That Nearly Matches Claude Code

The default advice when your AI agent disappoints you is simple: get a bigger model. Rent the newest frontier model, pay the per-token bill, and hope the extra brainpower covers the gaps. A new paper from a team building agents for government intelligence work suggests the opposite move: keep the model small, fixed, and self-hosted — and shrink the harness around it until almost nothing is left.

Their minimal agent, called CLEO, scored 83.8% on a held-out investigative-analysis dataset. The strongest out-of-the-box baseline, the Claude Code SDK, scored 85.0%. The gap is 1.2 percentage points. The gap in size is two orders of magnitude: the paper's comparison table puts CLEO's harness at roughly 3k against the Claude Code SDK's 512k. This is a paper about what actually makes agents good — and the answer is not the model.

The paper is "Optimize the Harness, Not the Model: Fitting a Minimal Agent to a Self-Hostable LLM for Investigative Analysis" by Shai Nahum Gefen, Vered Chassida, Dmitry Basin and Guy Feigenblat (arXiv:2611.03434, 2 November 2026).

What the researchers did

The starting point is a problem most agent demos never face. Government intelligence teams need LLM agents to analyse large, messy evidence collections — but security rules and cost often rule out hosted frontier models, and a broad, hard-to-audit tool surface is a liability. Every extra tool is another thing that can go wrong, leak data, or be exploited. The team needed an agent that could pass a security review, run on premises, and still do the job.

Their answer is CLEO: Constrained Loop for Environment Optimization. It is a method for fitting a minimal agent harness to a fixed, self-hostable LLM — not a new model, and deliberately not a bigger one. In their experiments the fixed model was GLM-5.1, an open-weights model they could run themselves. The model is treated as a constant. Everything around it is the variable.

The harness starts as small as an agent can plausibly get: a ReAct-style model-in-a-loop — the model thinks, acts, observes the result, and repeats — with a terminal toolset of exactly four tools: list files, read a file, search text, and a sandboxed, non-persistent Python environment. Nothing else. No web search, no databases, no plugin marketplace.

Then comes the part that gives CLEO its name: the justification gate. A proposed addition — a new tool, a guardrail, a memory policy, a loop component — is admitted only if it clears four tests at once. It must improve accuracy on an expert-validated dataset. It must remain general, not a hack for one case. It must remain auditable, so a reviewer can understand what it does. And it must be worth the added capability surface — the extra power it gives the agent has to justify the extra risk. Anything that fails any test stays out.

Almost everything the team tried failed the gate. The one addition that earned its place was persistent per-case memory: a small memory the agent keeps for each investigation, so it can carry context across steps instead of starting each turn from scratch. That single addition, plus the four tools, is the whole harness.

The evaluation was deliberately strict. The team tested on a held-out dataset — including out-of-distribution cases the harness had never seen — with a single Claude Opus 4.8 model acting as judge. The results, from the paper's comparison table:

• CLEO: 83.8% pass rate, 8.41/10 soft score, 114 seconds average per case
• Claude Code SDK (bare): 85.0% pass rate, 8.48/10 soft score, 76 seconds average per case

CLEO lands within 1.2 percentage points on pass rate and 0.07 points on the soft score — while being roughly 170 times smaller. It also matched or beat PI, a minimal execution-loop baseline the paper uses for comparison. And this is not a lab demo: the paper reports the agent is currently being deployed in client environments, small enough for a security review. The work was presented as an Industry Day talk at CIKM 2026 in Rome on 7 November 2026.

The paper is honest about the trade-off. CLEO is slower — 114 seconds per case against 76 — and slightly behind on accuracy. Minimal does not mean free. It means you know exactly what you are paying, and what you are getting for it.

Why it matters

First, the harness is the product. A growing pile of 2026 research says agent performance is dominated by the scaffolding around the model — the prompt, the tools, the memory, the loop — not the model itself. CLEO is the cleanest demonstration of that idea so far: change nothing about the model, shrink everything around it to almost nothing, and lose almost nothing on accuracy.

Second, auditability is a feature, not a tax. In regulated work — intelligence, banking, healthcare, legal — a harness of roughly 3,000 units can be read, reviewed and approved by a human. A harness of 512,000 cannot. Small is not a compromise here; it is what makes deployment possible at all. If your agent cannot pass a security review, it does not ship, however smart it is.

Third, cost and control. A fixed, self-hosted model means no per-token frontier bills, no data leaving your premises, and no capability silently changing under you when a provider updates a model. For teams handling sensitive material, that control is worth more than a point or two of benchmark accuracy.

Fourth, the gate generalises. You do not need their exact four tools. The method — every addition must earn its place on a validated dataset, and stay general and auditable — ports to any agent you build, in any domain. It is a design discipline, not a codebase.

Finally, it reframes what "minimal" means. Minimal is not weak. Minimal means every component justifies itself. Most production agents accumulate tools, prompts and retries until nobody can say why any single piece exists. CLEO shows what happens when you force each piece to argue for its life.

How YOU can use it

You do not need a government contract to use this. A CLEO-style minimal agent is a weekend project, and the recipe follows directly from the paper:

Step 1: Freeze the model. Pick one self-hostable open model and run it locally — Ollama or vLLM will do — then stop shopping. The whole point is that the model is not the variable. If you keep swapping models, you will never learn what your harness is doing.

Step 2: Give it four tools only. List files, read a file, search text, run sandboxed Python. A simple ReAct loop ties them together: the model reasons, picks a tool, reads the result, and repeats. Resist adding web search, databases and ten plugin servers on day one. Every tool you add now is a tool you will have to debug, secure and explain later.

Step 3: Build a validation set first. Collect 20–50 real cases from your own work with known right answers — for example, "given this folder of invoices, produce the Q3 summary" — and score the agent against them with a simple rubric, by hand or with a strong judge model. No validation set, no gate: without it, every addition feels like an improvement.

Step 4: Run the gate. Propose one addition at a time — a memory file, a new tool, a guardrail. Keep it only if accuracy on the validation set goes up AND the addition is generic (it helps many cases, not one) AND you can explain what it does to a colleague in one sentence. If it fails any test, it stays out. Write the decision down; the log is the audit trail.

Step 5: Add memory first. The one addition that earned its place in CLEO was persistent per-case memory: a small notes file the agent reads at the start of each turn and updates as it works — what it has found, what is still open, what it has already tried. For any long task, memory beats tools. It is also the cheapest addition you can make.

Step 6: Measure three numbers, like the paper did. Pass rate, quality score, and latency. Expect the minimal agent to be a little slower — CLEO was, at 114 seconds against 76 — because a small model with few tools takes more turns. You are trading a bit of speed for a system you can actually understand, audit and deploy. That is usually a good trade.

ILLUSTRATIVE sketch — not code from the paper:

# A minimal CLEO-style agent loop (illustrative — not the paper's code)
# Four tools only + per-case memory + a justification gate for additions.

TOOLS = ["ls", "read", "grep", "python_sandbox"]  # nothing else, to start

def run_case(task, case_file="memory.md"):
    memory = read_file(case_file) if exists(case_file) else ""
    for turn in range(MAX_TURNS):
        # ReAct: reason, then act, then observe.
        action = llm(f"Task: {task}\nMemory:\n{memory}\nTools: {TOOLS}\nNext step?")
        result = run_tool(action)  # sandboxed: no network, no writes outside the case folder
        memory += f"\n- {action} -> {result[:500]}"
        write_file(case_file, memory)  # persistent per-case memory: the one addition that earned its place
        if action == "done":
            return result

# THE JUSTIFICATION GATE — run before adding anything to TOOLS:
# 1. Does accuracy on the validation set go UP? (measure, don't guess)
# 2. Is it GENERAL — does it help many cases, not one?
# 3. Is it AUDITABLE — can you explain it in one sentence?
# 4. Is it WORTH the added capability surface?
# Fail any test → the addition stays out.

One guardrail from the paper's own logic: do not judge your agent with the same model that runs it, on the same cases it trained on. Keep the validation set fixed, human-checked, and separate from anything the agent sees while working — or the gate stops measuring anything. And resist adding tools to fix a single bad case: that is exactly how a 3k harness becomes a 512k one.

Key takeaways

• A minimal agent harness (CLEO) reached an 83.8% pass rate on held-out investigative analysis — within 1.2 points of the Claude Code SDK baseline at 85.0% — while being two orders of magnitude smaller (roughly 3k vs 512k in the paper's table).
• The method is the message: fix a self-hostable model, start with four tools (list, read, search, sandboxed Python), and admit any addition only if it improves accuracy on an expert-validated dataset while staying general and auditable.
• The only addition that cleared the gate was persistent per-case memory — for long tasks, memory beats tools.
• The honest trade-off: CLEO is slower (114s vs 76s per case) and slightly behind on accuracy — minimal means knowing exactly what you pay, not paying nothing.
• The practical move is yours: freeze one local model, build a 20–50 case validation set, and make every new tool argue for its life before you add it.

Read the original paper: Optimize the Harness, Not the Model: Fitting a Minimal Agent to a Self-Hostable LLM for Investigative Analysis (arXiv:2611.03434) by Shai Nahum Gefen, Vered Chassida, Dmitry Basin and Guy Feigenblat. Link: https://arxiv.org/abs/2611.03434