Your AI Agent Works in the Lab and Fails at Your Desk — Training It in 18 Setups Fixes That

You have seen this failure before. An AI agent looks brilliant in a demo: it files the ticket, updates the spreadsheet, writes the summary. Then you hand it your real setup — a different code editor, a different tool layout, a slightly different way of showing results — and the same agent starts fumbling. It calls the wrong tool, loses track of the task, or produces work that looks finished but is not. The model did not get dumber overnight. The setup around it changed, and the agent had only ever learned to work inside one setup.

A new paper from Renmin University of China, IQuest Research, and Beihang University offers a practical fix: train an agent across many varied harnesses, not just one. The result, they report, is an agent that transfers to setups it has never seen — including a popular coding tool it was never trained in.

What the researchers did:

The paper is "Harness Scaling for Generalizable Agentic Reinforcement Learning" by Fei Bai, Huatong Song, Chuan Hao, Ran Tao, Jian Yang, Bryan Dai, Wayne Xin Zhao, and Ji-Rong Wen, submitted to arXiv on 21 October 2026 and revised on 25 October 2026. First, a definition. A "harness" is not the model. It is everything wrapped around the model when it works as an agent: the system prompt, the tools it can call, the format of what those tools send back, how earlier steps are kept or summarised, and the workflow it must follow. Change any of those, and you have a different working environment — even though the model inside is identical.

Most agent training today happens through one fixed harness. That teaches the model to be good at that one environment, and it can quietly specialise to its quirks: where the answer is expected, how tool output is phrased, how much context is kept. Move the agent somewhere else and that specialisation becomes a liability. The authors treat harness diversity as a scaling dimension of its own.

Their method has three parts: build a factorised harness space, breaking a harness into components so variations combine systematically; select a compact training subset with balanced factorial coverage — in the headline experiments, 18 configurations chosen so components and their combinations are covered evenly; and organise training with locality-aware task assignment, so structurally nearby configurations are assigned to the same task, keeping variation controlled rather than chaotic.

They then trained a small open model, Qwen3-4B-Instruct, with GRPO reinforcement learning, and compared three versions across three benchmarks — ClawGym-Bench, JobBench-Easy, and PinchBench. The untrained base averaged 21.89 Pass@1. Training through a single fixed harness lifted that to 29.81. Training across 18 configurations lifted it further, to 39.04. Same model, same task family, but a much better agent — because training varied the setup, not just the questions.

The transfer test matters most. The researchers dropped the models into Claude Code, a harness none had seen in training. On JobBench-Easy inside Claude Code, the single-harness model scored 5.79 — below the untrained base model's 6.78. Training in one harness had made the agent worse in a new environment. The harness-scaling model scored 14.83. It had not seen Claude Code either; it had simply learned not to depend on any one setup's quirks.

Why it matters:

This reframes a common productivity failure. When an agent underperforms in production, the instinct is to blame the model and buy a bigger one, or to blame the prompt and rewrite it again. This paper says: check the harness first. If your agent was built, tuned, or evaluated inside a single setup, its score there tells you almost nothing about how it will behave in yours. A demo is a harness. A benchmark is a harness. Your desk, with your tools, permissions, and file layout, is a different harness — and that is the only one that counts.

There is a cost lesson too. The model here is a small 4B model, not a frontier giant. The gain came from how it was trained, not more parameters. A smaller model that survives a change of tools can beat a larger model that only shines in the lab. But these are scores on three benchmarks, not a guarantee of the same gain in your workflow.

How YOU can use it:

You probably cannot retrain a model across 18 harnesses this afternoon. You can steal the paper's discipline and apply it to any agent you already use.

1. Test your agent in a second harness before you trust it. Pick one real task your agent does well, such as "summarise five tickets and draft replies." Run it in its usual setup, then run the exact same task in a deliberately different one: a different chat tool, a different file layout, tool results as plain text, or a shorter memory window. Score both against the same checklist of what "done" means. If the score collapses in the second setup, you do not have a reliable agent; you have a reliable demo.

2. Vary one harness factor at a time. The paper's power comes from structured variation, not random chaos. List your factors — system prompt, tool list, tool-output format, memory handling, workflow steps — and change one per test. Illustrative sketch (NOT paper code):

# Illustrative harness-variation checklist, one task, same scorecard
# Run A (usual): rich tool output, full history kept
# Run B: tool output as plain text only
# Run C: history summarised after every 2 steps
# Run D: different tool order + stricter output format
# Score all runs on: correct tickets, correct sources, no invented facts, reply sent to the right place.

A drop in one run names the factor your agent depends on. Tighten the prompt for it, add a verification step, or change the tool format — then re-test. That is Harness Scaling by hand: removing single-setup dependence.

3. Write "done" so it survives a change of tools. Single-harness agents recognise success by surface cues: a button label, a familiar output format, a confirmation message. Define success independently of those cues. "The reply is sent and the ticket status is updated" survives a tool change; "the output looks like last time" does not. Make the agent verify it — read the ticket status back, do not assume the send worked.

4. Keep a harness changelog. Every time you change a tool, permission, model, or output format, note it as a harness change and re-run your standard task set. If a harness change drops your scores, you found the dependency before your users did.

Key takeaways:

- A harness is everything around the model — prompts, tools, output formats, memory, workflow — and changing it changes the agent's environment.
- Single-harness training can make an agent worse in a new setup than an untrained model: 5.79 versus 6.78 on JobBench-Easy inside unseen Claude Code.
- Harness Scaling across 18 configurations lifted a small Qwen3-4B model from 21.89 (base) and 29.81 (single harness) to 39.04 average Pass@1 across three benchmarks.
- Never trust an agent tested in only one setup. Vary the setup, score against a fixed definition of done, and fix the fragile factor.

Read the original paper: "Harness Scaling for Generalizable Agentic Reinforcement Learning" (arXiv:2610.29458) by Fei Bai, Huatong Song, Chuan Hao, Ran Tao, Jian Yang, Bryan Dai, Wayne Xin Zhao, and Ji-Rong Wen — https://arxiv.org/abs/2610.29458