Do It Once, Automate It Forever: This AI Learns Your Desktop Job from a Single Demo

You know that one repetitive job you do on your computer every week — copying data between spreadsheets, triaging the same kind of support tickets, formatting the same weekly report? You can explain it to a colleague in five minutes. But explaining it to an AI agent? That's where things fall apart. Agents can click and type, but the moment the workflow depends on your judgment — which row to skip, what counts as "done," when to break the pattern — they fumble. So you end up doing the boring work yourself, again, every week.

A new paper out of a 12-author team led by Yuqi Huang (submitted October 9, 2026) attacks exactly this gap. It's called Work2Skill, and the idea is disarmingly simple: instead of writing instructions for your agent, just record yourself doing the task once — and let the agent induce a reusable skill from your demonstration. In their tests, agents equipped with these induced skills scored 4.67 to 10.54 points higher on desktop-task benchmarks. The agent didn't get a bigger model. It got your procedural knowledge.

What the researchers did

Computer-use agents — systems that control a desktop the way a human would — have improved fast, but they still struggle with unfamiliar software and any workflow that depends on human domain knowledge. The problem with demonstrations is interpretation: a raw screen recording is just pixels and clicks. To reuse it, someone has to figure out the context of each action and separate the general workflow rules ("always verify the total before submitting") from the task-specific details ("this customer's order number was 4482").

Work2Skill solves this with two pieces:

1. WorkRecorder — infrastructure that captures human GUI experience as synchronized multimodal evidence: the screen recording itself, every mouse and keyboard event, and the accessibility-tree information behind the interface. These three streams are aligned in time, so an agent can inspect not just what was clicked, but the UI structure it clicked on.

2. An agentic skill-induction method — a second agent that inspects this recorded evidence on demand and distills it into a reusable skill. The skill isn't a replayed macro; it's a compact knowledge artifact capturing three things: the workflow rules (the norms and policies the human followed), the step dependencies (what must happen before what), and the completion checks (how you know the task is actually done).

To prove it works, the team built WorkDemo-350: 350 human desktop demonstrations, each screened for completeness and task success — the first synchronized multimodal dataset of its kind designed for skill induction. They then tested four different execution models with skills induced by each method. Work2Skill's induced skills delivered the highest scores and success rates across the board: mean improvements of 4.67–10.54 points on MyPCBench Bounded Action and 3.11–7.50 points on a 286-task subset of OSWorld-Verified, compared to the same agents running with no skill at all. The code and data are released open source.

Why it matters

Most agent tooling today treats every task as brand new. Your agent rediscovers, from scratch, how to do the thing you already do weekly. That is enormously wasteful — and it's why agentic automation keeps failing at the "last mile" of real office work, where the software is niche, the rules are unwritten, and the human operator is the only documentation that exists.

Work2Skill reframes the onboarding problem. You are the training data. One careful demonstration — performed once, with intent — becomes a durable skill the agent can reuse on variant tasks. The executor in the paper's setup never even watched the original recording; it just received the induced skill text and the new task, and still outperformed the baselines. That separation is the whole point: the skill captures the rules, not the recording.

For anyone running computer-use agents (Claude, OpenAI's operator-style tools, open-source desktop agents), this is the cheapest capability upgrade available: you don't need a better model, you need a library of your own best workflows, written the way you do them.

How YOU can use it

You can't run the researchers' exact pipeline yet — but their recipe translates directly into a workflow you can use today with any decent agent or LLM. Here's the four-step version:

1. Pick one repetitive desktop workflow

Choose something you do at least weekly, that has clear steps and a clear "done" state: reconciling invoices, generating a weekly status report from three dashboards, triaging a shared inbox, onboarding a new client in your CRM. Repetitive, rules-shaped, verifiable. That's your candidate.

2. Record a narrated demo

Do the task once, carefully, and narrate your decisions out loud — or type a running commentary as you go. The key is capturing the rules you never wrote down: "I always skip rows marked pending," "if the total doesn't match the invoice, I re-check line 3 before submitting," "this step only applies to EU customers." Work2Skill's insight is that the judgment calls are the skill; the clicks are just evidence.

3. Distill the demo into a skill card

Feed your demo notes (or transcript) to an LLM with this prompt:

You are distilling a human's desktop workflow demonstration into a reusable SKILL CARD for an AI agent. From the demonstration notes below, extract:

1. WORKFLOW RULES — the norms and policies the human followed (what's always/never done)
2. STEP DEPENDENCIES — what must happen before what, and why
3. COMPLETION CHECKS — how the agent can verify the task is actually done
4. EDGE CASES — the exceptions the human handled, and how

Write it as a compact reference card. Separate general rules from details specific to this one instance. Ignore irrelevant asides. End with a 5-item pre-submit checklist.

DEMONSTRATION NOTES:
[ paste your notes here ]

Save the result as a skill card — a small markdown file you keep in a folder of your own procedures, one card per workflow. You now own the asset the paper is talking about: workflow rules, step dependencies, completion checks, distilled from your own demonstration.

4. Run the skill, then harden it

Give your agent the skill card alongside the next real instance of the task ("Here is the skill card for how I do this; apply it to this new case"). Watch where it stumbles — those stumbles are missing rules. Add them to the card. After two or three rounds of this loop, you'll have what the researchers demonstrated: a skill that makes the agent reliably better, without retraining anything.

Try it tonight

Pick the single most boring repetitive task on your plate this week. Do steps 1–3 for it — twenty minutes of work. Then hand the skill card to your agent and let it take the next instance. That's the Work2Skill pattern in miniature: one demonstration, captured deliberately, compounding forever.

Key takeaways

• Demonstrations beat instructions: recording one careful run captures procedural knowledge that written instructions miss.
• Rules, dependencies, checks: a reusable skill needs the workflow's norms, its ordering, and its definition of "done" — not just the clicks.
• Measured gains: skill-equipped agents improved 4.67–10.54 points on MyPCBench and 3.11–7.50 points on OSWorld-Verified tasks over the same models with no skill.
• Your move today: narrate one repetitive workflow, distill it into a skill card with an LLM, and reuse it — your own WorkDemo library starts with a single file.

Read the original paper: "Work2Skill: Capturing Human GUI Experience for Reusable Agent Skill Induction" — Yuqi Huang, Hui Li, Yifei Cao, Zikun Cai, Tianci Xie, Yangfan Gao, Pinxue Guo, Chaoyun Zhang, Bingxuan Li, Rui Zheng, Hang Yan, Ning Liao — arXiv:2610.12984 (cs.AI, submitted 9 Oct 2026).