Point at Any Gadget, Wiggle Your Fingers — This Coding Agent Builds the Remote in 32 Seconds

Your desk is full of devices that refuse to talk to each other. The synthesizer speaks MIDI over USB. The phone gimbal speaks a vendor Bluetooth protocol. The video playing in your browser only listens to key presses. Normally, connecting your hands to any of them means writing an integration in advance — one per device, per transport, per mapping — before you ever touch anything.

A new paper from November 2026 flips that order. ControlAnything, by Bektur Ryskeldiev, lets you point at a device in mixed reality, say what your hands should do — and a coding agent writes the integration on the spot, tests it against the live device, and hands you the controls. Median time from request to ready: 32.5 seconds.

It runs on a Meta Quest 3 and was built for the Meta Global AI Developer Hackathon (November 2026). The code is open source under the MIT license, at github.com/Apolotary/controlanything. The paper is "ControlAnything: Generative UI and On-Demand Device Integration for Mixed-Reality Hand Interaction by a Coding Agent" (arXiv:2611.08321, submitted 5 November 2026).

What the researchers did

The interaction has five steps, and none of the device-specific work exists before you ask:

1. Select. In Annotate mode, SAM 3.1 outlines the objects the headset camera sees. A pinch on an outline creates a tag — a persistent record of the object — with a small card beside it showing its name. Tagging alone does not start control.

2. Ask. In Control mode, the card offers "Ask Muse" with no setup step. You speak a goal or tap a suggestion chip — for example, "give me knobs for my OP-1 filter connected to this Mac". A request can name a device reachable from the tagged object, as the evaluation's synthesizer and gimbal requests did.

3. Generate and test. An agent on your laptop, built on Meta's Muse Spark model, writes a connector: a small Node program that declares named actions such as cutoff and translates them into device messages. The agent runs the connector, sends test values through it to the live device, and reads the acknowledgements — log lines recording that the transport accepted each message. A code card streams the agent's commands and files while it works, so you can watch it happen.

4. Use. The agent returns AR controls bound to the connector's actions, a hand program that the headset runs on every frame to map tracked hands to those actions, or both. The transports themselves — Web MIDI, a Bluetooth bridge for the DJI gimbal, and macOS key presses via AppleScript — are hand-written and documented in the agent's prompt. Everything else is generated per request.

5. Stop. You can stop generation, a running program, or everything at once — and stopping never depends on the model.

Four design goals hold the system together. Integration at the time of use: no connector, control layout, or hand mapping is prepared in advance. Grounded and checked output: every control binds to an action of a running connector whose main action was exercised on the live device, and generated code passes automated checks before it reaches the headset. No model in the control loop: after generation, hands are mapped to device actions locally on every frame — a model call takes seconds, a frame takes milliseconds. Interruptible and bounded execution: each agent run is capped in turns, tool calls, time, and tokens.

Then the test. One session, on a real Quest 3, with the author as the only wearer: ten requests across five targets. A fingertip on the laptop screen rippled a visual synthesizer. A generated hand program scrolled Instagram Reels in Chrome, with generated Prev and Like buttons. Generated knobs and keys drove the filter of an OP-1 Field synthesizer. A DJI Osmo Mobile 4 gimbal was steered by a finger. Nine of the ten requests reached a ready state; the wearer stopped one. Median time to ready was 32.5 seconds (range 24.9–57.8 seconds, excluding speech transcription). Sequential model calls accounted for a median 89.9% of that time, and 91.7% of input tokens came from the prompt cache.

Read the honest limits in the same breath as the numbers. The system's evidence of success extends to transport acknowledgements, not to physical effects: it knows the message was accepted, not that the knob actually turned. The evaluation is a single session with a single wearer — the author. A user study with first-time wearers is planned, and the paper names its own next steps: reusing connectors across requests, and observing the physical effects through the headset's own sensors instead of trusting the acknowledgement.

Why it matters

First, the integration tax is the real bottleneck in mixed reality. Headsets already track your head and hands continuously. What has been missing is the glue: every new device needs its own integration, written ahead of time, for its own transport and mapping. ControlAnything removes the "ahead of time" part. If the pattern holds, the question stops being "is this device supported?" and becomes "can an agent figure out this device in thirty seconds?"

Second, generative UI grows up. Interfaces created by AI models so far have mostly been web pages, task panels, and virtual scenes — or AR control panels for devices whose integration already existed. Here the interface and the connection are generated together, and nothing reaches the wearer until the code is running and every binding has passed the server's checks. That "checked before shown" rule is what makes a generated interface safe to put in front of someone's hands.

Third, "generate slowly, execute fast" is the engineering lesson to steal. Hand interaction needs a response on every frame. No model call is fast enough for that, so the agent's job ends at generation: the hand program it produces runs locally, within fixed computational budgets. Any system you build where an LLM sits in a real-time loop has the same flaw — move the model to setup time and compile the fast path.

Finally, the bottleneck is now visible and measurable. Almost 90% of the 32.5 seconds is sequential model calls, which means faster or parallel calls cut the wait directly — and a 91.7% prompt-cache hit rate shows how much of the cost is repeatable context, not fresh thinking. Those two numbers tell the next builder exactly where to optimise.

How YOU can use it

You do not need a Quest 3 to use the pattern. Here is the practical version, starting today:

Step 1: Try the real thing. Clone github.com/Apolotary/controlanything (MIT license) and read the setup. You will need a Quest 3, a laptop, and Meta model keys. Start with one device you already own that speaks MIDI or keyboard shortcuts — a synth, a media player, a browser tab — not a mystery protocol.

Step 2: Write the connector contract before the code. This is the single highest-value habit from the paper, and it works with any coding agent today. Before the agent writes anything, make it declare the contract: named actions, the transport, and how it will prove the device received the message. The prompt template below is illustrative — it is the shape of the discipline, not the paper's code:

Write a connector for this device. First declare the contract,
then write the code — do not skip the contract.

DEVICE: [e.g. OP-1 Field synthesizer over USB MIDI]
ACTIONS (name, what it does, valid range):
  - cutoff: filter cutoff frequency, MIDI CC 74, 0-127
  - resonance: filter resonance, MIDI CC 71, 0-127
TRANSPORT: Web MIDI, port name must match the device.
PROOF: after sending each test value, read the transport
acknowledgement and report it. If there is no acknowledgement,
the action is NOT ready — say so and stop.
RULES: no action reaches the user until its test value was
acknowledged. Keep the connector small: one file, named
actions only, no extra features.

Step 3: Test against the live device and read the acknowledgement. Do not trust "code written" — trust "device replied". Send a test value, watch the log line that says the transport accepted it, and only then bind a control to it. That is the paper's grounded-output rule, and it is the difference between a demo that works once and a tool you can rely on.

Step 4: Compile the mapping out of the model loop. Whatever your agent generates — a gesture map, a hotkey script, a control panel — the finished artefact must run locally without calling the model again. A hand program that needs an LLM per frame is a slideshow; a compiled mapping is a tool. The sketch below is illustrative — it shows the shape of a connector, not the paper's implementation:

// Connector sketch: the ControlAnything idea in miniature
// (Illustrative — not the paper's code.)
const connector = {
  device: "OP-1 Field",
  transport: "webmidi",
  actions: {
    // name: [MIDI CC, min, max]
    cutoff:    [74, 0, 127],
    resonance: [71, 0, 127],
  },
  async test() {
    for (const [name, [cc]] of Object.entries(this.actions)) {
      const ack = await this.transport.send(cc, 64);
      if (!ack) throw new Error(name + " not acknowledged");
    }
    return "ready"; // only now may controls bind to actions
  },
};

Step 5: Steal the design goals for your own agent projects. Even if you never touch a headset: generate at setup time and execute locally; check every output against the live system before showing it; bound every agent run in turns, calls, time, and tokens; and always give the user a stop path that does not depend on the model. Those four rules are portable to any coding-agent tool you build.

Key takeaways

• ControlAnything points at a real device in mixed reality, and a coding agent writes the integration on the spot — no driver, no app, no pre-written glue code.
• The loop is select (SAM 3.1 outlines the object) → ask (speak a goal) → generate and test (agent writes a Node connector and reads transport acknowledgements) → use (AR controls or a per-frame hand program).
• In one real Quest 3 session, 9 of 10 requests reached a ready state across five targets, in a median 32.5 seconds — but success was measured to transport acknowledgements, not confirmed physical effects, and the only wearer was the author.
• The engineering lesson is "generate slowly, execute fast": the model never sits in the real-time hand-tracking loop, and nothing reaches the wearer until its binding passed a live check.
• Your move today: pick one device that speaks MIDI or keyboard shortcuts, make your coding agent declare the connector contract first, and only bind a control after the device acknowledges the test.

Read the original paper: ControlAnything: Generative UI and On-Demand Device Integration for Mixed-Reality Hand Interaction by a Coding Agent (arXiv:2611.08321) by Bektur Ryskeldiev. Link: https://arxiv.org/abs/2611.08321 — code: https://github.com/Apolotary/controlanything