Your Robot's Eyes Are Lying to It — This Encoder-Free Model Lets Actions Read the Pixels Directly

Picture a robot arm reaching for a coffee cup. Its camera sees the cup perfectly well — the rim, the handle, the millimetres of clearance its gripper needs. Then the robot's brain does something strange: it squeezes that whole rich image through a visual encoder, compresses it into a compact summary, and only then decides how to move. By the time the action starts, the exact details the grasp depended on — the edge of the rim, the depth of the handle, the tiny shift as the cup wobbles — have been smoothed away into an abstraction.

The robot fumbles. Not because it is blind, and not because it is dumb, but because the one part of the image that mattered most was thrown away before the decision was made. A new paper from Weichen Fan and colleagues asks a question almost nobody in robot learning had dared to ask out loud: should an action policy inherit a fixed visual summary, or should it get direct access to the visual computation itself? Their answer is a framework called NEO-VLA — a vision-language-action model with no visual encoder at all — and in controlled comparisons it consistently beat encoder-based counterparts, with more scale unable to close the gap. The paper is "From Pixels to Actions — Towards Native Vision-Language-Action Policies" by Weichen Fan and colleagues (arXiv:2611.06918, 4 November 2026), and it topped Hugging Face's Daily Papers on 5 November — the community's signal that this one hit a nerve.

What the researchers did

First, they named the hidden assumption. Nearly every vision-language-action (VLA) model today is built on the same premise: perception is first compressed by a visual encoder, and action reasoning begins only afterwards. That design is borrowed from chatbots and image captioning, where a summary of the scene is enough. But manipulation — actually picking things up, opening things, placing things precisely — hinges on precisely the visual signals most vulnerable to that compression: local geometry, boundaries, depth, and subtle state changes. A cup's rim is a boundary. A handle is local geometry. A wobble is a subtle state change. The encoder throws away exactly what the hand needs.

So they removed the encoder entirely. In NEO-VLA, images are tokenised by lightweight patch convolutional layers and then evolve jointly with language throughout the backbone — vision and language grow up together inside one model, instead of vision being pre-digested and handed over. A flow-matching action expert then draws from instruction-conditioned visual representations at different depths of the network, so the action policy can reach into whatever layer holds the signal it needs, rather than being stuck with the final encoder output.

Then they tested whether the architecture itself was the difference — not the training data, not the model size. Under controlled comparisons, the native encoder-free architecture outperformed encoder-based counterparts. Full fine-tuning of the encoder-based models narrowed the gap but did not erase it. And adding more scale alone could not recover it: the deficit is structural, not a matter of size. You cannot fix a bad information bottleneck by making the model bigger, because the information was already gone.

The most surprising finding came from probing the layers. When the researchers measured where action-critical signals live inside the network, they found the signal often peaks well before the final visual representation. The mid-layers — the ones an encoder-based design throws away entirely — frequently carry the most useful information for action. In other words, the standard design does not just compress the image. It discards the layers where the action-relevant information is strongest. Finally, they put NEO-VLA through diverse manipulation settings and distribution shifts — new objects, new arrangements, new conditions — and report that it achieves stronger and more robust control than its encoder-based counterparts across the board.

Why it matters

First, the bottleneck was hiding in plain sight. Everyone in robot learning assumed the visual encoder was a settled, solved component — a commodity you bolt on before the interesting work starts. This paper shows the encoder is the interesting work: it decides which information survives to the action, and it has been quietly discarding the most action-critical signals all along. The architecture question is not "which encoder?" but "why an encoder at all?"

Second, scale cannot fix a structural loss. The instinct in 2026 is to throw parameters at every problem. NEO-VLA's comparisons show that when information is destroyed upstream, a bigger downstream model just makes more confident decisions about less information. Full fine-tuning helps; scale helps; neither closes the gap. If your pipeline loses information early, no amount of compute later buys it back.

Third, the middle of the network matters more than the end. We tend to assume the deepest, most abstract representation is the most useful one — it is, after all, the one every system is designed to produce. But for control, the signal peaks mid-network, where geometry, edges, and depth are still intact. Any system that only ever sees the final representation is working with the leftovers.

Fourth, robustness is the real prize. Stronger control under distribution shifts — new objects, new scenes, new lighting — is what separates a lab demo from a robot you can actually deploy. Encoder summaries are brittle precisely because they were trained to summarise familiar scenes; when the scene changes, the summary degrades in ways you cannot inspect. A policy that reads visual computation at multiple depths has more to fall back on when any single view of the scene goes wrong.

Finally, it is a design pattern, not just a model. NEO-VLA is a research framework, not a product you can download today — the abstract announces no public code release. But its core idea is portable: wherever a pipeline compresses perception before action, ask what the compression is costing you, and whether the decision-maker could reach into earlier representations instead.

How YOU can use it

You cannot download NEO-VLA from the paper, but the idea is immediately usable whether you build robots, work with vision models, or design any system where perception feeds action:

Step 1: Find where your signal peaks. If you work with a vision model, do the paper's layer-probing experiment on your own system: attach a small probe to each layer and ask which layer best predicts the thing your downstream task actually needs. You may find, as the authors did, that the answer is a middle layer — and that you have been building on the wrong one. The illustrative sketch below shows the shape of the test.

Step 2: Stop treating the encoder as sacred. In your next vision-and-action design, ask the NEO-VLA question explicitly: does the action policy need a compressed summary, or does it need the visual computation itself? If the task depends on fine geometry, edges, depth, or small changes — grasping, inspection, alignment, tracking — a fixed encoder output is a liability. Design for multi-depth access from the start.

Step 3: Test under distribution shift, not just on your test set. The paper's headline result is robustness, not a leaderboard number. Evaluate your policy the same way: same task, new object, moved lighting, cluttered background, different camera angle. If performance collapses when the scene changes, your encoder is summarising a world that no longer exists.

Step 4: Don't just scale your way out. Before you double your model size to fix a control failure, check whether the failure is an information problem or a capacity problem. If the signal your action needs never survived the encoder, a bigger model is money spent on a louder guess. A one-hour probing experiment can save weeks of wasted training.

Step 5: Watch for the code and the follow-ups. Check the arXiv page for code links as they appear, and expect encoder-free VLA designs to multiply: this is the kind of simple, structural idea that spawns a family of follow-up papers. When you evaluate the next robot model, ask one question first — does its action policy see the pixels, or only the summary?

ILLUSTRATIVE sketch — not code from the paper:

# Layer-wise probe: which layer holds the action-critical signal?
# (Illustrative sketch — not the paper's code. The paper's finding:
#  action-critical signals often peak BEFORE the final representation.)

def probe_layers(model, images, action_labels):
    # Attach a tiny linear probe to each layer's visual features
    scores = {}
    for layer in range(model.num_visual_layers):
        feats = model.visual_features(images, layer=layer)
        # Train a small probe: features -> action outcome (e.g. grasp success)
        probe = train_linear_probe(feats, action_labels)
        scores[layer] = probe.accuracy_on_held_out()
    best = max(scores, key=scores.get)
    print("Signal peaks at layer", best, "of", model.num_visual_layers)
    print("Final layer scores", scores[model.num_visual_layers - 1],
          "— if the peak is earlier, your policy is reading the leftovers.")
    # If the peak is mid-network, your action head should draw from
    # that depth, not only from the final encoder output.
    return scores

One guardrail, from the paper's own logic: a probe result is only as honest as the distribution shift you test it on. Probe on your training scenes and you will flatter the final layer. Probe on new objects, new lighting, and new arrangements — the conditions the paper tested — and the mid-layer peak shows up. Test where the robot will actually live, not where it was trained.

Key takeaways

• Almost every robot action model today compresses vision through an encoder first — and that compression discards exactly the signals manipulation needs: local geometry, boundaries, depth, and subtle state changes.
• NEO-VLA removes the encoder: images are tokenised by lightweight patch convolutions and evolve jointly with language, while a flow-matching action expert reads visual representations from multiple depths.
• Under controlled comparisons, the native encoder-free design beats encoder-based counterparts — full fine-tuning narrows but does not erase the gap, and extra scale alone cannot recover it.
• Layer-wise probing shows action-critical signals often peak well before the final visual representation — the middle layers hold what the action needs.
• Across diverse manipulation settings and distribution shifts, NEO-VLA achieves stronger, more robust control.
• The practical move is yours: probe your own layers, design for multi-depth visual access, test under distribution shift, and never let a bigger model paper over an information bottleneck.

Read the original paper: From Pixels to Actions — Towards Native Vision-Language-Action Policies (arXiv:2611.06918) by Weichen Fan, Haiwen Diao, Penghao Wu, Siyuan Hu, Ziqi Huang, Tianqi Liu, Mingze Sun, Ziyi Wang, Zhongang Cai, Hanming Deng, Quan Wang, Lei Yang, Lewei Lu, Fangzhou Hong, Zhaoxi Chen, Dahua Lin and Ziwei Liu. Link: https://arxiv.org/abs/2611.06918