Vision language action models explained: the 2026 map

A vision language action model, or VLA, is one neural network that looks through cameras, reads an instruction, and outputs the motion commands for a robot. This guide explains the class in plain words. It covers where the class came from, how it turns frames and text into action chunks, which models exist in 2026, and what they still cannot do. Every number carries its source and its date. The tour maps each model family to its page on this site. The action lane on this platform is not on the public list. The class, the model set, and the call contract are real today. You can shape a client against the published contract now.

One scoping note before the definitions. This guide is about robot arms and humanoids: models that move a physical machine. The same three words also appear in driving research, and one large vendor applies them to autonomous vehicles. The ideas overlap. The models do not. Everything below is the robot kind, and the lane page for those models on this site is the VLA category page.

The short answer

A VLA takes three inputs: camera images, a plain language instruction, and the robot's own state as numbers, which means joint positions and gripper readings. It returns actions: joint targets, end effector moves, gripper open or close. There is no separate vision module, no planner in the middle, and no hand-written control rule. One trained network covers the whole path from pixels to motion.

It is not a chatbot with a camera. A language model answers you in text. A VLA answers you in motion. It is also not a world model, which predicts what a scene will look like next instead of deciding what to do next. The decoder table below separates these cleanly, because search results mix them freely.

The lineage is short and worth knowing. In April 2023, ACT introduced action chunking. It predicts a short run of future actions at once, which steadied fine manipulation on low cost arms. It reached 80 to 90 percent success on six hard tasks from 10 minutes of recordings (ACT paper, 2023). In July 2023, Google DeepMind fine-tuned a large vision language model on robot data. It expressed actions as text tokens, and it named the class: "we refer to such category of models as vision-language-action models" (RT-2 paper, 2023). From there the field grew fast. A May 2025 survey counted more than 80 VLA models from the previous three years (survey, 2025).

The vocabulary decoder: LLM, VLM, VLA, world model, reflex

Two questions recur in every search session on this topic: what is the difference between an LLM and a VLA, and what is the difference between a VLA and a VLM. Here is the whole answer in one table.

Each layer builds on the one above it. Most VLAs start from a pretrained VLM and add an action decoder. The VLM contains an LLM. Frontier labs use the same words. Google DeepMind describes its Gemini Robotics line as a vision language action model working together with an embodied reasoning model (Google DeepMind). The world model row is where people trip, because the terms appear side by side in papers. A world model predicts. A VLA decides. Many teams run both. One plans, one acts.

Three more words from this field will appear in every model card you read, and each has a plain meaning.

An action chunk is a run of future actions returned at once. The model does not output one move per call. It outputs the next stretch of motion, the robot works through it, and the next call arrives with fresh camera frames. Chunks are why a network round trip does not stall a robot stepping at 30 frames per second. Chunk length belongs to the model, not to the request.

An embodiment is a robot body: how many joints, where the cameras sit, what the gripper is. Models are embodiment sensitive. The same weights move a different body only after adaptation, and the publishers say so themselves.

Norm stats are the scaling numbers that map your robot's units onto the small range the model trained on. Wrong stats do not crash anything. The arm just moves wrong. That failure class has its own guide on this site, VLA normalization stats, and the per robot body tables live in the robot embodiment stats reference.

One more phrasing catch, because it shows up in search suggestions: "vision action language models" is the same thing with the words reordered. The standard order is vision, language, action, and the acronym follows it.

How one works, from camera frames to motion

Strip the papers away and a VLA call is a fixed trade. You send what the robot senses. It returns what the robot should do.

camera frames ----\
instruction --------> [ vision language core --> action decoder ] --> action chunk
robot state -------/                                                        |
                                                                            v
                                              robot executes the chunk while the next one is computed

The inside has two stages, and the split is common to almost every model in the class. A pretrained vision language model works as the eyes and the reasoning core. It turns images and the instruction into an internal representation. An action decoder turns that representation into continuous actions for the motors. What comes out is concrete: joint angles, end effector velocities, gripper commands.

The action decoder is where the families differ, and there are three main designs.

Tokenized actions turn motion into words. Each joint value is sorted into one of 256 bins, and the model emits actions as if it were writing text. RT-2 did this first, and OpenVLA inherited the idea. It keeps the model simple, because the output layer is the same one the language model already had. The cost is precision: binned values can blur fine motion.

Direct regression adds a small head that predicts continuous numbers straight out. It is fast and simple, and it suits smaller policies that do not carry a language core.

Flow matching treats the action chunk as something to refine. The decoder starts from noise and denoises it into a smooth trajectory, in a few steps. The pi 0 line brought this design to robot control, generating continuous actions at up to 50 Hz (pi 0 paper, 2024). Today it is the mainstream choice for dexterous work. The pi 0.5 family uses it. SmolVLA uses it. GR00T N1.7 describes its decoder as a flow matching action transformer over a chunk of actions (GR00T N1.7 model card). MolmoAct2 uses it with eight refinement steps (MolmoAct2 in LeRobot).

Then there is the loop. A robot control loop wants fresh decisions at a steady rate. One measurement study frames the rates the field uses. It counts about 10 decisions per second as acceptable for a policy. It counts 100 per second as high performance (VLA-Perf paper). The chunk is what makes those rates reachable. The arm executes actions it already holds while the next chunk is computed. That computation can sit on a graphics card under the desk, or behind a network call. The reflexes are a different story. Servo loops, balance, and the emergency stop run next to the motors, always, at rates no network can promise. The systems guide on Jetson versus cloud robot inference works this budget layer by layer.

The family tree, 2023 to 2026

The class is four years old and already crowded. Here is the shape of it, then the tour of the models this site carries pages for.

2023 was the founding year. RT-2 showed that web-scale pretraining transfers to robot control, with emergent skills the robot was never taught (RT-2 paper, 2023). ACT, published three months earlier, supplied the chunk habit the whole class kept.

2024 was the open year. OpenVLA put a 7 billion parameter model with open weights on top of 970,000 real robot episodes, beating the closed RT-2 X model by 16.5 points across 29 tasks with 7 times fewer parameters (OpenVLA paper, 2024). The training corpus behind that wave, Open X-Embodiment, pooled data from 22 different robots (OXE paper, 2023). Physical Intelligence introduced pi 0 with the flow matching decoder.

2025 split into two directions. Closed labs pushed scale and humanoids. Figure's Helix ran a two-system design: a slow reasoning half and a fast motor half. It controlled the whole upper body of a humanoid, 35 joints at 200 Hz, from about 500 hours of teleoperation data (Helix, 2025). Google DeepMind's Gemini Robotics line paired a VLA with an embodied reasoning model. Open teams pushed efficiency. SmolVLA trained a 450 million parameter model on community data that runs on consumer hardware (SmolVLA paper, 2025). The OpenVLA-OFT recipe lifted the LIBERO benchmark average from 76.5 to 97.1 percent. It also made action generation 26 times faster (OpenVLA-OFT paper, 2025).

2026 brought the small and the cross-body turn, plus explicit reasoning. NVIDIA now defines a whole branch, Reasoning VLA, where the model breaks a task into subproblems and explains its steps before acting (NVIDIA glossary). Ai2 released MolmoAct2, an Apache 2.0 family of action reasoning models (Ai2 repository). X-VLA showed a 0.9 billion parameter model adapting across robot bodies with learned soft prompts instead of retraining.

This is the map of the models with pages on this site. Each row links to its page, which carries the full story: training data, numbers, and the exact request shape. One shared fact first, because it is true for every row: none of these is on the public list.

The last two rows are the front half without the back half. They see and reason. They do not move anything. A modern robot stack often pairs exactly these layers: one model to understand the scene, one to act.

Two closed families complete the map, and both matter to expectations. Helix and Gemini Robotics set the public's idea of what a robot brain can do, through demo videos and press. The open rows in the table are the ones you can download, inspect, fine-tune, and host. One open family carries a trap. GR00T's weights, including N1.7, ship under a license that does not permit commercial use. The limit passes into your fine-tune. The GR00T license guide on this site quotes the file and lists the open alternatives.

Robots also need to see, not just act. The sibling lane on this platform covers segmentation and depth from the same kind of camera frames, and its guide walks the model set: robot vision, segmentation and depth. The machine vision category page is the lane surface.

What they cannot do yet

An honest explainer needs this section, because the demo videos are cherry picked and the papers report their best suites. Four limits matter in 2026.

Generalization outside training is still the open question. The strongest published result is pi 0.5 cleaning kitchens and bedrooms in entirely new homes (pi 0.5 paper, 2025). That is the high water mark for the class, not the average experience. The base checkpoints behind the pi family were trained on more than 10,000 hours of robot data (openpi repository), and the paper's own framing is that generalization "remains an open question".

Simulation numbers do not transfer as-is. The clearest pair of numbers in the field: OpenVLA-OFT reports a 97.1 percent average on the LIBERO simulation suites (paper), while the same pi 0.5 family reads 44.4 percent, plus or minus 4.3, on a real pick and place rig run by an independent leaderboard (PhAIL, read 2026-10-05). Different models, different rigs, so the two numbers are never averaged or compared directly on this site. The spread between simulated suites and real hardware is the point. On that real rig, human teleoperation reads 99.0 percent. GR00T N1.6 reads 46.1. ACT reads 42.5. SmolVLA reads 16.3. Every number is public, dated, and carries its interval. How to read such numbers, and how to run your own evaluation, is the LIBERO evaluation guide.

Rollout counts are small. A May 2026 audit of the field found that real world evaluations still rest on binary success at a fixed timeout, with 25 rollouts or fewer per condition, almost always without confidence intervals (PhAIL paper, 2026). A single success rate quoted without its interval is noisier than it looks.

Latency and contact. Flow matching decoders are fast enough for smooth motion, and chunks hide network delay. But force-sensitive assembly and reactive grasping want control rates the class is still growing into. The models are strongest in the approach phase of a task, before contact. The community's practitioner voice matches the papers. Teams that ship industrial robots use learned models for the semantic half, understanding what is in front of the robot. They keep explicit geometry and planning for the physical interaction. And whatever the model, the emergency stop belongs to the machine, not the network.

Three ways to run one

Every model in the table above ships open weights. The pi family base checkpoints alone represent more than 10,000 hours of robot data, pre-trained and published (openpi repository). Three paths cover how people actually run them.

Self-host on your own graphics card. The small models are genuinely small: SmolVLA trains on one card and runs on consumer hardware (SmolVLA paper, 2025). Two open toolkits carry most of the work: openpi from Physical Intelligence, and LeRobot from Hugging Face. If you own a SO-101 arm, the full community path from box to trained policy is written up in the SO-101 cloud brain guide.

Put the brain in an edge box. A compute module bolted to the robot removes the network from the loop. The tradeoffs, part classes, and the latency budget per layer are the subject of the Jetson versus cloud guide.

Call a hosted route. Your client program sends images, instruction, and state to an endpoint, and receives action chunks, with the bill metered by the call against a prepaid wallet. That is the published shape of the action lane on this platform: not on the public list. The break-even guide does the math on when hosting beats your own card, without hand-typed numbers.

What a call looks like

The contract for this platform's action route is fixed and published. You can build against the shape today, before anything serves. The request is a multipart form with three kinds of content. One is the model name. One is the instruction, a short string. One is the robot state, a JSON array of numbers, and an empty array is legal for robots without state sensors. Camera images ride along as file parts. The answer has three parts. The action chunk. A normalization block stating the scaling the model applied. And a usage object with the call count. Metering is per served call, so a looping robot has a predictable bill. A prepaid wallet at zero is a refusal, not an open tab.

import json, requests

BASE = "https://your-endpoint.example/v1/action"   # the lane's address
KEY  = "your-key"

def step(frame_jpg, state):
    r = requests.post(BASE, headers={"Authorization": "Bearer " + KEY},
        files={"image": ("front.jpg", frame_jpg, "image/jpeg")},
        data={"model": "pi05",
              "instruction": "pick up the green cube",
              "state": json.dumps(state)})
    r.raise_for_status()
    body = r.json()
    return body["actions"], body["normalization"]

while True:
    frame, state = read_cameras_and_joints()
    actions, norm = step(frame, state)
    for a in actions:                # work the chunk
        command_joints(a)

The field by field reference, with every error code and limit, is the run-action reference. The design notes live on the action route page. The perception sibling, the vision route, takes image parts in and returns masks and depth. Rates for the listed lanes render on the pricing page. That is why this guide types no numbers. Sign up and create a key now. The client above then needs one line changed, the address.

Where to go next

This is the last page of the robot set on this site, which makes it the crossroads. Every next read is one hop from here. Pick by your situation.

• You want to know if a model is actually good: start with the LIBERO evaluation guide. It reads the benchmark suites, the honest leaderboards, and the error bars.
• Your arm moves wrong and nothing crashes: read the normalization stats guide. It is the most common failure class in the field.
• You are choosing between your own card and a hosted route: the break-even guide is the math, and the Jetson versus cloud guide is the systems view.
• You have an SO-101 on the desk: the SO-101 cloud brain guide walks the whole loop, record to rollout.
• You are eyeing GR00T because a vendor course teaches it: read the GR00T license guide before you spend a month on it.
• Your robot is an SO-100 or SO-101 and you want the arm-native model: the MolmoAct2 guide is that model's own deep page.
• Your robot needs to see before it acts: the robot vision guide covers segmentation and depth.

For the survey and course tails of the query, the reading order is short. The May 2025 survey is the map of record for the field through early 2025. It covers more than 80 models in five themes (survey). The LeRobot documentation is the practical on-ramp, with one page per policy. And the VLA lane page on this site is where the callable versions of these models will appear. The whole model set, every lane, and every page on this platform is indexed on the models page.

Common questions

What exactly is a vision language action model? One neural network that takes camera images, a text instruction, and the robot's state numbers as input, and outputs robot actions directly: joint targets, end effector moves, gripper commands. It is built by extending a pretrained vision language model with an action decoder and training the pair on recorded robot episodes. Google DeepMind named the category with RT-2 in July 2023 (paper).

What is the difference between an LLM and a VLA? A large language model reads text and writes text. No cameras, no body. A VLA keeps the language skill, adds camera input and robot state, and writes motion commands instead of prose. The relation is literal descent: most VLAs start from a pretrained vision language model, which itself contains an LLM, and extend it with action outputs.

What's the difference between VLA and VLM? A vision language model sees images and reads text, and answers in text. Ask it to describe the scene and it writes a sentence. A VLA adds two things: the robot's own state as input, and an action decoder as output. In most current designs the VLM is the front half of the VLA. This site carries a VLM built exactly for that front-half job, the molmo2-ER page above, which perceives and points but moves nothing.

What are the most popular VLA models? By citation weight and community presence: the RT-2 line from Google DeepMind, OpenVLA and its OFT tune, the pi 0 and pi 0.5 family from Physical Intelligence, SmolVLA from Hugging Face, Helix from Figure, GR00T from NVIDIA, Gemini Robotics from Google DeepMind, and the 2026 entrants MolmoAct2 and X-VLA. The table above links the ones with pages here, each with its own numbers and sources.

What is the best VLA model? No honest page answers that with one name, because the answer depends on your robot, your task, and your control rate. The claims that exist are scoped: OpenVLA-OFT holds a 97.1 percent LIBERO average in simulation (paper), while an independent real rig reads the same pi 0.5 family at 44.4 percent on pick and place (PhAIL). Suite named, rig named, date named. That is what a real answer looks like in this field, and the evaluation guide teaches you to read every such claim.

What are vision action language models? The same thing. The words get reordered in searches and in some vendor copy. The standard order is vision, language, action, matching the acronym VLA: what the model reads, what else it reads, what it writes.

What does VLA stand for? Vision language action. The name lists the model's inputs and outputs in order: vision in, language in, action out. Google DeepMind's RT-2 paper introduced the term in 2023 (paper).

How do you train a VLA model? The standard path is imitation learning. Record episodes by driving the robot by hand, with cameras rolling and the instruction attached. Compute the normalization stats for your robot. Then fine-tune an open checkpoint on your episodes, using openpi or LeRobot. Fifty episodes with fixed cameras and consistent grasping is the published floor for a first policy, and the failure modes of skipping it are physical. The SO-101 guide above walks the whole loop on real hardware.

Is there a vision language action models survey or course? The survey of record is the May 2025 review covering more than 80 models (survey), with a public repository of papers. For a course-shaped path, the practical on-ramp is the LeRobot documentation, one page per policy, plus this site's set: the explainer you are reading, the lane page, and the guides listed above, which together cover definition, evaluation, data, serving, and cost.

How much does it cost to run a VLA? It depends on the route, and the honest answer lives in two places rather than one number here. For the self-hosted route, the break-even guide works the math against your own card. For the hosted route on this platform, metering is per call against a prepaid wallet, and the lane is not on the public list, so there is no rate to quote: the pricing page carries every listed lane's rate.

The short version

A vision language action model is one network that turns cameras, instructions, and robot state into motion. ACT taught the class to chunk in 2023, RT-2 named it the same year, and the open era from 2024 gave it model families you can download, fine-tune, and host: the pi 0.5 line, OpenVLA and its OFT tune, SmolVLA, ACT itself, MolmoAct2, and the cross-body X-VLA. The vocabulary stack is simple once separated: LLM writes text, VLM writes text about images, VLA writes actions, world models predict, reflexes guard. The honest limits are named and dated: generalization in new places is the open question, simulated suites outrun real rigs, rollout counts are small, and contact-heavy tasks still favor classical control underneath. You can run every model in the table yourself today, on your own card or an edge box, and this platform's hosted action lane follows the same contract the run-action reference publishes, beyond the public list. Sign up so a key is waiting.

Last verified: 2026-10-05