VLA API: what vision language action models are and how you call one
A vision language action model, or VLA, gives a robot three skills in one neural network. It looks at camera images. It reads an instruction. It outputs the commands that move the arm. A VLA API is the hosted form of that model. Your code sends images, an instruction, and the robot's state. It gets back an action chunk to run. This page explains the model class in plain words, decodes the terms around it, names the open weight models of the action lane's written record, and shows what a call looks like. The honest state comes first: three of the lane's eight models serve on this platform's action route, confirmed by the October 26, 2026 requests; the other five are written records beyond the list. The page stands on its own anyway. The model class, the call contract, and the member set are real today.
Last verified: 2026-10-05
What a vision language action model is
The name lists the three parts in order. Vision: the model reads camera frames. Language: it reads an instruction such as "put the cup on the plate". Action: it writes robot commands rather than text.
Google DeepMind introduced the class in July 2023 with RT-2, a vision language model adapted for robot control (paper, 2023). The idea spread fast. A May 2025 survey counts more than 80 VLA models from the previous three years (survey, 2025). The field spans factory arms, humanoids, farm robots, and driving research.
The common design has two stages. A pretrained vision language model works as the eyes and the reasoning core. It turns images and the instruction into an internal representation. An action decoder turns that into continuous actions. Those run directly on the robot. What comes out is concrete: joint angles, end effector speeds, gripper open or close. The NVIDIA glossary describes the output as exact joint angles or steering angles (NVIDIA).
Most of these models emit an action chunk, not a single move. An action chunk is a short run of future actions. It covers the next fraction of a second of control. The idea comes from ACT, the Action Chunking with Transformers policy from 2023. ACT showed that predicting whole sequences steadies fine manipulation (ACT paper, 2023). Chunks matter for hosting. A robot can call, run the chunk, then call again with fresh images.
One newer branch adds explicit reasoning. NVIDIA calls this Reasoning VLA. The model breaks a task into subproblems and explains its steps before it acts. Standard VLAs map perception to action in one pass. Reasoning variants add a thinking phase, at the cost of response time. Both shapes output the same kind of action data. The open side now has one too. MolmoAct2-Think, the reasoning variant of the MolmoAct2 family, trades speed for depth tokens and tops that family's LIBERO table at a 98.1 average.
The vocabulary decoder: LLM, VLM, VLA, world model
The acronyms stack, and search results mix them freely. Here is the decoder.
| Term | Reads | Writes | In one line |
|---|---|---|---|
| LLM | Text | Text | A language model, no cameras, no body |
| VLM | Images and text | Text | Adds eyes to the language model |
| VLA | Images, text, robot state | Actions | Adds hands to the vision language model |
| World model | Scene and optional action | Predicted next scene | Predicts what happens, does not act |
Each layer builds on the one above. A VLA is usually a VLM backbone with an action decoder attached. The VLM contains an LLM. The frontier labs use the same words. Google describes its Gemini Robotics model as a vision language action model that turns vision and language into motor control (Google DeepMind).
One more split lives below the decoder row, and it explains most of the speed numbers you will read. Early VLAs, RT-2 and OpenVLA among them, write actions as discrete tokens, reusing the machinery the language half uses for words. The newer line attaches a continuous action expert instead: a small network, trained with flow matching or diffusion, that refines a whole chunk over several denoising passes. Pi 0.5, SmolVLA, and MolmoAct2 all take the expert path. Token decoders are simpler to train. Expert decoders run faster at inference and move more smoothly. The 26 times faster action generation that the OpenVLA-OFT paper reports comes from carrying OpenVLA across that split.
The world model row is the one people trip over, because the terms appear together in research. A world model predicts results. A VLA chooses actions. The LeRobot library from Hugging Face lists them as separate policy classes in its docs (LeRobot). Many teams run a world model as the planner in front of a VLA as the actor. One more overlap: the same VLA words now cover driving research too. That is why "vla model" searches pull car papers next to robot arm tutorials. The models in this platform's lane are the robot arm kind.
What a VLA API is
Strip the papers away and a VLA API is a simple trade. You send what the robot senses. It returns what the robot should do.
The request has three parts. One or more camera images, sent as image parts. An instruction, sent as a short string. And the robot's state, sent as an array of numbers. Those are the joint positions and other readings that say where the arm is right now. A robot with no state sensors can send an empty array. The transport is a multipart form, the same shape a web page uses to upload a file. Nothing in the request needs a robotics SDK.
The answer has three parts too. The action chunk, an array of arrays of numbers, ready for the motors. A norm stats block, which states the scaling the model applied, so your controller can check or undo it. And a usage object with the call count and the cost of the call. Chunk length is a property of the model you call, not a request knob. Some models predict long chunks, some short.
Chunk length is also what turns model speed into call rate. A chunk of 30 steps on a loop ticking 30 times per second covers one full second of motion, so a robot working that way makes one call per second while it runs. Shorter chunks react faster and bill faster. The pi 0 paper measured its loop at 50 hertz with re-inference every 25 actions, twice a second. Your task's tolerance for surprise picks the number.
That contract is the whole product surface, and it is small on purpose. A robot control loop wants three things from its brain endpoint: low delay, an honest error code, and a price it can predict. Metering by the call fits a control loop better than metering by the token. A call is the thing your robot schedule actually makes. The full field reference for the action route lives at the run-action reference page. It lists every field, error code, and limit. The route's design notes live on the action route page.
The VLA family on this platform
The member panel on this page is machine-generated from the platform's model data. It renders a model only when that model serves. Today the panel carries three: SmolVLA, the OpenVLA fine-tune, and ACT, all on the public list since October 26, 2026, confirmed by that day's requests. The other five rows remain the written capability record: what each model is, what it was trained on, and what its publishers measured.
- pi05: the pi 0.5 model from Physical Intelligence, a flow matching VLA built to work in places it never saw. Its paper reports cleaning kitchens and bedrooms in homes the robot never visited in training (pi 0.5 paper, 2025).
- pi05-droid: the same base model, fine-tuned on the DROID dataset. That corpus holds 76,000 recorded runs across 564 real scenes (DROID). The publisher calls this tune fast, with good language following.
- pi05-libero: the LIBERO benchmark tune of pi 0.5, from the same public checkpoint line. LIBERO is 130 hand tasks in four suites. It is the standard simulation benchmark for this class.
- smolvla: a 450 million parameter VLA from Hugging Face, trained only on open community data. It runs on consumer hardware, including a laptop. Its authors report it beating larger models and ACT on simulation and real arms alike (SmolVLA, 2025).
- openvla-oft: OpenVLA plus the OFT fine-tuning recipe. OpenVLA is a 7 billion parameter open model trained on 970,000 robot episodes. The OFT recipe lifts the LIBERO average from 76.5 to 97.1 percent, and speeds up action generation 26 times (OpenVLA-OFT paper, 2025).
- act: the original action chunking policy. ACT learned six hard two handed tasks, such as battery slotting, at 80 to 90 percent success from just 10 minutes of hand recordings (ACT paper, 2023). Small, fast, and still a strong baseline.
- molmoact2-so100-101: MolmoAct2 from the Allen Institute for AI, an Apache 2.0 action reasoning model. This is its tune for the SO-100 and SO-101 low cost arms, with absolute joint pose control (MolmoAct2).
- x-vla-0.9b: a 0.9 billion parameter model built to cross robot bodies. It was pretrained on 290,000 episodes across seven platforms. It adapts to new hardware through learned soft prompts, not full retraining. Its LIBERO checkpoint reports 93 percent success (X-VLA in LeRobot docs).
Eight models, three sizes of budget, one shared call contract. The models index is the live catalog.
The sizes span an order of magnitude, and the memory classes follow them. SmolVLA is the smallest at 450 million parameters and runs on laptop class hardware. X-VLA sits near a billion. The pi 0.5 family carries 3.6 billion, and MolmoAct2 5.44 billion. OpenVLA-OFT is the heavyweight at 7 billion. For training, the official LeRobot compute guide groups them by memory need: ACT in roughly 2 to 6 GB, SmolVLA in 10 to 16 GB, the pi 0.5 family in 24 to 40 GB, with a 24 GB card already tight at a batch size of one. Serving is lighter than training everywhere in the list. The rule a buyer can keep: match the memory class to the model before falling for a benchmark number.
Embodiment and norm stats
An embodiment is a robot body: how many joints, where the cameras sit, what the gripper is. VLAs are embodiment sensitive. The same checkpoint moves different bodies only after adaptation. The publishers say so themselves. The openpi notes state that its expert checkpoints were tuned on specific platforms. They may or may not work on your robot. The MolmoAct2 card lists hardware, cameras, calibration, and action conventions as things your results depend on.
Norm stats are the other half of the same problem. Action data is stored normalized: each joint's range scaled to a standard band. A model trained on one robot's scaling will write wrong numbers on another robot unless the stats match. The openpi workflow has you compute these numbers before training. LeRobot ships them inside its checkpoints. When they are wrong or missing, the failure is ugly and physical. That is why the norm stats guide listed with this page exists. It is also why the action route returns the applied norm stats with every answer.
Cross-embodiment research attacks this cost directly. X-VLA trains small soft prompts per robot and keeps the backbone fixed. A new arm adapts without retraining the whole model. Pi 0.5 co-trains across many robots to generalize. The direction of the whole field is fewer per-robot secrets, not more.
What it costs to run a VLA
This section sets vocabulary, not numbers. The action lane meters by the call. One served request, one billed call, and the wallet pays. The wallet is prepaid, and the balance is the hard stop. When it empties, the next call is refused with a named error instead of quietly draining your deposit. A refused call is never billed.
The call is also the honest unit on the other side of the trade. Serving a VLA means holding a large model in memory whether or not anyone calls. A hosted lane pays that residency so you do not, and that is the real anatomy of a per-call price: the memory rent, the machine's time, and the calls that machine can answer per second at the chunk latency your loop needs. The break-even guide listed with this page works the local side of the same math: the card, the power, and the quiet hours of your own time. One number usually decides it, hours of robot runtime per month.
Every rate that exists renders on the pricing page, generated from the platform's model data. That page is where any change lands first. Three of this lane's models serve, and their rates render there. The table there follows the list exactly, and the models index is the live catalog. That is also why this page types no numbers.
Running one yourself
Every model above ships open weights, and the small ones truly run at home. SmolVLA trains on a single consumer graphics card. ACT and X-VLA sit in the same weight class. The big ones, OpenVLA-OFT and the pi 0.5 family, need a serious machine to train but less to serve. LeRobot, the open robotics library from Hugging Face, implements all of these policies in plain PyTorch. It ships training scripts, dataset tools, and deployment helpers. On Apple silicon, an M-series Max chip trains ACT slowly, overnight, and the official compute guide carries no Apple path at all for the bigger families. If you own an SO-101 arm, the community path from box to trained policy is written down step by step.
So why a hosted lane at all? The local road is real, and the guide listed below does the break-even math honestly. Hosting earns its keep on the parts around the model. That means an endpoint that stays up through a long session, metering and spend history you did not have to write, and someone else holding the inference stack together at 2 a.m. It also means the freedom to try a second model by changing one string in a request. For teams whose robot is a product rather than a research project, that trade is often worth paying for.
Getting started
- Request an invitation through the console. The beta is invite-only for now. The wallet, the keys, and the spend history live there. An account opened today is ready the day this lane opens.
- Read rates once, on the pricing page. The pricing page carries the rates for every listed row, generated from the platform's own model data.
- Read the route reference. The run-action reference documents every field of the action contract. The action route page carries the design notes.
- Pick your model and read its page. Each model page in the family carries its own measured story, its training sources, and the exact request shape.
Robots also need to see, not just act. The sibling lane for that is machine vision: segmentation and depth over the same kind of camera frames, under the same wallet. Everything else the platform serves is mapped on the models index.