What is a VLM in robotics: the definition, the parts, the VLA line

A VLM is a vision language model: a network that reads images and text together and writes text back. That last word carries the whole robotics story. A VLM can describe a bin, point at a part, or write the next step of a plan. It cannot command a motor, because its only output is words. This page defines the class from its three parts, traces the 2021 to 2023 lineage that named it, draws the exact line where a VLM becomes a VLA, and lists the VLM-class rows on this platform's public model list, dated 28 October 2026.

Last verified: 2026-10-28

The short definition

A vision language model, or VLM, is a neural network with one contract. Images and text go in. Text comes out. The name states the two input modes. Vision is one. Language is the other. The output mode is language alone. That one fact decides where the class fits in a robot stack, and where it stops.

The term is older than the robot use. The Flamingo paper of April 2022 called its own model "a family of Visual Language Models (VLM)" (Flamingo, April 2022). The model reads mixed streams of images, text, and video, and answers in words. Robotics did not invent the VLM. Robotics borrowed it for one narrow corner, where it earns more than anywhere else. The corner is this: turning what a camera sees into language that a planner, a policy, or a person can act on.

Generic explainers cover the chat side of these models well. This page stays in the robotics corner. It gives the definition a robot stack uses, the parts under the hood, the line to VLA models, and a way to read a robotics VLM card before you spend money on one.

The three parts of the standard VLM

Almost every VLM in production is three parts in a row.

The first part is a vision encoder. It cuts each image into patches and treats the patches like tokens. Early models used convolution encoders for this job. Current models use a vision transformer. Cards name the exact encoder, which is why you see SigLIP or a ViT variant in the architecture line.

The second part is a connector. It maps the vision vectors into the language model's input space, so one network can process both. Designs differ most here. BLIP-2 used a light querying transformer it called a Q-Former. It sat between a frozen image encoder and a frozen language model. The design beat the much larger Flamingo80B on zero-shot VQA while training 54 times fewer parameters (BLIP-2, January 2023). LLaVA connected a frozen CLIP vision encoder to a Vicuna language model with one trainable linear projection layer (LLaVA, April 2023).

The third part is the language model. It reads the image vectors and the text together and writes the output. Since the LLaVA work in 2023, these models are tuned on instruction data, so they follow requests like "describe this bin" rather than just finish captions. LLaVA's instruction data was itself machine written, by GPT-4 from images (LLaVA, April 2023).

One lineage table holds the parts and the dates:

A robot builder reads this table one way. The parts are standard now. When a new robotics VLM appears, the question is never whether it has a new kind of brain. The question is which encoder, which connector, which language model, and what data it was tuned on. That is what a model card is for. The card-reading section below works through two of them.

What a VLM gives a robot

Four skills matter in the robotics corner. All four end in words or pixel points. None ends in motion.

Open-vocabulary perception is the first. A classic detector knows the classes it was trained on and nothing else. A VLM answers about anything you can name in words. Ask for the hex key, the red bin, or the part that does not belong. The model answers about objects it was never trained to name. For a robot that meets new objects daily, that is the gap between a fixed cell and a flexible one.

Pointing is the second. Some VLMs output pixel points, not just words, so a robot can look and grab. The skill has its own suites now. The Molmo2-ER card lists "pixel-accurate pointing" among the embodied skills that action models downstream depend on, with the Point-Bench suite on its results table (Molmo2-ER model card, read 28 October 2026). Pointing is the bridge skill. It is perception with coordinates, and coordinates are what a picking step eats. The Molmo2-ER model page carries the full sheet for that row.

Scene text for planners is the third. A VLM turns a frame into a sentence a slow loop can reason over: "the left gripper is empty, the bolt is under the rag." The sentence feeds a planner model, a rules engine, or a human supervisor. Text is the interface.

Labeling and narration are the fourth. The same model that perceives can name what happened. It can say which step failed, what changed between two frames, or why a scene looks wrong. Builders use this for data work, writing tags and notes over robot footage at scale.

Notice what is absent from all four. Motion is absent. Every skill ends in words or coordinates. The step from words to motor commands is a different model class, and that line is worth drawing exactly.

The two-stage pattern: describe, then extract

Robot stacks that use a VLM for perception usually add one more small model behind it. Control code wants typed fields. A VLM writes free prose.

The pattern has two stages. The VLM describes the scene in text. A small extraction model then pulls typed fields out of that text. Fields are things like object names, counts, or states. The rest of the stack can branch on a field without parsing English.

The extraction half has a known open tool class in GLiNER-style models. The card states they are "capable of identifying any entity type" from a plain list of labels passed at call time, with no retraining per label set (GLiNER model card, read 28 October 2026). On this platform the row is gliner-extract, a classification route metered per search. For the full trade, the GLiNER versus LLM extraction guide works the comparison end to end. For typed fields, the small model is cheaper and steadier. The VLM is the one that sees.

Each half keeps its own job. The VLM is the eyes. The extractor is the clerk. Neither one is asked to be the other.

What a VLM cannot do

Three limits are hard, not soft. Each one is a fact of the definition, not a gap that more compute will close.

First, a VLM produces text only. No motor command lives in its output space. You can ask a VLM to write the words "close gripper". Your code can parse that string and act on it. But then your code is the actor, the loop runs at your parsing speed, and the model never learns whether the gripper closed. A hobby project on r/robotics frames its prompt-controlled arm exactly this way. The VLM is prompted, and other software moves the arm.

Second, a VLM has no sense of the robot's body. It reads images and text. It does not read joint angles, gripper state, or force. A policy model eats that state every step. Ask a VLM whether the arm is touching the table and it will guess from pixels. That is a guess.

Third, a VLM is not built for control-loop rates. These models answer in a seconds-per-answer class. They are built and sized for language, and every design choice rewards longer reasoning over faster reflexes. The latency tables, the loop-rate math, and the hybrid pattern that pairs a slow reasoning tier with a fast action tier live on the VLM versus VLA guide. This page states only the definition fact: text output, at text speed, by construction.

Where the VLA line sits

The action class got its name in July 2023. The RT-2 paper wrote: "We refer to such category of models as vision-language-action models (VLA) and instantiate an example of such a model, which we call RT-2." (RT-2 paper, July 2023). Read the sentence in the original, because the naming is the boundary. The new word is the A. The A is the action output.

The build behind the name is direct. A VLA is a VLM backbone plus action training. OpenVLA states its own recipe plainly. It is "a 7B-parameter open-source VLA" that "builds on a Llama 2 language model combined with a visual encoder that fuses pretrained features from DINOv2 and SigLIP". It was trained on 970 thousand real robot demos. It beat the closed RT-2-X 55B by 16.5 absolute points across 29 tasks, with 7 times fewer parameters (OpenVLA, June 2024). Llama 2 plus a fused vision encoder is a VLM. Add the action head and the demos, and it is a VLA.

Does the backbone choice matter, or is any VLM good enough once the action head is trained? The RoboVLMs study built policies on eight VLM backbones and scored them on the CALVIN suite. The spread was large. Policies on Paligemma reached an average task length of 3.82. KosMos backbones reached 3.59. A Qwen backbone reached 0.30. The study's own hypothesis names the likely cause. The strong backbones had enough vision-language pretraining (RoboVLMs, 2024). The VLM is not a detour from robot action. It is the floor the action head stands on.

The full family tree from RT-2 to the current open models, tier by tier, lives on the VLA explainer. The tier-versus-tier decision, with rates and costs, lives on the versus page. This page owns the definition and the line.

How to read a robotics VLM model card

Robotics VLMs are bought, not built, in most stacks today. The card is where you check one before you pay. Five checks catch most of what matters.

Check the pointing and grounding rows. A card that lists Point-Bench, RefSpatial, or spatial suites is claiming coordinate-level skill. That is what picking and handoff steps eat. A card with only captioning and chat rows is a general model wearing a robot badge.

Check the embodied suites. ERQA, EmbSpatial, and VSI-Bench style rows say the model was scored on physical scenes, not just web images. The Molmo2-ER card reports scores across 13 embodied reasoning suites, best on 9 of them, at a 63.8 percent average. That is a 17 point gain over its non-ER base (Molmo2-ER model card, read 28 October 2026). That is what a backed embodied claim looks like.

Check multi-image and video support. Robots work from several cameras, from head and hand views, and from short clips. The Molmo2-ER card lists multi-image input, matching between body-mounted and external camera views, and video time reasoning among its trained skills (Molmo2-ER model card, read 28 October 2026). A card that takes one image per call will not carry a two-camera cell.

Check the window you will actually get. A card's native context window is the model's ceiling. A hosted row enforces its own served window, which can be far smaller. The Cosmos-Reason2-2B card describes a 2 billion parameter model for physical AI and robotics. The card frames it as a planning model that reasons what steps an embodied agent takes next. It lists long context support up to 256 thousand tokens and video input at a recommended 4 frames per second (Cosmos-Reason2-2B model card, read 28 October 2026). The hosted cosmos-reason2-2b row serves a 4096 token window. Both numbers are true. You need both.

Check the license before the score chase. Apache-2.0, as on the Molmo2-ER card, permits commercial use. The NVIDIA Open Model License, as on the Cosmos-Reason2 cards, also permits commercial use. Terms differ more than scores do in this lane. The GR00T license guide shows how the terms read, and the host survey shows which big hosts carry robot models at all.

The VLM-class rows on the public list today

Dated state, one basis per row: this platform's own model data, read 28 October 2026.

The two chat rows have been on the public list since 6 October 2026. Both run the embodied reasoning role described above. Molmo2-ER is the pointing and matching specialist, on a Qwen3-4B backbone with a SigLIP2 encoder. Cosmos-Reason2-2b is the small planning reasoner from the NVIDIA Cosmos line. The extraction row is the clerk from the two-stage pattern.

The action side of the family is a different set of rows, with its own pages. The MolmoAct2 line, which uses Molmo2-ER as its vision-language backbone per the card, is not on the public list here. Its guides cover the model and its checkpoint handling regardless of list state. The smolvla pricing page and the OpenVLA-OFT pricing page cover action rows that are on the list.

What the reasoning rows cost, from the same model data:

One relation is worth stating while the numbers sit together. The 2B reasoner's input rate is a tenth of the 4B specialist's, on the same basis. A stack that runs the small model as a fast scene describer pays the small share on that half of the bill. Exact budget math for the ER specialist, with cache behavior and worked calls, lives on the Molmo2-ER pricing guide. The full rate sheet for every row lives on the pricing page. An account on the console with a prepaid wallet covers all three rows on one bill.

Questions people ask

What does VLM stand for?

Vision language model. Some papers, including Flamingo's own title words, write visual language model. Same class, same contract: images and text in, text out (Flamingo, April 2022).

Is a chat model that accepts images a VLM?

Yes, by the definition above. The big multimodal chat models are VLMs. The robotics corner is the subset of the class tuned and scored for embodied work: pointing, spatial reasoning, matching between views, and video. The gap between that subset and the chat mainline is exactly what the card checks above measure.

Can a VLM control a robot directly?

No. Its output space is text, so no motor command can come out of it. A stack can parse its words into commands. But the parser and the safety layer around it are your software, and the model gets no feedback about whether the motion happened. Direct action output is the VLA class, named in 2023 (RT-2 paper, July 2023).

What is the difference between a VLM and a VLA in one line?

A VLM writes the next word. A VLA writes the next action. The VLA keeps the VLM's eyes and words and adds an action head trained on robot demos. That is the OpenVLA recipe in one sentence (OpenVLA, June 2024).

What does the ER in Molmo2-ER mean?

Embodied reasoning. The ER tag marks models tuned and scored on physical-scene suites rather than general chat rows. The embodied reasoning term guide traces the term's own history, including the Gemini Robotics pair.

Do I need a VLM if my policy already works?

If the cell never changes and the object set is fixed, possibly not. If anything varies, the VLM is usually the cheap flexibility you can buy. New objects arrive as words, not as new training runs. The two-tier pattern and its cost shape are worked through on the versus page.

Does a VLM help with documents and OCR too?

The class overlaps document work wherever reading is the job. The VLM OCR decision guide draws that line for text extraction. It is the same line as here: what the model reads, what it writes, and what it cannot touch.

Where to read next

The VLA explainer owns the action side of the family tree. The VLM versus VLA guide owns the tier decision, with rates and costs. The Molmo2-ER pricing guide works the specialist's budget math. The MolmoAct2 guide covers the action model built on the same backbone. The perception alternatives guide surveys the robot vision lane this class lives in.