VLA evaluation: LIBERO and the benchmark zoo, honestly

Vision language action models get judged on LIBERO. This guide explains what the benchmark is, suite by suite, with the task counts that trip people up. It shows the two open paths for running it. It lists the published scores for the models this platform writes up on its action route, each with source and date. Two runs of the same model are never merged here. It also covers what LIBERO cannot tell a buyer, the robustness suites, and how to read a leaderboard without getting fooled. The robot lanes are not on the public list. The benchmark, the numbers, and the run commands are real today.

The field agrees on the problem, in print. The PhAIL paper (May 2026) opens by naming the habit: binary success at a fixed timeout, 25 rollouts or fewer per condition, almost never with intervals. A single success number is noisier than anyone treats it. The largest public leaderboard today carries a warning label in its own header. This guide collects what those sources say, adds the run commands, and keeps every number named and dated.

What LIBERO is, and the suite arithmetic that trips people

LIBERO is a simulation benchmark for robot manipulation. It was built to study lifelong learning: how well a robot policy carries what it learned from one task to the next. The paper's own framing splits robot knowledge into declarative and procedural, and asks how much of each a policy keeps as tasks change. The paper came out in June 2023 (arXiv 2306.03310, NeurIPS 2023, Datasets and Benchmarks track). The tasks run in MuJoCo, on a simulated Franka-class arm in tabletop kitchen scenes. It needs Linux.

The arithmetic first, because pages get it wrong. The official repository groups the benchmark into 130 tasks across four suites:

Ten plus ten plus ten plus one hundred is one hundred thirty. The fourth suite confuses people. LIBERO-100 holds 100 tasks with mixed demands. The repository splits it for you. libero_90 holds ninety short tasks for pretraining. libero_10 holds ten long-horizon tasks, held back to test what the pretraining bought. That is why the LeRobot docs list five suite names while the repository names four. Both are right; they count at different layers.

One more distinction does real work. The published score tables cover four suites: spatial, object, goal, and the long-horizon ten. Those are the evaluation set; the ninety-task suite is training material, and the published pi 0.5 runs do not report it. A page saying a 97 on LIBERO means those four suites, forty tasks, and nothing else.

The data behind it is public. The LeRobot copy of the demos is 1.9 GB as video: 1,693 episodes, 273,465 frames over 40 tasks, two camera views at 256 by 256 pixels. A second copy stores raw images and weighs 69.9 GB. The original set was published by Physical Intelligence. All three live on the Hub. The paper's own appendix counts the demonstration pool behind each controlled suite at 500 recordings over its ten tasks, which is the 50-per-task rule arriving from a second direction.

How to run it: two working paths

Path one is LeRobot, and it is the shorter one. Both paths assume a Linux machine with a GPU, and the rendering backend is the one environment variable people forget. Install LeRobot with the LIBERO extra, set the MuJoCo rendering backend, and call the evaluation command:

pip install -e ".[libero]"
export MUJOCO_GL=egl

lerobot-eval \
  --policy.path=lerobot/pi05_libero_finetuned_v044 \
  --env.type=libero \
  --env.task=libero_spatial,libero_object,libero_goal,libero_10 \
  --eval.batch_size=1 \
  --eval.n_episodes=10 \
  --policy.n_action_steps=10 \
  --env.max_parallel_tasks=1

The protocol behind the published tables is short. Ten episodes per task across the four evaluation suites. 400 episodes in total, averaged over three seeds, run with the seed, the initial states, and a single batch held fixed. The action-step count in the command matches the original publisher implementation, per the docs. Pin the dataset revision when you report. The Hub re-uploads data. Scores are only comparable against the same revision. Use hard resets when you reproduce; hard is the default and soft is an opt-in flag. Soft resets are faster but not bit-identical. And check the control mode. The environment defaults to relative actions. An absolute-action checkpoint will move the arm wrongly instead of erroring out.

Path two is the publisher's own, in the openpi repository. It runs under Docker Compose, the policy in one container and the simulator as a client. The two sides meet in environment variables: SERVER_ARGS on the policy side picks the checkpoint, CLIENT_ARGS on the simulator side picks the suite, and the libero_10 flag selects the long-horizon suite by name. The reference numbers below came out of this path.

Three traps fill the issue trackers, and all three look like model failure:

1. The wrong checkpoint. The LeRobot port of the pi 0.5 LIBERO tune ships a base conversion and a scored fine-tune. Evaluate the base and you will read zero. An open tracker thread shows exactly that table (issue 2533): zero across suites for the base conversion, and 98, 100, 98, 96 for the scored fine-tune, the one past 25,000 downloads. The base is a starting point for your own training. The pi 0.5 LIBERO page walks the full checkpoint menu.
2. The wrong statistics. These models normalize inputs and outputs with numbers that travel with the checkpoint. Wrong numbers produce wrong motion, not error messages. The normalization stats guide and the embodiment stats notes cover that failure class.
3. The wrong version. Benchmark results move with library versions. The MolmoAct2 team pins a whole inference branch so their LIBERO numbers reproduce exactly; the MolmoAct2 guide shows the pattern. Pin whatever you run, and write it down.

The number zoo, honestly

Here is the zoo, one table, every row named and dated. Two rules govern it. Never merge two runs of the same model. They are different implementations or training runs, not one truth. And never average across suites in your head. The long-horizon ten is the hard one. A single average hides it. A third rule rides the source column: a row that cannot name its source and its date does not board the table.

Read the first two rows together, the honest-numbers lesson in miniature. One model, one tune, two implementations. The port team trained 6,000 extra steps to adapt the weights to their pipeline, in bfloat16 at batch 256 on eight H100s, drawing from the HuggingFaceVLA copy of the demos. They gave up 1.8 points on spatial, gained 3.6 on the long suite, and landed at a different average their own docs call consistent with the original. Both numbers are true. Neither is the number.

Then read the middle for the saturation problem. The top rows sit within fractions of a point: 97.2, 97.0, 96.9 in one comparison table from the MolmoAct2 paper. Near the ceiling, a number tells you the tune works and little else. Differences inside that band are mostly protocol noise, which is why the docs ask for seeds and revision pinning. The OpenVLA rows carry the other lesson, a column jump: the fine-tuned base at 76.5 against the tuned recipe at 97.1, the OFT paper's own headline result. And the paper's fuller table runs deeper than the rows above: TraceVLA at 74.8, SpatialVLA at 78.1, CoT-VLA at 83.9, ThinkAct at 84.4, MolmoAct-7B-D at 86.6, NORA-1.5 at 94.5. Every one is an average from the same dated table.

The X-VLA row carries only an average, because that is what its source publishes per suite, and the number is the checkpoint's own claim. The SmolVLA row carries its paper's Table 2 cells. The pattern is the useful part: the long-horizon suite is where the small model gives ground, 71 against the 90s elsewhere. X-VLA is 0.9 billion parameters, pretrained on 290K episodes across seven platforms before its LIBERO tune. SmolVLA is 0.45 billion. Neither keeps pace with the 3 to 7 billion class. Both cost far less to run. The GR00T row rides the paper's table even though its weights ship under a research-only license; benchmark runs are research, so the number publishes and the robot does not sell. The license guide on this site covers that split. The VLA lane page collects the family with sizes and license positions.

One absence, stated plainly: this platform has no measured numbers of its own. Nothing serves under these names yet, so there is nothing to measure. Our rows appear only after they are measured and held, on a later dated pass.

What LIBERO cannot tell you

Three limits matter, and each has a dated source.

It is simulation. The tasks run in MuJoCo kitchens on a simulated Franka-class arm. A high score says the model learned the task family, the words, and the motion. It says nothing about your table, lighting, camera mounts, or gripper backlash. The gap between a sim score and a real robot is the most expensive thing a first-time buyer learns.

The standard suites can be memorized. Two teams stress-tested LIBERO itself and found the same thing. LIBERO-PRO (October 2025, revised May 2026) perturbs objects, start states, instructions, and environments. Models above 90 in the standard setting collapsed to 0.0 under perturbation, which the authors attribute to memorized layouts and action sequences. LIBERO-plus (October 2025, revised December 2025) perturbs seven dimensions across roughly 10,000 task variants: object layout, camera viewpoints, robot start poses, reworded instructions, lighting, background textures, and sensor noise. Success falls from about 95 to below 30 under modest shifts. Models largely ignore wording changes. One install note: the LeRobot port replaces vanilla LIBERO, so you cannot keep both at once.

Small rollout counts hide wide error bars. A success rate is a share of trials. Few trials carry wide bounds. NVIDIA's evaluation blog works the math with Clopper-Pearson. A 90 percent rate over 70 rollouts carries a 95 percent interval of 80.5 to 95.9, a span of 15.4 points. Plus or minus 2 points takes about 1,030 rollouts. The blog's own verdict on the field: most published benchmarks do not run a sufficient number of rollouts. The PhAIL paper (May 2026) audits the habit: binary success at a fixed timeout, 25 rollouts or fewer, almost always without intervals. The published protocol of 10 episodes per task exists for comparability, not tight statistics. Know which one you are reading.

A community data point closes it. An October 2026 report in the publisher's tracker ran two pi 0.5 fine-tunes with the same training loss and the same open-loop action error. The closed-loop scores were 0 of 150 versus 60 of 150. Open-loop metrics do not predict closed-loop success. Only rollouts do.

So the buyer's rule is simple. Treat a near-97 as a smoke test that the model class and the tune work. Treat the differences between 96.9 and 97.2 as noise. The signal that predicts your robot is an evaluation closer to your robot.

The benchmark zoo beyond LIBERO

LIBERO is one cage. The largest public harness covers fourteen simulation benchmarks and six model servers, with real-robot transfer named as planned work. The AI2 evaluation harness is open source under Apache 2.0, first released March 2026. It runs benchmarks under Docker with the model decoupled, talking over a WebSocket and msgpack protocol, which is what lets the model and the benchmark live in separate containers. It reports a 47-fold wall-clock speedup from sharding: 2,000 LIBERO episodes in about 18 minutes (arXiv 2603.13966). What the main cages measure:

Pick by your claim. Generalization: LIBERO and its stress ports. Kitchen chores: RoboCasa. A product: the real rig, where the honest numbers hurt. One dating note on that table: the RoboCasa project's own front page now headlines RoboCasa365, a 365-task 2026 expansion, so the dated row describes the 2024 release, not the project's current headline count. Date your rows, or they drift.

How to read a leaderboard with suspicion

The AI2 VLA leaderboard is the largest unified comparison. Read on October 5, 2026, it carried 3,160 models across 15 benchmarks and 3,576 results from 2,386 reviewed papers, with monthly updates and a footer dating its last build to August 10, 2026. The harness README's own note from May 2026 counted 2,456 models over 18 benchmarks from 2,087 papers. Two official surfaces of one project, three months apart, disagree in every cell. That is normal for an aggregator. Read the page's own counters and its date, never the README's. Read its own header before its table, too. The leaderboard is beta. The page is largely maintained by AI, and errors are possible. Evaluation protocols are not fully standardized across benchmarks. That is the honest warning label of an aggregator. Each row is a per-paper number folded into one table; the LIBERO column is literally titled Avg (w/o 90), an average over suites with the pretraining suite dropped, which is the double-averaging this guide just told you not to do.

The PhAIL leaderboard is the opposite shape: one task family, one real rig, few models, deep protocol. Its author built it after failing to find honest numbers on how well these models work on commercial tasks, and describes the protocol as blind. Read live on October 5, 2026, its single-arm pick-and-place table on a DROID setup showed human teleoperation at 99.0 plus or minus 0.4 at 10.4 seconds per item, GR00T N1.6 at 46.1 plus or minus 3.9, open pi 0.5 at 44.4 plus or minus 4.3, both leaders at 15.7 seconds per item, ACT at 42.5 plus or minus 5.6, SmolVLA at 16.3 plus or minus 4.5. Its metric is plain. Success at 30 seconds is the share of items placed within 30 seconds of the previous one. The spread is half the 95 percent range. Tied ranks mean chance can explain the gap. The two robot leaders on the live page carry the same tied rank, which is that rule doing its job. PhAIL's two published reads, the live table and the paper's human-relative throughput, are never mixed here. The paper read puts public VLAs on a Franka FR3 and scores them by human-reference throughput; those numbers live on the model pages of the robots that ran them.

Six questions to ask any row, on any leaderboard:

1. Which suite? A "LIBERO" number without a suite name is four numbers blurred into one.
2. Which checkpoint? Base conversions, fine-tunes, and paper-specific variants of the same model score differently.
3. How many episodes and seeds? No denominator, no result.
4. Which metric, and whose definition? Success at task end, success within a time budget, and throughput are different units.
5. What date, and what versions? Suites and datasets get revised.
6. Is the row one run or an aggregate of papers? Aggregates inherit every protocol at once.

What the platform's action route is, and what you can do today

This platform publishes a metered action route contract for these models. You send camera frames, an instruction, and the arm state as numbers; you get an action chunk back plus the normalization block the model applied, so your driver can undo the scaling. The field-by-field contract is in the run-action reference and the action route notes. The lane is not on the public list.

A hosted policy route does not host the benchmark. LIBERO needs Linux and a MuJoCo rendering backend, so the simulator stays on your machine. What a served policy changes is the GPU story. Your eval loop sends observations and receives chunks. The memory-class card moves off your desk. The SO-101 cloud brain guide walks that split, and the break-even guide works the cost.

Three moves: read the VLA lane page for the family, the pricing guide for the listed rates, and sign up so a key is ready.

Common questions

Is 95 percent on LIBERO actually good? It is a real measurement of a narrow thing: those forty simulated tasks, under that protocol, on that data revision. Two robustness studies dated late 2025 show scores that high can reflect memorized layouts. Treat it as a smoke test, not a forecast.

How many rollouts does a VLA evaluation need? For comparability, ten episodes per task over the four evaluation suites, 400 total, three seeds. For statistics, more is better: the 95 percent interval of a 90 percent rate over 70 rollouts is 80.5 to 95.9, and plus or minus 2 points needs about 1,030 trials. The comparability protocol and the statistics protocol are different tools for different questions. Report your count either way.

Is LIBERO-10 the same as LIBERO-Long? Yes. LIBERO-Long names the ten long-horizon tasks and libero_10 is the command-line value. Both sit inside LIBERO-100 alongside libero_90, the ninety short tasks used for pretraining.

What is LIBERO-Spatial? One of the three controlled suites: ten tasks where objects and goal stay fixed and placements change, isolating reasoning about where things sit. Its siblings isolate object identity and goal following.

How big is the LIBERO dataset? The LeRobot copy is 1.9 GB as video: 1,693 episodes, 273,465 frames, 40 tasks, two cameras at 256 by 256. An image-stored copy is 69.9 GB. Each controlled-suite task ships 50 demonstrations.

What is the best VLA model? No honest flat answer. On LIBERO the top published runs cluster near 97, too close to separate. On real hardware the one live commercial-task leaderboard reads 46 percent for the best open model, 99 for a trained person. The right answer is the model whose evidence matches your robot, budget, and latency. The VLA lane page carries the family; the class explainer covers the class.

Should I evaluate in simulation or on real hardware? Both, for what each can say. Simulation gives cheap, repeatable comparisons; real hardware gives the number that predicts deployment. Tune and compare in sim, then run a small, counted, real-robot evaluation before any purchase.

What is a VLA in machine learning? A vision language action model: a robot policy that reads camera images and a task sentence, senses its joint state, and outputs motion commands. The class explainer covers it end to end.

The short version

LIBERO is 130 simulated tasks: three controlled suites of ten, and LIBERO-100, splitting into ninety for pretraining and ten long-horizon for testing. Published scores cover the four evaluation suites. Run it through LeRobot's eval command or the publisher's Docker path. Ten episodes per task, three seeds, revision pinned, right checkpoint. Read the zoo with every number named and dated. Never merge two runs. Never blur the suites. Treat near-97 as a smoke test; ask leaderboards the six questions. Count your rollouts before you trust a gap: seventy trials at 90 percent still spans 15 points. When a hosted policy route fits your plan better than a local GPU, the contract is written. Sign up and keep going.

Last verified: 2026-10-05