Robot evaluation cost: the 1.93 per 500-episode report

A 500-episode LIBERO evaluation cost 1.93 in compute and 47.18 hours of wall time on one rented 24 GB card. This page shows every input: the rate basis, the episode clock, the restarts, and the report it bought, 482 successes out of 500. The money query has no other answer in search today. Every ranking page is a how-to, a protocol paper, or a raw card price. The measurement record is ours, dated 2026-10-10, and the arithmetic below re-runs with a pencil.

Amounts here are United States dollars, written as bare numerals. The tables are static. No script is needed to check them.

Last verified: 2026-10-26

The price in one line

cost of an evaluation report  =  W x R

W:  wall hours, set by episode count and the reset clock
R:  card-hour rate, set by where the hours come from

The measured run: 500 episodes across the four LIBERO evaluation suites. Wall: 47.18 hours. Rate basis: 0.041 per card-hour. Cost: 1.9344, written as 1.93 in the title. Divide the other way. The same run bought about 258 episodes per dollar. The table below is what the money bought.

Every ranking page for this query tells you how to run the benchmark. None of them tells you what it costs. This page is the price tag, down to the restart tax.

What the money buys

A report is a table of successes over counted episodes. This one, for a pi 0.5 class policy fine-tuned 15,000 steps on open LIBERO demonstrations. The platform's pi05 page covers that model class itself. The report:

Three properties of the report matter more than the number itself. The episodes are counted. The rate has a denominator. The suites are separate. The long-horizon suite cannot hide inside an average. And the run is one policy, one tune, one protocol. That is the shape the benchmark's own documentation asks for. It warns that scores move a few points across evaluation seeds.

The training that produced the policy cost 0.1497 on the same basis. That is 3.65 hours for 15,000 steps. Train plus report: 0.1497 + 1.9344 = 2.08 for the whole fine-tune-and-evaluate loop at the run's basis. The product-level math against what the market charges for that loop is its own page. It is the fine-tune and eval product math guide.

Before the long run, the record shows a 40-episode quick check across the four suites. It took 3.863 hours, about 0.16 at the run's basis. That is the shape of a sane budget. Spend four hours proving the pipeline, the checkpoint, and the cameras. Then spend two days on the real count. The most common way to waste a long evaluation is running the wrong checkpoint. A quick check catches it while it is still cheap.

The basis: one rented card, one clock

The run rented community-cloud cards of the RTX 3090 class, 24 GB, with 10 vCPU and 32 GB of RAM each. It pinned its library versions (LeRobot 0.6.1). Nothing about the rig is exotic. That is the point: the number transfers.

The rate basis deserves plain words. Its owner pays a below-list batch rate at that community cloud, 0.041 per card-hour. The cloud's own pricing page says volume, long-running, and committed work is priced below its published tiers. So the basis is a named program at a named cloud, not an invention. A member of the public renting the same card class today pays 0.090 per hour at the lowest public tier. The page re-prices the full wall at public rates in a later table. Nobody should mistake 1.93 for a price they can type into a checkout.

The arithmetic, in the open:

cost  =  W x R
        47.18 h x 0.041 per card-hour  =  1.9344

episodes per dollar  =  N / cost
                        500 / 1.9344  =  258.5

The quick check's arithmetic, same shape: 3.863 h x 0.041 = 0.1584.

The wall nobody budgets: resets, not rollouts

Here is the law this measurement pins. It is the one that breaks budgets. An episode has two parts. The rollout is where the policy acts. The reset is where the simulator rebuilds the scene for the next episode. The rollout is cheap. The reset is the evaluation.

The clock, measured: rollouts ran 23 to 43 seconds per episode on the 280-step suites. The long suite's 520-step episodes ran 80 to 83 seconds. Wall per episode, reset included: about 5.8 minutes. The rollout is under a quarter of the clock. The scene rebuild between episodes takes the rest.

This is not a quirk of one rig. LIBERO runs on CPU-side MuJoCo simulation. The official integration preserves what its documentation calls hard resets. It rebuilds the simulator model and the renderer after every episode. The docs offer soft resets to skip that rebuild. They warn in the same breath that soft resets are faster but not bit-identical. Results may differ. Anyone reproducing published numbers pays the rebuild. The benchmark repository fixes initial states for the same reason. Comparability has a cost, and the cost is the clock.

Now the contrast that explains every wrong budget you have seen. An October 2, 2026 vendor blog on parallel evaluation models 500 LIBERO episodes at 85.3 minutes and 1.42 GPU-hours, serial. Its own labels call the model illustrative, not a benchmark. The load-bearing assumption: 2 seconds per reset. The measured run paid about 5 minutes of rebuild per episode. That one input, 2 seconds assumed against 5 minutes real, is a factor of about 150. It is the entire difference between an 85-minute plan and a 47-hour run. If you budget an evaluation from someone's timing model, check the reset line first.

The second consequence is stranger: a bigger card does not fix it. The wall is CPU-bound in the reset, not GPU-bound in the policy. A faster card shortens the rollout slice, the smaller share. It leaves the rebuild untouched. The lever is on the CPU side. Pack more environments per machine, opt into soft resets when bit-identity does not matter, or shard episodes across more machines. Each of those changes the cost basis or the comparability of the numbers. The honest pages say so. The GPU-parallel simulators change the benchmark itself. Their assets and physics differ. Their numbers are not interchangeable with CPU LIBERO scores.

What the same run costs at rates you can rent today

Take the measured wall, 47.18 hours. Buy it at every public counter, all rates read on 26 October 2026:

Rates: Salad pricing, RunPod pods, and Hugging Face GPU pricing, each read on the date above. The spread is a factor of about 28 across one page, for the identical wall.

Read the vCPU column before the price column. The Hugging Face A10G small row carries 4 vCPUs and 15 GB of RAM with its 24 GB card. The measured run had 10 vCPUs and still spent most of its clock in CPU-side rebuilds. A cheap card behind a thin CPU allocation makes the reset wall worse. No card speed fixes that. When you rent for evaluation, buy the CPU alongside the GPU.

The tier choice is not only price. The run restarted twice across its window. One restart was deliberate: a planned stop to prove the resume path worked. The other was real: a node loss on the provider side. Both restarts rode the same design. The design ran 25-episode chunks, saved progress after each, and resumed on restart. Fourteen chunks finished on one machine life. Six rode through the two restarts. A cold re-run from scratch would have cost about 50 percent more wall. On an interruptible tier, that design is the difference between an inconvenience and a lost week. The pattern costs nothing to copy. Use seed-delimited chunks, saved progress, resume, and a run book that says which chunk is next.

Nobody prices an evaluation

The absence is the finding. It held when re-checked today. The query space around evaluation cost returns four kinds of pages. None of them puts a price on the answer.

How-to pages and official docs: the run commands, the suite arithmetic, the reset options. Protocol papers: LIBERO-PRO shows near-ceiling scores collapsing to 0.0 under perturbed settings. It argues the field's evaluation habits inflate results. A unified open harness reports a 47x wall speedup from sharding, 2,000 episodes in about 18 minutes, with no cost figures attached. And the productized end: a parallel-evaluation vendor, early access by waitlist, promising "a LIBERO sweep that takes 8 hours lands in about 30 minutes." Not one dollar figure appears on the page. A search for evaluation as a service returns whole-robot rentals and diagnostic shops. They want a call. The category exists in demand. In supply, the price column is empty.

That empty column is why this page exists. An evaluation report is a thing with a measurable cost of production. It took 47.18 measured hours and 1.93 at the owner's basis. The same hours run 4.25 to 47.18 at public counters. The protocol side is covered elsewhere on this site. That page covers which suites, which checkpoints, and how to read a leaderboard. This page owns the budget line.

Budget your own run

The recipe, all of it from measured or published numbers above.

1. Episodes decide hours. At the measured 5.66 minutes per episode, wall hours = episodes x 5.66 / 60. The published comparability protocol is 400 episodes (10 per task, four suites), about 38 hours. The measured 500-episode run was 47.18 hours.
2. Check what your episodes flag actually means. On the pinned library version (0.6.1) it counted per suite: a request for 500 delivered 2,000 episodes, about 8 days, caught at design time. Today's documentation describes the flag as per task. Print the episode plan before the long run, whatever version you run.
3. Buy the CPU, not just the card. The reset wall is CPU work. More vCPUs and more RAM per machine move the wall more than a faster GPU does.
4. Run the quick check first. Forty episodes, four hours or less, catches the wrong checkpoint and the wrong normalization while the mistake is still cheap.
5. Chunk and resume. Twenty-five episodes per chunk with saved progress turned two interruptions into non-events. Budget about 50 percent extra wall for cold re-runs if you skip this.
6. Price your precision. A statistics guide from NVIDIA works the math. A 90 percent rate over 70 rollouts carries a 95 percent interval of 80.5 to 95.9, a span of 15.4 points. Tightening that to plus or minus 2 points takes about 1,030 rollouts. At the measured per-episode clock, 1,030 episodes is 97 hours of wall, which is 8.75 at today's lowest public tier. Precision is a budget line, not a preference. Most published comparisons, the same guide notes, do not run enough rollouts to separate the policies they rank.

The three budget shapes side by side. Wall is episodes times the measured 5.66-minute clock, except the 500 row, which is the measured wall itself. Costs at three counters read today:

Read the table this way: the same claim, the same benchmark, the same clock. Where you buy the hours is an 11x spread. The episode count you choose adds another 2.6x on top. Nothing else in an evaluation budget moves money like those two decisions.

What each number is

Nothing on this page is a projection. The one number that is not ours, the vendor timing model, is labeled assumed by its own author. The page never blends it with the measured clock.

Frequently asked questions

What does a 500-episode LIBERO evaluation cost?

At the measurement's own basis, 1.9344 in compute: 47.18 hours of wall at 0.041 per card-hour. At public counters today, the same wall runs 4.25 at the lowest community tier. It runs up to 47.18 on a per-hour managed platform card. The episode count sets the hours. Where you buy the hours sets the price.

Why does my evaluation take days when a blog says 85 minutes?

The reset, not the rollout. The simulator rebuilds its model and renderer between episodes, measured at about 5.8 minutes of wall per episode all included. By contrast, the rollout itself is 23 to 83 seconds. Timing models that assume a 2-second reset underestimate the wall by a factor of about 150 on that line alone. The evaluation protocol guide covers the suite mechanics. This page covers the clock.

How many episodes do I actually need?

For comparability with published tables, the answer is 400. That is 10 per task across the four suites, averaged over three seeds. For statistics, more. A 90 percent rate over 70 rollouts spans 15 points of interval. Plus or minus 2 points needs about 1,030 rollouts, which is 97 measured hours. Choose by what you plan to claim. Report the count either way.

Will a bigger GPU make my evaluation cheaper?

Not much. The wall is CPU-bound in scene rebuilds. A card upgrade shortens the smaller share of the clock. The real levers: more vCPUs per machine, several environments in parallel, soft resets when bit-identical replay is not required, or sharding episodes across machines. GPU-parallel simulators are fastest of all. Their scores are not interchangeable with CPU LIBERO numbers.

What goes wrong in a long evaluation run?

Three things, all met in the measured run. The episode-count accounting trap. One flag meant per suite on the pinned version. It turned 500 episodes into 2,000, about 8 days. Restarts: one deliberate stop to prove the resume design, and one real node loss on the provider side, inside a single 40-hour window. The run survived on 25-episode chunks and saved progress. Without them, expect about a 50 percent wall penalty. And the wrong checkpoint, which scores a clean zero. It costs you two days if you skip the quick check.

Is a LIBERO report enough to buy on?

It is the comparability layer, not the whole truth. Two late-2025 studies stress the benchmark itself. They perturb objects, starts, words, and scenes, with near-ceiling scores collapsing in the worst case to zero. A clean 96.4 on the standard suites says the tune works on the memorized layouts. Before a purchase, spend a slice of the same budget on a small, counted evaluation that looks like your robot. That means your tables, your cameras, your parts. The cost arithmetic on this page prices that check in hours, not weeks.

Where does the 1.93 number come from?

From the arithmetic in the open: 47.18 measured wall hours times the run's 0.041 per card-hour basis. Every input is named, dated, and re-derivable with a pencil. The same wall is re-priced at seven public rates in the ladder table. The fine-tune side of the loop adds 0.1497 for training. Training plus report gives the 2.08 product figure, worked on the product math page.

The budget line for a robot program has many entries. Evaluation is the one nobody publishes. It no longer needs to be a guess. A counted report costs 47.18 measured hours and 1.93 at the owner's basis. The same hours are 4.25 to 47.18 at counters you can check today. For the serving side of the same program, the robot-hour cost derivation works the inference line. The card-rate lane guide covers renting by the month. Listed rates for every listed lane sit on the pricing page. Sign up, create a key, and the first thing worth measuring is your own loop.

Last verified: 2026-10-26