IronStratum

The robot moves wrong and nothing crashes: the norm stats error family

The robot moves wrong and nothing crashes. You fine-tuned a vision-language-action policy, or loaded a pretrained one. The arm jitters, freezes, drifts, or barely moves. No traceback appears. The training loss fell the whole time. In most cases the cause is the norm stats: the numbers that map your robot's units into the band the model thinks in. Missing, skipped, or mismatched, they produce quiet wrong motion instead of an error. A wrong map is still a valid map to the code. This page walks the failure family. You get the symptoms in builders' own words, and the reason no error fires. You get four silent causes with fixes and prevention, plus the other causes to check when the stats are fine. Keep an emergency stop within reach on every first hardware run.

One point needs to come first, because people search for an error string that does not exist. When norm stats are missing, skipped, or mismatched, there is no thrown error. The training runs. The checkpoint saves. The policy serves numbers. The arm is the only place the failure becomes visible. That is the trick of this family. The fix starts with data, not with logs.

The background you need is short. A policy never sees your joint angles. It sees values shifted and scaled into a small band, most often minus one to one. Norm stats are that shift and scale. One pair of numbers per joint, computed once from your dataset. The same numbers run backwards, to turn model output back into your units. The deep version of this background, with the formula modes and the per-family defaults table, lives in the norm stats concept guide. This page stays on the failure path.

The symptoms, in the words of the people who hit them

Every symptom below is a quote or a close paraphrase of a real report in a public tracker. None comes with an error message. Learn that pattern before you touch anything.

The jitter. A builder fine-tuned GR00T N1 on a SO-100 arm for a pick and place task. The report says the "robot exhibited jittering behavior without any meaningful task completion" (Isaac-GR00T issue 130). Other users in the same thread add their versions:

the gripper kept jittering around the object, without grasping

the best-case is the gripper keeps grabbing at air

the robotic arm exhibits persistent oscillations at its initial position

The last one matters extra. That user saw correct-looking joint command trends offline. The oscillation appeared only on the real arm.

The dead or tiny motion. A SmolVLA user tested a pretrained model first. Their words: "didn't work well, happy I didn't break the robot." Then they fine-tuned on 120 episodes. The model still failed, and more data changed nothing (LeRobot issue 2915).

The vanished success rate. A pi0.5 fine-tune on LIBERO came back with "the success rate is very low (1% on spatial task suite)." The same user noticed the computed stats differed from the checkpoint's (openpi issue 711).

The silent skip, measured. One report makes the invisible visible with code. A fake action went into a policy post-processor. The output equalled the input. That proves the un-scaling step did nothing. The issue title says it directly: "Normalization/unnormalization is silently skipped" (LeRobot issue 4415).

Read those four as a set. Jitter, dead motion, one percent success, and an output that equals its input. Four robots, three frameworks, one shape. The numbers moved, the code stayed quiet, and the arm told the truth.

Why there is no error

The silence is not an accident of some old version. It is structural. The code that applies the stats has a branch for "no stats found." That branch returns the data untouched. Here is the function in LeRobot today, from the main branch, read October 26, 2026 (normalize processor source):

norm_mode = self.norm_map.get(feature_type, NormalizationMode.IDENTITY)
if norm_mode == NormalizationMode.IDENTITY or key not in self._tensor_stats:
    return tensor

Two conditions send your data through unchanged. The mode says do nothing, or the stats key is absent. Neither raises. No exception exists on that path, so no log can print one.

The issue that documented the live failure of this path ships a four-line test. Load the published multi-dataset SmolVLA base. Build its post-processor. Push a fake action through it. Compare input to output:

fake_action = torch.tensor([0.5, 0.5, 0.5, 0.5, 0.5, 0.5])
result = postprocessor(fake_action)
print(torch.equal(fake_action, result))  # True

The healthy answer is False, because a real post-processor un-scales the values. The issue reports True. The values passed straight through. The root cause was a key mismatch. Checkpoints trained on several datasets keep their stats under dataset-prefixed keys, like so100.buffer.action. The processor looks up the plain key, action. The lookup misses, and the missing-key branch returns the tensor. A later report in the same issue adds that the state-side stats were dropped outright in that migration. A user confirmed the effect in one line: "I can confirm that currently the invocation of SmolVLA through LeRobot does not use normalization."

The deeper reason the silence survives review is math, not manners. These numbers apply a shift and a stretch. The wrong shift and stretch is still math that runs. The model predicts values in its band. Your code un-scales them with whatever numbers it holds. The arithmetic completes. The result is degrees or positions, on plausible scales, and the arm obeys them. Nothing in that pipeline can notice that two files disagreed. A wrong map does not break the computation. It breaks the robot.

The one error in this family that actually fires

For contrast, and to place your own case, one related failure is loud. Start a quantile-family model like pi0.5 on a dataset recorded before quantile stats existed. The first batch stops with this string, verbatim from the official pi0.5 docs:

ValueError: QUANTILES normalization mode requires q01 and q99 stats

That error is the lucky outcome. It stops the run before any bad checkpoint exists. Its two fixes are documented: recompute the stats, or switch that run to mean and std mode (pi05 docs). If you got that ValueError, you do not have the silent problem. The loud crash page covers it end to end. Everyone else on this page has the quiet version. The only honest signal is the arm.

Four silent ways the stats go wrong

The failures sort into four causes. Each one gets the same treatment: what happens, why, the fix, and the prevention.

Cause one: never computed, or borrowed from another robot

The clearest report is a question, asked by a user fine-tuning pi0.5 on DROID data (openpi issue 773):

when i fine-tune on droid dataset, and if i skip the compute norm stats step, it does not return any error

Their config shows why it ran. It pointed at the published DROID stats that ship with the pi05 DROID checkpoint. Training found numbers, so training proceeded. The user's own follow-up names the risk: borrowed stats "could be far away from my customized fine-tuning set."

The official answer exists, one file over. The openpi docs frame it as a choice (openpi norm stats docs). Reuse the pre-training stats when your robot and task match one of the published families. Compute fresh stats over your own data when they do not. When in doubt, "always try both" and keep what scores better. The same page is strict about one condition. Your robot and dataset must follow the action space definitions used in pre-training. Otherwise the borrowed map is wrong, no matter how close the tasks feel.

Cause: the stats describe another robot's distribution, and training never mentions it. Fix: compute stats for your own config before the next run, with the project's script. Or point your config at the published family that truly matches your body. Prevention: never start a run without knowing which stats it read. The project README puts the step before training and lists missing stats in its troubleshooting table (openpi README).

Cause two: skipped at load time

The second cause is crueler. The stats exist. The code simply does not apply them. The history explains how this became possible. Normalization used to live inside the model weights. A design change moved it out into separate processor files (backward compatibility page). After that, a checkpoint and its stats were two files that had to find each other at load time.

When they fail to find each other, you get the key-miss silence of the test above. The shape is older than that change, too. A 2025 report titled "Normalization mode sliently fall to no normalization" flagged the same silent fallback in an older code layout (LeRobot issue 821). That reporter's own case turned out to be two similarly named config settings. The silent fallback path they flagged was real. The surrounding surface still cracks in ways that touch normalization. One crack: a YAML override path that crashes on the normalization mapping field. Another: a migration script that writes an empty config file (issue 4840, issue 4649). The full census of tracker reports in this family, open and closed, is the stats missing with pretrained policies page.

Cause: the checkpoint and its stats stopped being read as a matched pair. Fix: run the four-line test against your own post-processor. Load the checkpoint and its processor files through one path, from one pinned version. Prevention: after any framework upgrade, re-run the pair test before you trust the arm again.

Cause three: from the wrong body, split, or dataset

The third cause is a match that should never have been made. The published stats cover named robot families: ALOHA-class dual arms, DROID Franka setups, UR5e, ARX, and more. Reusing them helps when your body matches. It hurts when it does not. Deployment has the same trap in a different shape. OpenVLA's deployment script loads a dataset_statistics.json that ships beside the fine-tuned weights. Every request carries an unnorm_key, naming the dataset whose stats should un-scale the output (deployment code). Ask for the wrong key, and a working model un-scales with the wrong map. The MolmoAct2 port has the same idea as a tag. The norm tag picks which embodiment's stats, prompt text, and horizon travel with the actions (MolmoAct2 docs).

The quietest version is the split. One check from that thread is worth quoting whole (openpi issue 711 comment):

print which norm-stats file the eval config actually loads, at startup. A stats file computed on a different split than the fine-tune fails quietly and only shows up as lower success

Cause: the stats that un-scale the output belong to a different body, data split, or dataset than yours. Fix: name the body explicitly, as a tag or key. Diff the checkpoint's stats against your dataset's. Prevention: one stats file per body and per split. The eval run must read the same file the training run wrote.

Cause four: describing the wrong representation

The fourth cause is the deepest, because the file is internally consistent and still wrong. An action vector can express the same motion in more than one way. Absolute joint targets are one way. Offsets from the current state are another. Real configs mix both. NVIDIA's example SO-100 config does exactly that (SO-100 config). The arm joints use relative offsets. The gripper uses absolute position. Both live in one action vector:

# single_arm: RELATIVE = delta from current state (better generalization)
ActionConfig(rep=ActionRepresentation.RELATIVE, ...),
# gripper: ABSOLUTE = target position (binary open/close works better absolute)
ActionConfig(rep=ActionRepresentation.ABSOLUTE, ...),

A stats file summarizes whichever numbers it was computed over. It writes down what the numbers were. It never writes down which way they were meant. Switch the action format and reuse the file, and both directions of the map go quietly wrong. Relative offsets scored through an absolute band land in a narrow sliver of the range. They can also clip at a rail, when the joint's absolute range sits far from zero. Absolute targets scored through a relative band saturate at the rails, because the relative band is several times too narrow. A clipped joint can even train beautifully. The model learns to output the rail. The loss for that joint falls toward zero. The joint never moves. For contrast, the openpi stats script applies the data transforms before it computes, so its numbers describe the action format the model trains on (compute script).

Cause: the file describes numbers your pipeline no longer produces. Fix: recompute after any action format change. Keep one dataset per format. Prevention: treat a format switch as a new dataset, with new stats, always.

When the stats are fine and the robot still moves wrong

Honest triage runs both ways. Not every wrong motion is stats. Three other causes produce symptoms that rhyme.

The uncommitted chunk. A SmolVLA thread pinpoints a hunting symptom with a precise cause (LeRobot issue 2915):

the symptom is the robot cannot commit to action, going back and forth or around the same position

That happens when the run executes only one step of each predicted action chunk. The advice there: commit a full chunk of actions per inference. If your arm hunts and overshoots, rather than jitters or stalls, check the chunk setting before the stats.

The camera that moved. A GR00T user had a simple task that worked. It "stopped working entirely when I shifted the camera only by 1cm." Their closing advice: "double check the camera did not move" (issue 130 comment). A view change is a data change. No stats file will absorb it.

The data itself. The official data guidance is numeric: record "at least 50 episodes, with 10 episodes per location. Keep the cameras fixed and maintain consistent grasping behavior" (data guide). Below that line, weak motion is the expected outcome, not a bug.

Two more candidates get one line each. Low-rank adapters: a GR00T maintainer notes plainly "I do observe lower performance with LoRa," and the one percent report above also ran LoRA. And plain forgetting: fine-tuning on new tasks can cost old ones. The research literature measures the effect and names action normalization as a first-class factor in whether adaptation works at all (continual VLA study, Foresight Action Normalization). In the continual-learning benchmark, each task is trained and tested against that task's own local action statistics, which is why the normalization choice rides along with every adaptation decision.

The fifteen-minute diagnosis

Run these in order. Most need no GPU. All of them read data you already have.

First, print the raw model output before your code un-scales it. Count where the values sit. A large share pinned at the exact band edges means clipping. That points at cause four. Values clustered tight around zero, or matching the input on the way out, point at cause two, the silent skip. The four-line test above is the template.

Second, print which stats file the run loaded, at startup, in both training and eval. This is the highest-value line on this page. It catches cause one and cause three in one stroke. It costs one print statement.

Third, open both stats files and compare the arrays side by side. The one beside your checkpoint, and the one computed from your dataset. Same keys. Same dimensions. Values of the same order. A body mismatch shows up here as numbers from a different joint range entirely.

Fourth, scan the spread of every dimension. Take q99 minus q01, or max minus min, per joint. A dimension whose spread rounds to zero never moved in the data. It will blow up everything scaled near it. The openpi troubleshooting table points at exactly this check for diverging training loss (openpi README).

Fifth, run a dataset doctor. The community tool lerobot-doctor checks LeRobot datasets for stats problems, zero-spread dimensions, and drift between the stats file and the data (lerobot-doctor).

Sixth, the sanity run. Train a small ACT policy on the same dataset. It is cheap and forgiving. The SmolVLA thread's own advice: check the dataset quality with ACT first (issue 2915). If ACT cannot learn the task either, the problem is the data. No stats fix will save the run.

Prevention: the pair law

Everything above collapses into one habit. A checkpoint and its stats are one object split across two files. Break the pair, in either direction, and you break the arm.

The practical rules follow from the four causes. Compute stats for your own config, or point at a published family that truly matches your body. Keep one dataset per action format. Recompute whenever the format changes, the robot changes, the calibration changes, or the dataset grows. Make the eval run read the same file the training run wrote. Print that file's path at startup, so the proof lands in your log. After any framework upgrade, run the pair test before the next hardware session. And on every first run of a new policy, go slow. Guard the workspace. Keep the emergency stop in reach. The failure mode of wrong stats is not an exception. It is motion.

What a served policy owes you

When the policy runs on your own machine, you own both halves of the pair. The checks on this page are the whole job. When a policy is served to you over a route, you can no longer read the host's disk. The pair law has to travel inside the response. The numbers that shaped the actions must come back stated with the actions, every time. Otherwise you are flying blind in exactly the way this page describes.

The concept guide carries the full version of that contract. It shows what the response's normalization block looks like and what stays yours to send correctly (norm stats concept guide). The platform's action route reference documents the response shape. Each robot model page on this site states its family's current status. For what any metered route costs, the answer lives on the pricing guide, never in prose. If you would rather fix this class of failure on hardware you own, the local versus hosted break-even guide works through that trade.

Common questions

Is there an error message for missing norm stats? No. That is the defining property of this failure family. The stats lookup has a branch for missing keys. That branch returns the data unchanged instead of raising. The only loud error in the neighborhood is the quantile ValueError. It means a different thing: your dataset predates quantile stats. It stops training before a bad checkpoint exists. That makes it the lucky outcome.

My robot barely moves, or the outputs sit near zero. Is it the stats? It is the most likely cause. Run the checks in order. Print what the model outputs before un-scaling. Print which stats file loaded. Compare the checkpoint's stats against the dataset's. Values passing through untouched, or a file borrowed from another robot, cover most reports like yours.

I skipped the stats computation and training ran fine. Was that wrong? It ran because your config found numbers somewhere. Usually a published set belonging to a robot family. That is correct when your robot and action space match that family. It is quietly wrong when they do not. The official guidance: try both, borrowed and fresh, and keep whichever evaluates better. Never leave the choice implicit.

How do I know which stats file my eval or server actually loaded? Print the resolved path at startup. This is the documented advice from a tester on the one percent success report. A stats file computed on a different split than the fine-tune fails quietly. It shows up only as lower success.

Do norm stats travel with the checkpoint? They should, and in the major stacks they do. As a stats file beside the weights, or as processor files the checkpoint expects to find. Treat the two files as one object. Copy them together. Version them together. After any migration or upgrade, prove the pair still finds itself with the four-line test.

After fine-tuning, the robot got worse, not better. Is that the stats? Possibly. Work the differential first. Check the chunk setting, the cameras, the episode count, and whether the run used adapters. Blame the stats last. If all four check out, recompute the stats and fine-tune again. The forgetting effect is real too. Fine-tuning can cost old skills even with perfect stats, and the research literature measures it.

The short version

Wrong motion with no error is the signature failure of norm stats that are missing, skipped, or mismatched. There is no thrown error, by structure. The code branch for a missing stats key returns your data untouched. Wrong numbers still run as valid math. The four causes are borrowed stats, load-time skips, wrong-body or wrong-split matches, and action format switches. The fix path is data-first. Print the raw output. Print the loaded file. Diff the pairs. Scan the spreads. Run the doctor. Sanity-check with a small ACT run. Prevention is one habit: the checkpoint and its stats are a pair. Guard the pair, and guard the workspace while you verify it. For the concept background, read the norm stats concept guide. To measure whether a fix worked, the evaluation guide is the method. Costs for any hosted route live on the pricing guide. You can sign up when you are ready to make a first call.

Last verified: 2026-10-26