How to use MolmoAct2: from artifact to SO-101

MolmoAct2 is an open vision language action model from the Allen Institute for AI, released in May 2026 under Apache 2.0. This tutorial walks the working path. It covers which checkpoint to download, what the honest numbers say, and how to record data and fine-tune on an SO-101. Then it covers what hardware each precision needs, and how the platform's published action route contract serves it with no GPU. Every number here is sourced and dated.

The short version is a loop. Pick the right artifact. Feed it two camera views and the arm's joint state. Keep the normalization statistics that ship with the checkpoint. Read action chunks out. On an SO-100 or SO-101 the model predicts chunks of 30 joint targets at 30 control ticks per second. If you own no GPU, the last section is for you.

What MolmoAct2 is, in one paragraph

Camera frames, a plain-language instruction, and the arm's joint state go in. An action chunk, a short run of joint targets, comes out. The backbone is Molmo2-ER, a vision language model trained for spatial reasoning. A flow-matching action expert is attached through per-layer key-value conditioning. Every expert block reads the keys and values of the matching backbone layer. The Allen Institute for AI released the model in May 2026. It shipped with the paper, the code repository, the datasets, and the weights. The license file in the repository is Apache 2.0, read directly, not assumed from a tag. The full fact record for the lane lives on the molmoact2-so100-101 model page. This guide stays on the working path.

The honest numbers before you spend a weekend

Two benchmark families float around this model. They tell very different stories, and most posts repeat the flattering one.

First, LIBERO, the standard simulation suite. From the paper (arXiv 2605.02881, May 2026, suite rows checked October 5, 2026):

The top three models sit within 0.3 points. A cluster that tight is a warning, not a ranking. It says the suite is close to solved for this model class. It tells you almost nothing about your table, your lighting, or your camera mounts.

The real-world tables spread out. The paper runs a real-world zero-shot test on the DROID setup, a Franka-class rig, out of distribution for these models. Five tasks, novel scenes and objects, randomized camera poses, 15 trials per task, scored as successful trajectories. The DROID fine-tunes scored:

On the SO-100 platform, five pick-and-place tasks ran with novel objects, randomized camera poses, 15 trials per task, and partial credit. MolmoAct2-SO100/101 averaged 56.7. A pi0 variant fine-tuned on the same data scored 45.3, and SmolVLA scored 2.3. That 56.7 is the realistic number to plan around on this arm class. It sits well clear of the smaller model and nowhere near the sim row.

Two limits come straight from the paper's limitations section. Chunks run open loop. The model commits to a full chunk before it sees what happened, so it cannot react inside one. And its reported 55.79 Hz control rate on one H100 with CUDA graphs is amortized chunk throughput. That is not closed-loop reactivity. For the benchmark landscape in depth, see the vla-evaluation-libero guide on this site. The VLA lane page lists the whole model family.

Which artifact to use

The release is a family, not one file. Download the wrong member and you spend an evening loading a foundation checkpoint. It was never meant to drive an arm alone. The repository's own words: these are "foundation checkpoints rather than one-size-fits-all deployment policies."

The SO100_101 fine-tune rests on a large open dataset: 38,059 episodes, 183.6 hours, 19.8M frames, from 1,222 public datasets by 377 contributors, filtered in four stages. Of those hours, 141.7 were recorded on SO-100 arms and 41.6 on SO-101 arms. Fine-tune from it rather than from the base when your arm is close. The repository recommends exactly that.

There are two loading paths. The allenai repositories load through transformers with trust_remote_code=True. The LeRobot-format checkpoint loads through LeRobot's own policy path. It ships the processor pipeline, including the SO-100/101 joint-frame correction described below. One boundary matters if you had your eye on the reasoning variant. The official LeRobot policy documentation says MolmoAct2-Think, which supports adaptive depth reasoning, "is not included in this LeRobot policy yet". Think loads through the allenai path only. One version note for that path: the processor imports the video utilities that transformers 4.52 introduced, so anything older fails at import.

To install LeRobot with the MolmoAct2 extras:

uv sync --locked --extra molmoact2

Each checkpoint download is about 22 GB. The SO100_101 repository measured 21.78 GB on October 5, 2026, and the LeRobot-format copy is about the same size, 21.77 GB measured the same day.

The SO-101 path: record, train, serve

This is the loop the announcement pages never spell out. The SO-101 with its brain in the cloud guide covers the arm side in depth. Here is the MolmoAct2-specific version.

The contract your data must meet

Three things travel with the checkpoint. Get them wrong and the arm misbehaves in ways that look like model failure.

Cameras. The SO100_101 checkpoint expects two views under the keys cam0 and cam1, primary and secondary. The fine-tune's model card says camera order does not need to be fixed for this checkpoint. The card's own sample episode shows two third-person views, top and side. Community builds report wrist views can make the model behave poorly. If that hits you, a documented workaround exists. Feed the external view twice. DROID uses the same key names, cam0 and cam1, as external and wrist. The bimanual checkpoint wants top, left, right, and order matters. If your dataset uses other names, LeRobot's rename map bridges them without touching the data or the policy.

Joints. The training data used an older joint calibration convention than LeRobot 0.5.0 and newer. The fix is a transform, joint_signs of [1, -1, 1, 1, 1, 1] and joint_offsets of [0, 90, 90, 0, 0, 0] degrees. It is already baked into the converted LeRobot checkpoint, so never re-derive it. Feed the raw joint state. Let the processor apply the transform. Without it, the arm may move in the wrong direction. The most reported failure is the arm slamming into the table at startup. The correction landed in LeRobot itself in June 2026, in a change titled "Enable MolmoAct2 rollout on SO-100/101 with calibration correction". Run a current version and this failure class disappears.

Norm stats. The per-embodiment normalization statistics ship inside the checkpoint as norm_stats.json. The SO-100/101 tag is so100_so101_molmoact2. The documentation is blunt about a tempting shortcut. Do not substitute approximate dataset metadata quantiles for the released statistics. The values are not the same, and the difference shows up as degraded actions. The vla-normalization-stats guide on this site explains what these files are and why they follow the checkpoint.

Record

Drive the leader arm, let the follower record. Name the cameras to match the checkpoint keys and the loop stays clean:

lerobot-record \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.cameras='{cam0: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, cam1: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30}}' \
  --teleop.type=so101_leader \
  --teleop.port=/dev/ttyACM1 \
  --dataset.repo_id=${HF_USER}/so101_pick_place \
  --dataset.single_task="pick up the red cube and place it in the bin" \
  --dataset.num_episodes=50

The official dataset advice: at least 50 episodes, 10 per location, cameras fixed, grasping style consistent. The same page gives a rule of thumb. You should be able to do the task yourself looking only at the camera images. MolmoAct2 was trained with image augmentation on, so keep it on when you train. The defaults are a 95 percent random crop, a 5 degree rotation, and color jitter.

Camera hardware has quiet failure modes on this arm, all documented in the community build the repository links. Lock each camera's white balance at startup. Drift between sessions can produce near zero actions that read as model failure. The checkpoint's sample episode shows two RealSense views, and a RealSense wants a USB 3 data cable. On a USB 2 cable it drops to 15 frames per second and warms up slowly. The official setup note for this arm family recommends the standard wrist configuration plus a third person camera, which is the shape that build runs.

Train

For fewer than about 200 episodes, the documentation recommends LoRA mode. It says LoRA "usually gives similar or even better performance than fft" on small datasets. LoRA trains small adapter matrices on the vision language side while the action expert trains fully. A single 24 GB card works at batch 1 with gradient checkpointing:

accelerate launch --num_processes=1 --mixed_precision=bf16 \
  -m lerobot.scripts.lerobot_train \
  --policy.type=molmoact2 \
  --policy.checkpoint_path=allenai/MolmoAct2-SO100_101 \
  --policy.train_mode_vlm=lora \
  --policy.action_mode=continuous \
  --policy.norm_tag=so100_so101_molmoact2 \
  --policy.dtype=bfloat16 \
  --policy.gradient_checkpointing=true \
  --policy.num_flow_timesteps=8 \
  --dataset.repo_id=${HF_USER}/so101_pick_place \
  --dataset.image_transforms.enable=true \
  --batch_size=1 \
  --steps=10000

For scale, the documentation's full LIBERO recipe is an eight H100 run at batch 32. The mixed-precision path keeps the action expert in fp32 and stores the large vision language matrices in bf16. That split is what makes the small-card path possible. The compile path follows the same split: it compiles only the action-expert blocks, because the vision language side did not pass parity checks compiled. If your dataset lacks quantile statistics, recompute them with the dataset script or train with mean/std normalization instead. Nobody has published solid single-card training timings for this model. Treat any blog's hour estimates as unverified until you run your own.

Three more train-side notes from the documentation and the open issue list. The adapter modes live behind one flag, which takes fft for full fine-tuning, freeze to lock the backbone, and lora for adapters. On the small dataset profile the documentation trains LoRA at batch 16 to 32 when memory allows. And a known issue fails training with a batch-shape error on datasets whose state vectors have length one. The pull request that fixes it is open in the library. If you hit a shape error on day one, check your state dimension before you blame the policy.

Serve on the arm

Zero-shot first, if your task is close to the training distribution:

lerobot-rollout \
  --policy.path=lerobot/MolmoAct2-SO100_101-LeRobot \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.cameras='{cam0: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, cam1: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30}}' \
  --task="pick up the red cube" \
  --duration=30

Then your own fine-tune, by pointing --policy.path at your checkpoint. The repository's safety words apply to every rollout. Validate outputs before deployment. Bound the action space by your controller limits. Use an emergency stop, and keep a person watching. Start with a per-tick motion cap and raise it only when the motion is boring.

Know what one call buys you. At the paper's SO-100/101 horizon, a chunk is 30 steps at 30 Hz, one second of motion. The horizon is fixed per embodiment in the post-training: 30 steps for this arm family and the bimanual YAM rigs, 15 for the DROID setup, 10 for LIBERO. Actions pad to a maximum horizon of 30 and a width of 32 dimensions. The arm plays the chunk back open loop, then your code re-queries with fresh state.

Two serve-side traps are quiet enough to cost an evening. If the feetech servo package is missing, inference silently falls back to a simulated arm. The arm never moves and no error prints. Check it before your first rollout. And expect the first call to be slow. Graph capture and warm-up dominated the first chunk in the community build, measured near 700 milliseconds on a laptop GPU before settling. The model card's own advice is to run warm-up calls before you measure anything. On RTX 50 series boards, install the CUDA 12.8 wheels the repository documents. Older pinned wheels fail with a no kernel image error.

What hardware you actually need

The memory story is a ladder, and each step is sourced. From the model card and the LeRobot v0.6.0 release post:

A 24 GB consumer card runs inference comfortably at bf16 and LoRA training at the edge. Full fine-tuning wants data-center memory. The card also notes bf16 "usually does not hurt performance much".

On speed, two anchors. The paper measures 23.02 Hz per action call on one H100 unoptimized, 55.79 Hz with CUDA graphs, at horizon 10. A community optimization project measured 611.3 ms for an unoptimized 30-action call on a Jetson AGX Thor. The project brought it to 113.4 ms with CUDA graphs and TensorRT. Treat it as one team's data point, not a promise. The first call after load is always slower while graphs are captured. Warm up before you measure.

This is not a CUDA-only model. The repository reports inference validated on Intel XPU with no code changes. In September 2026, AMD ran the full stack on a Ryzen AI Max+ 395 mini PC. The team drove one arm with it, and moved the same policy zero-shot to two other arms.

Serving it as an API

Maybe you do not want a GPU at all. The pattern already exists in public. The repository ships FastAPI servers with an /act endpoint per embodiment, running bf16 by default. The graph flag roughly doubles per-call speed for about 2 GB more memory. A community Docker space serves this exact checkpoint over the web on one 24 GB board. What does not exist yet, anywhere, is a metered API product shape for it.

This platform plans one. The request is a multipart form. It carries the model name, the instruction text, the robot state as a JSON array of numbers, and one or more camera images. The answer is an action chunk plus the normalization block the model applied, so your side can undo the transform. Calls are metered per served request, and the full contract is in the action route reference:

curl -sS $IRONSTRATUM_BASE_URL/v1/action \
  -H "Authorization: Bearer $KEY" \
  -F model=molmoact2-so100-101 \
  -F "instruction=pick up the red cube" \
  -F "state=[0.0, 90.0, 90.0, 0.0, 0.0, 0.5]" \
  -F image=@cam0.jpg \
  -F image=@cam1.jpg

The molmoact2-so100-101 lane is not on the platform's list. The models index is the live catalog, and rates render on the pricing page from the platform's model data. Sign up and create a key now.

Common questions

How to use MolmoAct2? Pick the artifact for your arm: SO100_101 for SO-100 and SO-101, the LeRobot-format copy if you train in LeRobot. Feed it two camera views, the instruction, and the joint state. The norm stats and the joint transform ship inside the checkpoint; keep them there. Read 30-step action chunks out. Go zero-shot for tasks close to the training set, and LoRA fine-tune for the rest.

Does MolmoAct2 work on SO-101? Yes. The SO100_101 fine-tune covers both arms. The calibration fix that made rollouts safe landed in LeRobot in June 2026. The converted LeRobot checkpoint ships with the fix applied. The community build the official repository links runs it zero-shot on an SO-101.

Can MolmoAct2 run on 24 GB? Inference yes: bf16 lands under 16 GB, about 12 GB in the LeRobot port. LoRA fine-tuning yes, at batch 1 with gradient checkpointing. fp32 inference needs about 24 to 26 GB, and full fine-tuning needs more than one consumer card.

Do I need to fine-tune it for my own robot? If your embodiment is close to SO-100/101, DROID, or bimanual YAM, try zero-shot first on simple in-distribution tasks. The repository warns that out-of-the-box use was verified on SO-100 and the Franka DROID setup only. A different arm means fine-tuning, and the base checkpoint exists for exactly that.

What cameras does it need? Two views for SO100_101, under the keys cam0 and cam1. The card's sample input carries two third-person views, top and side. Community builds report wrist views can make the model behave poorly, so if the arm misbehaves, feed the external view twice.

What is the norm tag for SO-100 and SO-101? so100_so101_molmoact2. It selects the normalization statistics, prompt metadata, image-key order, and action horizon from the checkpoint's norm_stats.json.

Is MolmoAct2-Think available in LeRobot? Not yet. The official policy documentation states Think "is not included in this LeRobot policy yet". Think loads through the allenai custom code path, and it is a foundation checkpoint for reasoning research, not an arm driver.

What license is MolmoAct2 under? Apache 2.0, per the LICENSE file in the allenai repository, intended for research and educational use under Ai2's responsible use guidelines. The file is what governs your copy, which is exactly why this site tells people to read license files, not tags. The GR00T license guide shows what happens when a README and a license file disagree.

Start with the artifact table, not the benchmark table. The artifact decides your weekend. Keep the norm stats with the checkpoint. Keep a hand on the emergency stop. And if you would rather skip the GPU, watch the pricing page for the action route's rates.

Last verified: 2026-10-05