Segmentation and depth for robots

A robot that reaches into the world needs two answers from its cameras. Which pixels belong to the thing it must handle. And how far away those pixels are. Segmentation answers the first question. Depth estimation answers the second. This guide is for the builder putting both into one robot. What each output feeds, which family and tier to pick, how the two combine into a grasp point, and how perception sits next to a policy. Two of the vision lane's rows, sam2.1-base and da3-small, serve on this platform today; the large tier does not. The model facts, the numbers, and the recipes are real today.

The lane's category page is the vision page. This guide is the how-to layer under it, and every technical claim below comes from the publishers' own repositories, papers, and documentation.

What a robot actually needs from vision

A person wants a photo labeled. A robot wants an action input. That difference decides which vision output is worth paying for.

A mask, the output of segmentation, is a per-pixel boundary. To a grasping arm it is the outline of the thing to pick. To a navigation stack it is the edge of the walkable floor. To a safety layer it is the shape of the person who just walked in. A detection box says roughly where the object is; a mask says exactly where it ends, and grasp quality follows from that edge.

A depth map is a per-pixel distance. A mobile robot reads the shape of the free space ahead. An arm reads how far the wrist travels before contact. A pick pipeline gets the third dimension that turns a flat outline into a placed object.

Two properties matter to a control loop. Latency has a budget: a pass answering in tens of milliseconds fits inside a control tick; one answering in seconds is a slow sensor. And behavior must be predictable. A model that finds the cup in 95 frames of 100 is a statistical component. Your code needs a default for the misses.

The four prompt types, and the one a robot can use

Segmentation models in this family are promptable. The vocabulary is four words wide, and it decides what client you build.

Points and boxes are annotation vocabulary: someone is looking and clicking. A perception pass runs many times a second with nobody watching. The prompt that matters needs no person: automatic masking, in the family since the first generation (SAM 2 repository).

Text prompts are the newcomer. The current generation takes a phrase like "delivery van" and masks every instance. Meta serves it with narrow rules: text only, one concrete object per phrase, several requests for several object types (Meta Model API docs). For a robot, that is "mask only the cups" without training a classifier.

The SAM family, mapped honestly

Meta's Segment Anything line went through five releases in under three years, and the licensing changed along the way.

The pick for per-frame robot work today is still SAM 2.1: the mature image-mode generation, Apache 2.0 checkpoints, four sizes with published numbers.

Speed is Meta's own benchmark, an A100 with torch 2.5.1 and CUDA 12.4 (SAM 2 repository); your hardware will differ. The ranking is the transferable fact. Smaller tiers run faster. Middle tiers sit near the top on accuracy. Base-plus at 64.1 FPS is the volume pick for a loop. Large is the accuracy pick for crowded frames and thin structures. This platform serves both tiers as rows: sam2.1-base for volume, serving today, and sam2.1-large for the hard passes, which is not on the list.

Two catches. First, licensing: SAM 3 ships under Meta's own SAM License, updated November 19 2025. Commercial use is royalty-free. Redistribution of the model or derivatives must stay under the same terms, license attached. The file also adds trade-control clauses an Apache license never raises (SAM 3 repository). Read it before shipping on those weights. Second, mode: the video predictor keeps memory between frames and its code path assumes frames loaded from disk (SAM 2 repository). Image mode is stateless, and a perception loop wants the stateless shape. State on the serving side is a queue you manage.

What SAM 3 added is real even if you stay on 2.1. Its data engine labeled over 4 million unique concepts. Its benchmark reaches 75 to 80 percent of human performance on 270,000 of them (SAM 3 repository). Research already extends the line to robots: GraspSAM couples the promptable mask with grasp prediction, taking points, boxes, and language prompts (GraspSAM paper, 2024).

The known failure classes of the 2.1 generation are community-documented: shot changes, crowded scenes after long occlusions, thin structures, and visually similar neighbors. The mitigation is always the same shape. Filter masks by area, require the same mask across a few frames before acting, and keep the large tier for the frames that matter.

Depth: relative or metric, and how to choose

One family matters on the depth side, and one decision trips people up.

The family is Depth Anything. The current generation, Depth Anything 3, comes from the ByteDance-led team. It predicts geometry from any number of views, with or without known poses. The paper reports it beating the prior state of the art, VGGT. The margin is 44.3 percent in camera pose accuracy and 25.1 percent in geometric accuracy. It also beats Depth Anything 2 on monocular depth (DA3 paper, 2025). One model gives a depth map per camera; several views of one scene return consistent depth plus the poses between them.

The decision is relative versus metric. Relative depth orders the scene, nearest to farthest: which object is closer, where the free space opens, the approach direction. It does not measure meters. Metric depth carries real-world scale. The main DA3 series returns relative depth, and the metric output is a separate fine-tune. The publisher's documentation gives the conversion: for the metric large model, meters equal the focal length in pixels times the network output divided by 300. The nested any-view models output meters directly, and both nested checkpoints carry the non-commercial license (DA3 repository). True distances from a relative model need scale recovered by hand: a known size or a baseline.

The license split is per tier. It catches commercial builders. From the repository's model table: small at 0.08B parameters and base at 0.12B are Apache 2.0. Large at 0.35B and giant at 1.15B are CC BY-NC 4.0, which bars commercial use. Separate large models fine-tuned for metric and monocular use are Apache 2.0 again (DA3 repository). The split is not new. The previous generation licensed only its small tier as Apache 2.0, with base, large, and giant non-commercial (DA2 repository), and cheap community endpoints have served those weights without saying so. This platform serves the small tier on its vision lane: da3-small, Apache 2.0, returning depth, confidence, and camera geometry.

A stereo camera computes metric depth from two lenses, out of the box. The cost is dedicated hardware, calibration work, and trouble with textureless surfaces. Monocular depth needs one ordinary camera you already own. Many robots run both.

The mounting reality breaks more deployments than model choice. Tape the cameras down. A camera bumped a few degrees puts every depth measurement in the wrong place until someone realigns it. Match your timestamps. A color frame and a depth value captured 100 milliseconds apart show different scenes when anything moves. Fix the exposure. Boring rules, and they are the difference between a demo and a robot that works on Monday.

The pairing: mask times depth equals a grasp point

Here is the arithmetic no ranking page performs, and the reason a robot wants both outputs in one stack.

One: segment the frame. The mask gives you the pixels of the target object. Two: run depth on the same frame. Each of those pixels now has a distance. Three: read the depth values at the mask's pixels. Their middle value is a working distance to the object. Four: lift the pixels into space. With the camera's focal length and optical center, each pixel plus its depth becomes a 3D point. The cluster is the object's position in the camera's frame. Five: plan the grasp against that cluster, and hand the result to your planner or policy.

MIT's robot manipulation course teaches exactly this shape for bin picking: run an instance segmentation over the bin, then apply grasping strategies to the segmented point cloud only, one object type at a time, with the clutter out of the way (MIT manipulation coursebook, chapter 9). That pipeline predates the current model generation by years. What changed is step one: the foundation families mask things they have never seen.

One distinction belongs here, because search results mix it up: segmenting the robot itself is a different job from segmenting the scene. An arm camera sees the arm's own gripper, articulated and moving, and general segmenters struggle with it. RobotSeg, a CVPR 2026 model built on SAM 2, exists for that; its authors list visual servoing, data augmentation, real-to-sim transfer, and safety monitoring (RobotSeg paper, 2025).

Four workflows that use both

Run the volume tier per frame and the accuracy tier where the mask decides the grasp. Depth carries navigation, masks label the obstacle, and the person mask plus depth is a geofence computed from pixels.

Perception and action in one stack

Where does this sit next to a vision language action model, which also takes camera frames? There are two ways to build the stack, and both are legitimate.

The end-to-end way: the action model does its own seeing. Frames, an instruction, and the arm state go in; an action chunk comes out. Models like pi05 and molmoact2-so100-101 work this way, and the VLA lane page describes the family. The perception is inside the model. You trade interpretability for simplicity.

The modular way: explicit perception in front, policy or planner behind. Masks and depth are data your code can inspect, log, and bound. Obstacle logic that must fire. Grasp selection you can audit. Scene parsing your planner reasons over. Many real stacks are hybrids. The action model handles the routine. The perception layer guards the edges and cleans the fine-tune data.

The loop mechanics are the same either way. The robot library ecosystem splits them cleanly. A policy server computes action chunks. A small client next to the arm drives the servos and cameras. The robot works through one chunk while the next computes. The knobs are chunk length and the ask-early threshold (LeRobot async inference docs). A perception pass slots into the same shape: a call your client makes on a schedule, budgeted by the frames you can skip between passes. The hosted action side has its contract on the action route reference, the SO-101 guide walks a full arm loop with one, and the class explainer covers the model class.

Serving the loop: your GPU or a per-image route

Everything above runs on your own hardware today. Both families ship open weights. The volume tiers are small. A single consumer GPU serves them. The small depth tier runs without a discrete GPU. The cost of that path is not the model. It is the machine you keep alive and the queue you babysit.

The other shape is a hosted perception route, metered per image, and this platform publishes one. You post image file parts and get masks or depth back, billed by image count. Three fields in: model name, one or more image parts, and a format switch between annotation data and rendered bytes. Whole-image automatic mode is the launch shape; point and box fields are refused with a named error rather than a guess, and the body cap is 25 MiB. The contract is on the run-vision reference, and the vision route page walks it end to end.

Said plainly: the three vision rows, two SAM 2.1 tiers and the Depth Anything 3 small tier, sit in the platform's model data, and none is on the public list. The models index is the live catalog; a listed row's per-image rate renders on the pricing page from the platform's own data.

Budget a loop before you build it, in images: a perception pass at 5 frames per second over an 8 hour shift is 144,000 images a day, and 28,800 at 1 frame per second. That is the number for any cost comparison, and the VLA local versus API break-even guide shows the method. A robot does not need to see at camera frame rate, only fast enough for the thing it is doing. Picking that rate is the first optimization to reach for.

Common questions

Are you focusing on visual scene segmentation or motion and skill segmentation? Scene segmentation, the pixel kind: which pixels belong to which object in one frame. Motion and skill segmentation cuts a demonstrated behavior into skills over time. A third neighbor, segmenting the robot itself, has its own model line.

What sensors or models, like YOLO, SAM, or depth cameras, should I use? One ordinary RGB camera covers both halves. A SAM 2.1 tier for masks. A Depth Anything 3 tier for depth. Code combines them. Add stereo hardware only for metric precision from the sensor. YOLO-class detectors give boxes, not boundaries; boxes are too coarse for grasp edges.

What are the best segmentation models? There is a match, not a best. Prompt-free per-frame work: the SAM 2.1 tiers, base-plus for volume and large for hard scenes. Text-driven masks: the SAM 3 generation, license read first. Depth: the Depth Anything 3 tiers, license checked per tier.

What is segmentation in AI? Assigning a label to every pixel, not one box around a region. The output is a mask per object: an exact boundary your code can act on.

Which segmentation model should a robot use? Start with SAM 2.1 base-plus in automatic mode: it masks everything at published speed, the weights are Apache 2.0, and your logic picks the object. Move up to large for crowded or thin frames, and to text prompts only when you want filtering by name.

How do robots use depth estimation? Navigation reads the free-space shape, arms read the approach distance, and pick pipelines combine depth with a mask to place the object in 3D. Relative depth covers ordering: stop, steer, approach direction. Metric depth is what measurement and mapping need.

What is the difference between segmentation and depth estimation? Which pixels belong to the object, versus how far away they are. A grasp wants both: boundary and distance.

Is Segment Anything free for commercial use? By generation. SAM 2 and SAM 2.1 are Apache 2.0, weights included. SAM 3 uses Meta's own license: commercial use is granted. Redistribution terms and trade-control clauses apply, and the checkpoints sit behind an access request. The Depth Anything 3 family splits by tier: small and base are Apache 2.0, the main-series large and giant are non-commercial.

The two outputs are cheap, small models with big effects. One finds the boundary, one the distance. The pair places an object in front of your gripper. Pick the tier for the job, keep the cameras taped down, plan for the frames the model misses, and budget the loop in images. An account on the console today covers every lane on the public list.

Last verified: 2026-10-05