Image segmentation and depth estimation APIs: the robot vision lane

A robot that reaches into the world needs two answers from every camera frame. Where does this object end, pixel by pixel. And how far away is it. Image segmentation answers the first question. Depth estimation answers the second. The market sells them separately: model hosts with per-request pricing on one side, prompt tricks on general chat models on the other, and nothing at all on the depth side. This page is the map for a builder who wants both halves behind one route. It covers what the SAM family has become, why depth estimation as a service barely exists, and how this platform's vision lane is put together.

The honest state comes first. The lane's three model rows, two segmentation tiers from Meta's SAM 2.1 family and one depth tier from the Depth Anything 3 family, sit in the platform's model data. Two of the three, sam2.1-base and da3-small, are on the public list and serve, confirmed by the October 6, 2026 requests; the member panel below carries those two for that reason. The large tier is not on the list. The page stands complete on its own either way. The lane definition, the family maps, and the route contract below do not depend on any row being live, and the models index is the live catalog.

Two neighbors crowd the search results, and neither is this product. A stereo camera SDK computes depth from two lenses on dedicated hardware. A headset depth API runs inside a virtual-reality operating system. This lane is neither. It takes ordinary images, one or many, and returns masks or depth geometry as data your program can act on.

Last verified: 2026-10-05

What a vision API does for a robot

Three output types cover machine vision, and mixing them up costs money.

A mask is a per-pixel boundary. It can be a binary cutout of one object, or a set of cutouts with a label and a confidence score each. Masks drive grasping, defect checks, background removal, and any job where the exact edge matters. The prompt decides what gets masked: the strong segmentation families take text like "yellow school bus", a clicked point, a drawn box, or a reference mask.

A depth map is a per-pixel distance. It can be relative, which orders the scene from nearest to farthest and is enough for obstacle logic, or metric, which carries real-world scale for measurement and mapping. Depth drives navigation, arm approach, 3D reconstruction, and any job where the shape of the free space matters.

A caption is text about the scene. General multimodal chat models produce captions, labels, and bounding boxes. They do not produce clean geometry by default. Asking a chat model for segmentation works, after a fashion. You prompt carefully, request JSON, and parse. Google's own recipe for it tells developers to stay close to one recommended prompt. It asks for bounding boxes and masks in fixed JSON keys. And images sent as input bill at token prices. That is a workable hack for occasional queries. It is a poor foundation for a perception loop that runs on every frame a robot sees.

The dedicated-route alternative fixes the pricing unit and the contract. One image in, one structured answer out, one metered unit. That is the shape this lane follows.

A word map helps, because the search results mix four product names that sound alike. An image processing API edits pixels: resize, crop, convert, compress. A computer vision API is the big-cloud label for their bundled vision services, and those pages own that phrase. A machine vision API usually means factory inspection software tied to industrial cameras. A vision API for robots, the subject of this page, returns perception data: masks and depth that a control loop can act on. If you came here searching for free image processing, the honest answer is that classic resizing and format conversion do not need a model at all, and this lane is not that product.

Image segmentation today: the honest family map

Meta's Segment Anything line went through five releases in under three years, and the licensing changed along the way. The map, from the official sources:

The license row is where buyers get caught, so it deserves plain words. SAM 2.1 is commercial-clean. Meta's repository states that the model checkpoints, demo code, and training code are all Apache 2.0. SAM 3 is not Apache. It ships under Meta's own SAM License. That license grants a royalty-free commercial license and allows derivative works. But any redistribution, including your derivatives, must happen under those same terms, with the license attached. It also carries trade-control clauses an Apache license never raises. If you build a product on SAM 3 weights, read that file, not a summary of it. The SAM 3 checkpoints also sit behind an access request, where SAM 2.1 downloads open.

What SAM 3 added is real. Its paper came out in November 2025. It describes prompting with concept phrases like "yellow school bus", or with example images. A data engine labeled four million unique concepts. A detector and a video tracker share one backbone. Meta's Model API serves SAM 3.1 today. One integration covers images and video, and in video it follows each match across frames. Its prompt rules are narrow in a way that matters to builders: text prompts only, one concrete object per phrase, several requests for several object types.

Around Meta sits a fragmented market. One host sells SAM 3 per request with four prompt types and a knob that caps how many masks come back. Community hosts sell the SAM 2 large variant, image-only, with no tier choice. One gap stands out from the published tier table: the SAM 2.1 family has four size tiers, tiny, small, base-plus, and large, and almost every hosted lane serves only the large tier. The volume tier, base-plus, 80.8 million parameters at 64.1 frames per second in Meta's A100 benchmark, has no host. That tier is where per-frame robot work wants to live. This platform's segmentation rows are built around exactly that gap, with both tiers so the choice is real.

One more shape rounds out the map. Google's approach is published on its developers blog. It treats segmentation as a conversation with a general model. Ask for "the people who are not sitting" or "the homes with weather damage". The answer comes back as masks in JSON. It is the most flexible prompting story on the market. The cost is the contract. Output rides on prompt discipline, and images bill as tokens.

Depth estimation today: the empty lane

Ask a search engine for a depth estimation API and you get hardware. The top results for the query are a stereo camera SDK, a virtual-reality depth API, a GitHub repo, an explainer, and a 2022 blog post about portrait photos. Not one commercial service selling monocular depth as an API appears on the first page of Google results for the query. The demand exists. People who want Depth Anything served to them literally autocomplete the phrase "no api found" after the model name. Builders ask on forums whether anyone serves these models at all, because their machines cannot run them.

The technology is not the blocker. It is mature. Depth Anything 3, released by the ByteDance-led team in 2025, predicts geometry from any number of images, with or without known camera poses. It uses one plain transformer backbone and a single depth-ray target. The paper's claim: detail and generalization on par with Depth Anything 2 for monocular depth, plus camera pose estimation as a second output. A streaming variant released in November 2025 handles long video on under 12 GB of GPU memory. A fourth generation surfaced as research in August 2026, aimed at 4D scene reconstruction, and is not a product yet.

The family's licensing splits by tier, which matters to anyone commercial. On Hugging Face, the small tier card is Apache 2.0, as is the base tier. The large and giant tiers are non-commercial. A refreshed large tier returned to Apache. So the question "is Depth Anything free for commercial use" has a per-tier answer, and the cheap community lanes that serve the older generation run weights you cannot legally ship in a product.

That is the whole picture on the depth side: strong open models, per-tier license care, no commercial contract page anywhere on the head query, and autocomplete evidence of people looking for one. This platform's depth row exists because that hole is real.

One route for both: the perception contract

The vision lane puts both halves behind one route. No code on this page; the contract lives in the API reference and on the vision route page.

The call is a multipart form with three moving parts. A model field names the row. One or more image file parts carry the payload, a single image or a batch, one kind of input. A response format field picks the answer shape: JSON, the default, or binary. A JSON answer carries the model's annotations and a usage block with the billed image count and the cost of the call. The wallet math is in the response, not on a separate page. A binary answer returns the rendered bytes, a mask image for instance, under the model's own content type, with billing on the input side.

Two contract facts deserve plain statements. First, the day-one prompt shape is the whole image: the route does not accept point or box prompt fields yet, and sending one returns a named unknown-parameter error rather than a guess. Automatic, full-image masking is the launch behavior. Second, metering is per image, taken from the count of images in the request, and the family upload cap of 25 MiB applies. Under load, a saturated lane answers with a rate-limited response that carries retry timing, never a silent failure.

The route is part of the platform's published API contract today, and two of its rows serve: sam2.1-base and da3-small serve, confirmed by the October 6, 2026 requests. The contract above is the buildable fact: the request shape, the response fields, and the metering unit are published and stable.

Three mistakes to avoid

One: assuming metric depth. Relative depth orders a scene from near to far. It does not measure it. If your robot needs real-world scale, check which output the row returns before you plan around numbers.

Two: assuming point and box prompts. The day-one route takes whole images. A pipeline built on click-to-mask prompts needs a design change, not a config change.

Three: picking a lane on price alone without reading its license. The least costly community depth lanes run weights whose license bars commercial use. The SAM family changed licenses between generations. The tables above are the two-minute check that saves a rewrite.

The vision catalog on this platform

Three rows make up the lane, each with its own page: sam2.1-base, sam2.1-large, and da3-small.

Choosing by the job:

All three rows sit in the platform's model data. sam2.1-base and da3-small are on the public list and serve; sam2.1-large is not. The member panel on this page carries the listed rows. The models index is the live catalog. Rates are set per image and render on the pricing page from the platform's own data, the same source the meters read. Calls draw down a prepaid wallet, and an empty wallet refuses the next call with a named error instead of running a debt.

Every model behind the lane ships open weights, so self-hosting stays open to anyone with the hardware, and the model pages each carry the honest local-versus-hosted trade. What a hosted row sells is the service around the weights: a fixed contract, per-image metering, and no machine to keep alive.

For the other half of robot perception and control, the VLA lane serves vision-language-action models that turn what a camera sees into actions. The two lanes are designed as a pair: this one answers what the robot sees, that one answers what the robot does.

Getting started

1. Read the contract first. The run-vision reference page carries the exact request fields and responses, and the vision route page walks the route end to end.
2. Open a console account. The beta is invite-only for now. The wallet, API keys, and full spend history live there, and an account opened today covers every lane the platform lists.
3. Watch the models index. Two of the three vision rows are on the public list and serve. The index is the live catalog.
4. Read the guides listed with this page. One covers segmentation and depth for robots, the models in this lane, and what each generation of the SAM family is for. The other works through where a robot's inference should live at all, on the robot, on a local box, or in the cloud.

The first call is one curl command with a key from the console and an image from your robot's camera. The route contract above is already fixed, so nothing about that call is a surprise.