Robot vision API: the perception layer as a product
A robot vision API is an endpoint that takes camera frames and returns the geometry a robot acts on. That geometry comes in three forms: masks, depth maps, and camera poses. The robot sends what its cameras saw. The endpoint answers what the scene looks like, in a form code can use, on a clock the control loop sets. As a named product category, it barely exists. The first page of results for the phrase is a mix of hobby repositories, integrator pages, academic papers, and generic cloud vision docs. None of it sells what a robot builder is looking for. This page names the category anyway, because the parts are real. The outputs are settled. The sellers exist in three shapes. A per-image lane is live on this platform as of October 28, 2026.
Last verified: 2026-10-28
What counts as a robot vision API
Three neighbors crowd the search results. Splitting them apart saves a wasted test.
The first neighbor is generic cloud vision. The big cloud providers sell vision endpoints that return labels, faces, landmarks, printed text, and safety flags. Those are facts about photos. A robot that reaches for a cup needs the cup's boundary in the frame, not a caption. Generic endpoints also publish no latency story for control work. They are built for occasional queries, and a perception loop is anything but occasional.
The second neighbor is the machine vision integrator. Search the phrase and you will find industrial vision guides and cell builders. Their product is a hardware-tied system, cameras plus processing plus acceptance testing, sold as an installed cell. That is a legitimate purchase. It is not an API. Nothing about it meters per image, and nothing about it changes when your robot does.
The third neighbor is the name itself. A perception API in the wild can mean a lidar zone-occupancy product for security monitoring. It has no robot control in it anywhere. Before you compare two endpoints, read what they return. Here is the test that cuts through all three neighbors. A robot vision API takes images in, returns geometry out, and bills by the image or the request.
What the robot consumes
A robot acts on three kinds of geometry. Every serious vision output falls into one of them.
| Output | What it is | The robot job it feeds |
|---|---|---|
| Mask | A per-pixel boundary around each object | Grasping, part checks, workspace tests |
| Depth map | A per-pixel distance, relative or metric | Navigation, arm approach, free-space shape |
| Camera pose | Where each view sits relative to the others | Multi-camera rigs, scene reconstruction |
Masks answer where the object ends. Depth answers how far away it is. Poses answer how several views fit together, which is what turns a two-camera rig into one scene instead of two. Relative depth orders the scene from near to far, which is enough for obstacle logic. Metric depth carries real-world scale, which measurement and mapping need.
The outputs are not exotic research objects. They arrive as arrays your code reads directly. A 2025 paper from the team behind the Depth Anything line describes a single model that recovers depth and camera poses. It works on any number of views, with or without known poses, on one plain transformer (paper, 2025). The mask half is just as settled. The vision lane page maps both model families, their licenses, and their tiers.
Three ways vision inference is sold
Nobody sells a product called a robot vision API today. The market still sells the capability, in three shapes, and they differ in what you prompt and how you pay.
| Shape | How you prompt it | How it bills | Built for |
|---|---|---|---|
| Prompt a general multimodal model | Ask in text, hold the shape with a response schema | Per token, images billed as tiles | Occasional queries |
| A hosted copy of one segmenter | A short noun phrase, one object per request, or clicked points | Per request | Annotation workflows |
| A dedicated per-image route | The whole frame, no prompt fields | Per image | Control loops |
The first shape works better than its reputation. Google publishes the recipe. Ask for masks with fixed keys and coordinates on a 0 to 1000 grid, and hold the model to that shape with a response schema (Gemini API docs). It parses, and for a one-off question it is fine. The catch is the meter. Images bill as tiles, 258 tokens per 768-pixel tile in the published math. Every frame costs tokens, and the cost scales with resolution. A loop that sends every frame pays photo prices for geometry.
The second shape narrows the prompt to a noun phrase. Meta serves the current generation of its own SAM line this way. The prompts are text, boxes, and points: one concrete object per phrase, and several requests when you want several types (Meta docs). That rule set fits a labeling tool, where a human clicks and waits. A loop has no human in it, and no human clicks points at control rate.
The third shape is the one this platform takes, and the reason is the loop. The whole frame goes in as file parts. No prompt fields exist. The model returns everything it finds, and your code filters. Masking everything and filtering by logic is a loop-native design. It is not new. Automatic mask generation, with no prompt at all, has been part of the SAM line since the first generation (SAM 2 repository).
The loop sets the deadline
Speed is not a feature here. It is the entry condition, and the numbers come from the action side of the stack.
A policy, the network that decides what the robot does, runs at tens of calls per second. A 2026 study frames 10 calls per second as the acceptable floor and 100 as high performance. The same study measures a pi 0 class policy at about 19 cycles per second on a robot-grade Jetson module (VLA-Perf study, 2026). The original pi 0 paper reports control at up to 50 cycles per second. Fresh inference runs twice a second, every 25 actions (pi 0 paper, 2024).
Two design facts fall out of those numbers. First, answers arrive in chunks. The robot runs the current chunk while the next one computes. A slower answer degrades smoothness instead of stalling the arm. Your perception layer joins that pipeline, and it does not get a second and a half per frame. Second, the network fits the budget when the serving side is close. Physical Intelligence runs its policy inference remotely in production. The published case study reports 10 to 15 milliseconds of added network overhead, with deployments pinned near the robots (case study, 2026).
This is where the generic tools fail, quietly. A labeling endpoint that answers in seconds is measuring human patience. A perception answer that lands in tens of milliseconds, at camera rate, is a different product with a different meter. When a vendor prints no latency number anywhere, assume the second product is not what is for sale.
The metering shape, without numbers
This section sets the math, because no rate on this page is typed by hand.
The three shapes above bill three ways: per token, per request, per image. For a robot, the unit that predicts the bill is the camera. One camera at 10 frames per second produces 36 thousand frames an hour. Two cameras double it. A perception pass on every frame multiplies the metering unit by that count, whichever unit it is. The token shape adds a second multiplier, since bigger frames tile into more billed units.
The honest planning move is to compute the hourly number for your own loop. Do that before you compare any two vendors. Rate times frames times hours. The pricing page renders every listed rate on this platform from the platform's own model data. The pricing guide explains the units and the wallet. Spend draws down a prepaid balance. The balance is the hard stop: a call that arrives after the money is gone is refused with a named error, not billed. A robot that loops all night cannot run a debt by accident.
Where the vision lane stands today
The dated state, as of October 28, 2026. Each sentence mirrors the platform's own model data, and the models index is the live catalog.
Two of the three vision rows are on the public list, confirmed there since the October 6, 2026 requests. Those two are sam2.1-base, the base mask model, and da3-small, the small depth model. The large mask tier, sam2.1-large, is not on the public list, and no date rides with that fact. The vision lane page carries the lane in full: the families, the license map, and the member panel.
The route meters per image, takes multipart form data, and returns the model's findings plus a usage object. Both served families ship under Apache 2.0. The grant covers the model weights, not just the code (SAM 2 repository, Depth Anything 3 repository). Perception is also only half of what a robot senses. The action half is the VLA lane: vision language action models that return motor commands. Scene reasoning rides two listed rows on the chat category. The robot inference API page maps all three classes in one place.
A first call, on a row that serves
The route contract is published, and the request is three fields. This call runs against a listed row today:
curl -sS -w '\n%{http_code}\n' $IRONSTRATUM_BASE_URL/v1/vision \
-H "Authorization: Bearer $IRONSTRATUM_API_KEY" \
-F "model=sam2.1-base" \
-F "file=@cam0.jpg" \
-F "response_format=json"Set the key and the base URL once. Every published example then runs the same way. The body is multipart form data, the same thing a web form uses to upload a file. One file part per frame: a request with four parts bills four images. The answer holds the model's findings, passed through unchanged. It also carries a usage object with the billed image count and the wallet debit. A response_format switch asks for rendered bytes instead. That returns a mask overlaid on the frame, ready for a dashboard or a review step. Send a prompt field of any name and the answer is a 400 that names it. The field-by-field contract lives on the vision route page and the run-vision reference. Keys start at the signup page.
Running perception yourself
Both served families carry open weights, so the self-host path is real: download the models, write the loop, own the machine. Two costs hide in that sentence.
The first is glue. A self-hosted stack usually runs a segmenter beside a depth model. The glue is hand-written code that fuses the two, projecting masks onto depth to get 3D points. That glue code is plain engineering work, and it is yours forever. The second is throughput. Practitioners report that SAM-class segmentation crawls on CPUs. That pushes you toward a graphics card per robot, or toward a smaller, less accurate model. The Jetson versus cloud guide works that trade with real numbers. The segmentation and depth guide covers the model-choice side.
Hosting earns its keep on the parts around the model. An endpoint that stays up through a long session. Metering and spend history you did not write. Headroom on the busy shifts. And the freedom to try the other tier by changing one string in the request. Teams whose robot is a product rather than a research project tend to make that trade. Teams still learning what their robot needs tend not to yet.
Where to go next
- Open a console account. The wallet, the keys, and the spend history live there, and you can start at the signup page. One account covers every lane the platform lists.
- Read the rates once, in one place. The pricing page renders every listed row's rate from the platform's own model data, and the pricing guide explains the units.
- Pick your surface. Perception: the vision lane page. Action: the VLA lane. Scene reasoning: the chat category. The robot inference API page maps all three.
- Read the spokes listed with this page. The segmentation and depth guide carries the output decision. The mask generation guide covers the whole-frame workflow. The SAM versus YOLO guide handles the speed tradeoff, and the CPU-tier guide handles the depth side. The alternatives guide maps the perception-layer vendors, and the VLM versus VLA guide draws the boundary between seeing and acting.
The robot lanes are young on every platform, this one included. Read them the way you would read any metered service. Check the dated state, run the first call, and watch the meter before you scale. The models index lists every row the platform carries, and every lane draws from one wallet.