Vision route: the /v1/vision endpoint contract

What the /v1/vision endpoint takes and returns: image file parts in, JSON or rendered bytes out via the response_format field, billed per image.

Three models are named on this page, so the state comes first. Not one is on the public list. Each alias returns 404 by name. No rate for this route renders anywhere; the models index is the live catalog, and a listed row's per-image rate appears on the pricing page, drawn from the platform's model data. The contract itself is already published. Write the client now and check it against these shapes today.

What the route is

Strip the wrappers away and a perception endpoint is one remote procedure. You send what a camera saw. It returns what the scene contains: masks that say which pixels belong to which object, or depth data that says how far away those pixels are. Nothing else. No chat, no tokens, no prompt engineering.

The market sells this three ways today. One way is to prompt a general multimodal chat model and parse its JSON. Google publishes the recipe: ask for masks with the keys box_2d, mask, and label, coordinates normalized to a 0 to 1000 grid, and hold the model to that shape with a response schema (Gemini API image understanding docs). It works, and it bills your images as tokens. The second way is a hosted copy of one segmenter that takes prompt fields. Meta serves the current generation of its own SAM line this way: text prompts only, a short noun phrase like "delivery van", one concrete object per phrase, and several requests when you want several object types (Meta media segmentation docs). The third way is a workflow platform's endpoint, built around point clicks sent as JSON coordinates.

This route is a fourth shape, built for loops instead of clicks. The body is multipart form data, the same thing a web form uses to upload files. Your images go as file parts, not as URLs to fetch and not as base64 text inside JSON. The whole image is the input: there are no prompt fields to fill, because a control loop has no human clicking points at control rate. And both halves of perception ride the same contract, so the mask half and the depth half are one integration, not two.

The field by field listing lives on the run-vision reference page, generated from the same contract, field for field. The notes below cover the request, the two answer shapes, the prompt law, the model families, the billing arithmetic, and the error rows. The route that answers what the robot does with what it sees, the action endpoint, has its own page: the action route.

The request, field by field

Every alias on this route returns 404 today. The example below is the exact request for the day that changes. Wire it up now and expect the refusal:

curl -sS -w '\n%{http_code}\n' $IRONSTRATUM_BASE_URL/v1/vision \
  -H "Authorization: Bearer $IRONSTRATUM_API_KEY" \
  -F "model=sam2.1-base" \
  -F "file=@cam0.jpg" \
  -F "file=@cam1.jpg" \
  -F "response_format=json"

Those two variables are the convention across these docs: set your key and the base URL once, and every example runs the same lines against production or a local stack. The quickstart covers creating the key.

The file slot takes one kind of payload, images, and takes any count of parts. One request with four file parts is a normal batch, and it bills four images, not one request. There is no JSON body to build and no encoding to inflate your bytes: the images cross as raw file parts, exactly the way file uploads were designed to cross.

This route has its own forbidden knob: the prompt fields. Send point, or box, or mask, or a text prompt field of any name, and the answer is a 400 unknown_param naming the field you sent. The route takes whole images and returns what the model finds in them. That is deliberate, and the next section explains why.

The response, field by field

The envelope, with placeholder values where the real payload is the model's:

{
  "annotations": [
    { "mask": "the model's own encoding", "area": 18411, "box": [214, 305, 178, 231] }
  ],
  "usage": {
    "images": 2,
    "cost": "the wallet debit, six decimals, USD"
  }
}

The annotation keys above are illustrative. Each row relays its model's own annotation objects byte for byte, and the family sections below describe what those objects hold. The usage object is ours, always. The model's own usage block, whatever it reports about itself, never crosses. Your bill is computed from the request and the served answer, never from a number the model hands back on its own authority.

The count has a precise law. On a JSON answer, the model's reported image count bills when it is a whole number between 1 and one billion. A missing count, a zero, a fraction, or an absurd value falls back to the basis you can see: the number of file parts in your request. You can always predict your bill from what you sent.

One more relay rule, for the rare bad day. If a served JSON answer comes back malformed, the route passes it through verbatim and the bill stands on the count law above. You see exactly what the model produced.

An Idempotency-Key header is accepted and logged, nothing more. Replay is the one thing it cannot do here: this route carries image bytes, and half of a replayed upload is corruption, not safety. A retry after a dropped connection is a second billed request. Make retries deliberate.

The binary answer

Set response_format to binary and the route changes what it hands back. There is no JSON envelope. You receive the bytes the model rendered, unchanged, under the content type the model produced them with: an image/png header over PNG bytes for a rendered mask overlay, for example. Nothing is rewritten on this leg, because there is nowhere to write a usage block into a picture.

Billing stays on the input side. A binary answer bills by the file-part count of your request, always, because the binary leg carries no JSON to read a reported count from.

When do you want binary? When you want the picture, not the polygons. A mask rendered onto the frame is ready for a dashboard, a training sample, or a human review step. When your code needs to compute with the mask, take the default JSON instead and read the annotation objects.

Prompts: what the ecosystem takes, and what this route takes

The prompt vocabulary of the segmentation world is four words wide. Knowing it saves you from building the wrong client.

This route takes none of them, day one. The request has three fields: model, file parts, and the format switch. Send a prompt field and the 400 names it.

The reason is the loop. Points and boxes are annotation tools' vocabulary: a human looks, clicks, and the model segments what was clicked. A robot perception pass runs many times a second with no human in it. The whole-image shape is the publisher's own answer to that: SAM's automatic mask generation mode finds and masks every object it can see in the frame, no prompt required, and it has been part of the line since the first generation (SAM 2 repository). Mask everything, then filter by your own logic, is a loop-native design.

Point and box prompt fields do not exist on this route; sending either draws the 400 unknown_param answer. Build against what is printed here.

The vision families, model by model

Three rows are written up for this route. None of them serves today. What follows is what each family returns, from the publishers' own documents.

The segmentation half. The SAM 2.1 generation is the improved checkpoint set of Meta's Segment Anything line, released September 29 2024 across four size tiers: tiny, small, base-plus, and large (SAM 2 repository). This route plans the two tiers robot work lives in. The base-plus tier is the volume pick: 80.8M parameters at 64.1 frames per second on the publisher's A100 benchmark. The large tier is the accuracy pick: 224.4M parameters at 39.5 on the same benchmark, sharper on crowded and textured scenes. Both tiers ship under Apache 2.0, and the grant covers the model checkpoints themselves, not just the code (SAM 2 repository). The Hugging Face model cards are open to download and tagged for mask generation: base-plus and large. The license story of the wider SAM family, including what changed in the newer generations, is mapped on the vision lane page.

The depth half. Depth Anything 3 comes from the ByteDance-led team behind the Depth Anything line. The 2025 paper describes a model that recovers geometry from an arbitrary number of views, with or without known camera poses, using one plain transformer backbone (DA3 paper, 2025). The small tier is 0.08B parameters under Apache 2.0 (DA3 repository), open to download on its model card. What it hands back, in the publisher's own API shapes:

One law matters more than the rest: relative depth. The main DA3 series, this row included, returns relative depth, which orders the scene from nearest to farthest. It does not measure meters. The metric variant of the family is a different tier, fine-tuned for real-world scale (DA3 repository). If your robot needs true distances, plan the conversion on your side, or watch for a metric row. None is promised.

The publisher's API also takes a list of images in a single call (DA3 API doc). Send one image, get its depth map. Send several views of one scene, get depth consistent across them plus the camera poses between them. On this route, the file parts you send are the list the model gets. The batching lever and the multi-view lever are the same lever: more parts in one request.

How a call is billed

The unit is the image. Not the request, not the token, not the mask. A request carrying three file parts bills three images.

The count law, stated once more in one place: a served JSON answer may carry a reported count from the model. A whole number between 1 and one billion bills as the count. Anything else, missing, zero, fractional, or absurd, falls back to your file-part count. A binary answer always bills by the file-part count. Refused requests bill nothing, whatever the refusal: an empty wallet, a body over the cap, a saturated lane. And a retry bills again, because image bodies are not replayed.

The whole multipart body, every file part plus the form fields, sits under a 25 MiB cap shared by every upload route on the platform. Cross it and the answer is a 413 before anything is sent to the model, and the error names the file slot. A camera frame resized to a sensible transport size fits easily. If you must shrink an image to fit, know the trade: detail removed before the bytes leave your machine never comes back.

Some perception hosts keep the encoded image between calls, so a second prompt against the same picture costs less than the first. This route keeps nothing between calls. Every request is a fresh turn, and its bill is the images it carries. If your loop re-sends the same frame, budget it as a new image every time.

No rate for this route renders. The pricing page carries the listed rows' rates, drawn from the platform's model data, the same source the meters read.

Calling the route from perception code

A perception pass in Python:

import os
import requests

BASE = os.environ["IRONSTRATUM_BASE_URL"]
KEY = os.environ["IRONSTRATUM_API_KEY"]

def see(model, frames):
    response = requests.post(
        f"{BASE}/v1/vision",
        headers={"Authorization": f"Bearer {KEY}"},
        data={"model": model, "response_format": "json"},
        files=[("file", f) for f in frames],
        timeout=30,
    )
    response.raise_for_status()
    return response.json()

answer = see("sam2.1-base", [open("cam0.jpg", "rb"), open("cam1.jpg", "rb")])
findings = answer["annotations"]      # the model's own mask objects
billed = answer["usage"]["images"]    # the count law above

The same request as a one-liner with any HTTP CLI, this one HTTPie, fetching a rendered binary answer from the depth row:

http --form POST $IRONSTRATUM_BASE_URL/v1/vision \
  Authorization:"Bearer $IRONSTRATUM_API_KEY" \
  model=da3-small response_format=binary \
  file@frame000.jpg --download

The pattern that makes both halves worth one route is old arithmetic. A mask tells you which pixels belong to the object. A depth map tells you how far those pixels are. Select the mask's pixels inside the depth map and you have the object's position in space, which is what a grasp planner or an avoidance check actually needs. The open source robot stack runs this pattern with a self-hosted segmenter beside a self-hosted depth model and hand-written glue. Here it is two requests to one contract, and the glue is a lookup.

How fast to call is not this page's arithmetic. A perception pass feeds a control loop, and the chunk and buffer math that keeps a loop from stalling belongs to the action route.

Errors, limits, and the launch posture

Every response, success or error, carries an x-request-id header. Quote it in any support conversation. The full error envelope, the stable code list, and the SDK mapping live on the errors page.

Everything above describes a contract that is published and three rows that are not on the public list. The models index is the live catalog. The run-vision reference page carries the machine shape of this contract at all times. The model class, and the lane this route serves, lives on the vision page.

Frequently asked questions

What is the /v1/vision endpoint?

A hosted perception endpoint. Your code posts one or more image file parts. The answer is either annotation JSON, masks or depth data in the model's own objects plus our usage block, or the model's rendered bytes when you ask for binary. It is the perception half of robot work: what the camera saw, turned into data a program can act on.

What does the response contain?

Two fields on a JSON answer. annotations, an array of the model's own finding objects, masks for the segmentation rows and depth data for the depth row, passed through unchanged. usage, our billed image count and the wallet debit. The model's own usage block never crosses.

Can I send point or box prompts?

No. The day-one shape is the whole image: the route runs the model's automatic mode over everything in the frame. Sending point, box, or any prompt field is a 400 unknown_param that names the field. If prompts ever join the route they will be new named fields.

Can I send video?

No. The route takes image file parts. A video stream is a sequence of image requests if you drive it that way, but the route holds no state between them, so tracking one object across frames is not something it does for you. The SAM line's own video mode, which carries memory between frames, is a different design, and no video row is promised here.

How is a call billed?

By the image. A request with three file parts bills three images. On a JSON answer, a model-reported count that is a whole number between 1 and one billion bills as the count, and anything else falls back to your file-part count. Binary answers bill by the file-part count. Refused requests bill nothing. A network retry bills again, because image bodies are not replayed. No rate renders for this route; the pricing page carries the listed rows' rates.

What is the difference between json and binary?

The response_format form field picks. json, the default, wraps the model's annotation objects with our usage block: it is what code computes with. binary returns the model's rendered bytes, a mask image for instance, unchanged under the model's own content type, billed on the input side: it is what you show or save.

Why does sam2.1-base answer 404?

It is not on the public list. None of the three rows on this route serves today. The 404 is the honest answer, and the models index is where that state reads.

Small is the point of this contract. Three form fields in, annotations and usage out, one image per billed unit, and a format switch that trades polygons for pictures. The parts that save you a rewrite are here: the day-one prompt law said plainly, the count law you can predict your bill from, and both halves of perception behind one shape. The vision page holds the lane and the families. The run-vision reference page holds the machine-derived fields. What the robot does with what it sees starts at the VLA lane. An account opened on the console today covers every lane the platform lists.

Last verified: 2026-10-05