SAM 2.1 automatic mask generation: the exact call, every knob, and the real runtime

Automatic mask generation is the mode where SAM 2.1 masks every object in an image, with no prompt from you. One class in the official repository runs it: SAM2AutomaticMaskGenerator. One call does the work. The cost is arithmetic, not money. At the stock settings the class fires 1,024 point prompts per image. A model rated at 64.1 frames per second still takes seconds per image in this mode. This page walks the exact call, every knob with its stock value and its cost, the runtime receipts from named sources, and a hosted lane that runs the same sweep per image. Each source was read on 28 October 2026.

Last verified: 2026-10-28

What the sweep actually does

The design comes from the original SAM paper. Meta writes that in fully automatic mode, "we prompted the model with a 32 x 32 regular grid of points and for each point predicted a set of masks that may correspond to valid objects" (SAM paper, section 4.3). The model returns up to three masks per point. One point can sit on a part, a subpart, or a whole object. The class then filters the pile in three steps. Masks whose own quality score clears a threshold stay. Masks that hold still when the cutoff shifts stay too. Box non-maximal suppression then drops the duplicates. Small objects get a second pass on zoomed crops when you turn crops on.

The 2.1 release changed the weights, not this design. Meta shipped the improved checkpoints on September 29, 2024 across the four size tiers, tiny to large (SAM 2 repository). The AMG code itself is adapted from SAM 1, as the header comment in the source file states.

Each mask comes back as a record with seven keys, named in the official notebook: segmentation (the mask itself), area in pixels, bbox in XYWH form, predicted_iou (the model's own quality guess), point_coords (the sampled point that made this mask), stability_score, and crop_box. Sort on predicted_iou when you want the best masks first. The hosted Replicate copy of this exact pipeline exposes that sort as its mask_limit knob, so you can ask for the top N masks only (Replicate row, read 28 October 2026).

The minimal path

Three steps, all from the official README and notebook. The code wants Python 3.10 or newer, torch 2.5.1 or newer, and torchvision 0.20.1 or newer.

Install the package:

pip install -e .

Fetch one checkpoint. The 2.1 files live under the 092824 folder:

wget -P checkpoints/ https://dl.fbaipublicfiles.com/segment_anything_2/092824/sam2.1_hiera_large.pt

Build the model and run the sweep. The config name must match the checkpoint: the large 2.1 weights pair with configs/sam2.1/sam2.1_hiera_l.yaml.

import numpy as np
from PIL import Image
from sam2.build_sam import build_sam2
from sam2.automatic_mask_generator import SAM2AutomaticMaskGenerator

sam2 = build_sam2(
    "configs/sam2.1/sam2.1_hiera_l.yaml",
    "checkpoints/sam2.1_hiera_large.pt",
    device="cuda",
    apply_postprocessing=False,
)
mask_generator = SAM2AutomaticMaskGenerator(sam2)

image = np.array(Image.open("frame.jpg").convert("RGB"))
masks = mask_generator.generate(image)
print(len(masks), "masks")

That is the whole official path. The notebook that ships it recommends an A100 or L4 class card in Colab, and warns that a T4 "might be slow and might run out of memory in some cases".

There is a second path with no checkpoint download and no config file: the transformers library. The model card for the base-plus tier shows it in four lines:

from transformers import pipeline

generator = pipeline("mask-generation", model="facebook/sam2.1-hiera-base-plus", device=0)
outputs = generator("truck.jpg", points_per_batch=64)
print(len(outputs["masks"]), "masks")

The repository package gives you every knob. The pipeline gives you the main ones. Both run the same sweep under the hood.

Every knob, its stock value, and its cost

The class constructor takes 17 named arguments. Sixteen arrive with stock values, and the source file documents each one. The table lists the ten that change what you get or how long it takes.

One number in that table trips people who copy settings from older guides. The SAM 2 class ships pred_iou_thresh at 0.8. SAM 1 shipped it at 0.88, and guides written against SAM 1 still carry the older number. Both numbers are read from the two source files, this session. The stock settings in SAM 2 are tuned, in the words of the source docstring, "for SAM 2 with a HieraL backbone".

The official notebook also ships a dense config, meant to sample more masks. It is worth seeing side by side with the stock values, because the one hosted copy we read runs exactly this dense set:

mask_generator = SAM2AutomaticMaskGenerator(
    model=sam2,
    points_per_side=64,
    points_per_batch=128,
    pred_iou_thresh=0.7,
    stability_score_thresh=0.92,
    stability_score_offset=0.7,
    crop_n_layers=1,
    box_nms_thresh=0.7,
    crop_n_points_downscale_factor=2,
    min_mask_region_area=25.0,
    use_m2m=True,
)

Read the shape of that config before you copy it. The point grid doubles per side, from 32 to 64, so the prompt count quadruples, from 1,024 to 4,096. The batch doubles to 128. One crop layer adds 4 more crops, each roughly two thirds of the image per side. The quality filters all loosen. That config finds more, smaller masks, and it costs roughly four times the compute of stock.

The runtime nobody quotes you

The frame rates in the repository table measure prompted prediction, not this mode. The base-plus tier reads 64.1 frames per second on an A100 in that table, one prompt per frame. Automatic mode is a different job: one image costs 1,024 prompts at stock, run 64 at a time, which is 16 decoder passes over one encoder pass. A single mask request is one prompt. That is why the two numbers never match, and why a card that serves a camera fine can still take seconds per image here.

Measured receipts, each with its owner and date. The SAM 1 rows are the v1 class at stock settings; the sweep design is the same in 2.1, so they set the order of magnitude, not an exact forecast.

That last row is the newest receipt on the page. The 2026 study linked in the tuning section below ran the SAM 2.1 Hiera-Large checkpoint at unmodified library defaults on an RTX 4080 with 16 GB, and its stock evaluation averaged 8.01 seconds per patch across 540 patches of 512 x 512 pixels. The patches are crops of microscope slides, not camera frames, so read it as a per-patch anchor, not a per-image forecast. What no guide or thread quotes is a per-full-image consumer-card timing. Four searches were run for it on 28 October 2026:

• "sam2 automatic mask generation time seconds per image benchmark"
• "sam2 automatic mask generation seconds colab A100 4090 timing"
• "SAM2 automatic mask generation slow"
• "Effective SAM2AutomaticMaskGenerator settings"

All four returned guides and wrapper notebooks, never a measured per-image number. For whole frames the SAM 1 band above still sets the order of magnitude, with the 8.01 second patch figure as the one measured SAM 2.1 anchor. Measure on your own frames before you promise anyone a number.

The levers that actually move the runtime, in the order they pay:

1. Cut points_per_side. From 32 to 16 is 256 prompts instead of 1,024, a quarter of the decoder work. The official SAM 1 issue thread on this exact question says the same: fewer points, fewer masks, faster.
2. Raise points_per_batch if memory allows. Fewer decoder passes, same prompt count. This is also the one knob that matters for AMG memory pressure; the memory writeup covers the rest of that story.
3. Turn crops off. crop_n_layers=0 is stock, and one extra layer is 4 more encoder passes, two layers is 16.
4. Drop a tier. Tiny runs 91.2 FPS against 64.1 for base-plus on the publisher's A100 table, at some accuracy cost. The tier sheet carries the full table.

Running several generators side by side on one card is its own subject, with its own failure modes. The thread safety page and the predictor pool writeup own that story in full.

What tuning buys, from a 2026 study

A peer-reviewed study from October 2026 tuned these knobs on a fixed SAM 2 model. The domain was hard: petrographic thin sections, rock slices under a microscope. Its authors searched nine generator parameters with an Optuna budget over five slides. They then tested the winner on a held-out sixth slide (Minerals, MDPI). Their result, in their own words: "parameter tuning produced only modest gains, while spatial agreement with the associated references remained limited." The clock agrees: tuning cut the sweep from 8.01 to 6.72 seconds per patch on the same card, about 16 percent.

That is the measured state of the "best settings" question. In an easy domain, like the cars photo in the official notebook, stock settings already find the objects. In a hard domain, no knob setting rescues a model that was never trained on your texture. The order of operations that follows: run stock first, count what you get, loosen filters one at a time, and only then pay for crops and a denser grid. Tuning is a scalpel, not a rescue.

The errors people hit first

The torchvision NMS crash. Prompted prediction works, then AMG fails with:

NotImplementedError: Could not run 'torchvision::nms' with arguments from the 'CUDA' backend

The cause is a torchvision build without CUDA support. AMG reaches the NMS step, prompted prediction never does, which is why the error waits for this mode. The fix on issue 220 was to reinstall torchvision alone with the CUDA build. The reporter's result: "works about 30 times faster now". Check this before you debug anything else.

The missing opencv postprocess. Setting min_mask_region_area above zero requires opencv in the environment, per the source docstring. Without it, the cleanup step has nothing to run on. The apply_postprocessing=False flag in the build call plays the same role one level down: it skips the model-side cleanup at build time, which is what the official notebook does.

The config and checkpoint mismatch. A 2.1 checkpoint with a 2.0 config name fails at model build, not with a mask error but with a Hydra config exception. Pair sam2.1_hiera_large.pt with configs/sam2.1/sam2.1_hiera_l.yaml, exactly as the README table pairs them. The version map between SAM 2 and 2.1 has its own comparison page.

The lane with no knobs

The same family runs as a hosted lane on this platform's vision route, under the sam2.1-base model id. The route runs the automatic mode by design: whole image in, every mask out, no prompt fields at all. Send a point or box field and the answer is a 400 that names the field. The row has served since 6 October 2026 and sits on the public list as of 28 October 2026. The large tier is not on the public list, and no date rides with it.

A first call, whole thing:

curl -sS $IRONSTRATUM_BASE_URL/v1/vision \
  -H "Authorization: Bearer $IRONSTRATUM_API_KEY" \
  -F "model=sam2.1-base" \
  -F "file=@frame.jpg" \
  -F "response_format=json"

import os
import requests

resp = requests.post(
    os.environ["IRONSTRATUM_BASE_URL"] + "/v1/vision",
    headers={"Authorization": "Bearer " + os.environ["IRONSTRATUM_API_KEY"]},
    data={"model": "sam2.1-base", "response_format": "json"},
    files=[("file", open("frame.jpg", "rb"))],
    timeout=30,
)
resp.raise_for_status()
answer = resp.json()
print(len(answer["annotations"]), "mask objects, billed", answer["usage"]["images"], "images")

The bill counts images, not masks and not prompts. A request with three file parts bills three images. The sweep cost is ours, not yours: the 1,024 prompts inside one image bill as one image. A refused request bills nothing. Rates render on the pricing page, never typed by hand on pages like this one:

The field-by-field contract lives on the run-vision reference, and the route's full walkthrough is on the vision route page. The model sheet for this tier is the sam2.1-base page. When a hosted call fits your loop, open the console, create a key, and send one frame. For what masks feed once you have them, read the segmentation and depth guide.

Frequently asked questions

How do I run automatic mask generation with SAM 2.1?

Build the model with build_sam2, wrap it in SAM2AutomaticMaskGenerator, and call generate on a NumPy image array. The three-step path with the exact code is on this page above. The transformers pipeline is the shorter path: pipeline("mask-generation", model="facebook/sam2.1-hiera-base-plus"), then call it on an image.

Why is automatic mask generation slow?

Because the stock sweep fires 1,024 point prompts per image, 64 at a time. The image encoder runs once, then the decoder runs 16 batches. The receipts above run from 3.4 seconds on an A100 to 27.5 seconds on a consumer card. One published SAM 2.1 stock timing averages 8.01 seconds per 512 x 512 patch on an RTX 4080, and a hosted dense-config run took 16.2 seconds on an L40S. Cut points_per_side first: 16 per side is a quarter of the work.

What are the best settings for SAM2AutomaticMaskGenerator?

There is no universal best, and a 2026 peer-reviewed tuning study found "only modest gains" from optimization in a hard domain. Start at stock: 32 points per side, 0.8 quality threshold, no crops. To find more small masks, copy the official dense config on this page: 64 points per side, batch 128, one crop layer, looser filters at 0.7 and 0.92. Expect about four times the compute.

What is the default pred_iou_thresh in SAM 2?

0.8. The SAM 1 class used 0.88, and many copied settings blocks online still carry the older number. Both values were read from the two source files on 28 October 2026. The official dense config goes the other way, down to 0.7, to keep more candidate masks.

Can I run it on CPU only?

Yes, slowly. The repository code runs on CPU. The install needs a CUDA toolkit only to compile the package, a point confirmed on issue 71. One reporter in the NMS thread ran the CPU path and called it "infinitely slow". A 1,024-prompt sweep on CPU cores is a batch job, not a loop. For camera-rate work with no GPU, the hosted lane above is the practical path.

Is there an API for SAM 2.1 automatic mask generation?

Yes, this platform's vision route, metered per image against a prepaid wallet. Community copies also run on Replicate, and Meta's own API serves the newer SAM 3.1 with text prompts, not the 2.1 sweep. The seller census with dates lives on the sam2.1-base model page.

The sweep is 1,024 prompts, the stock values sit in one table with their costs, and the receipts are named above with their owners. Run stock first, measure on your own frames, and loosen one filter at a time. When the setup cost is not worth it, the vision lane runs the same automatic mode per image, with rates on the pricing page and a start at signup.

Last verified: 2026-10-28