The SAM 2 mask generator is not thread-safe: the crash we measured, the cause in the code, and the two fixes

The SAM 2 automatic mask generator is not thread-safe, and the crash has an exact shape. One generator owns one private image predictor. Every generate call rewrites that predictor's inner state, then reads it back. Two generate calls racing on one generator corrupt that state. In our wave of 26 requests on 2026-10-05, five died on the same line: TypeError, 'NoneType' object is not subscriptable, inside automatic_mask_generator._process_crop. Nineteen came back clean. Two were turned away by the queue. This page carries the receipts: the code that makes the generator unsafe, the lock that fixed it and what it cost, and the pool that replaced the lock one day later. Every source was read on 28 October 2026.

Last verified: 2026-10-28

The receipt: 26 requests in, 5 crashes out

The setup was plain. The open SAM 2 code, one shared mask generator, and a queue in front of it. The queue let four requests run at once, with sixteen waiting. On 2026-10-05 we fired 26 requests at it in one burst. The table is the whole census, read from the service logs that day.

Read the two 429s as the queue doing its job. Read the five 500s as the bug. Each failing request returned the same error body, quoted from the log record:

sam21_base_failed: TypeError("'NoneType' object is not subscriptable")

And every traceback landed at the same place, the crop loop inside the generator:

automatic_mask_generator._process_crop

No report in the model's own tracker matches this. We searched it on 28 October 2026. The tracker holds four reports containing the word NoneType. None of them is a crash from concurrent generate calls. If your service log shows this string under load, you have found what we found.

Why generate corrupts its own predictor

From the outside, the generator looks stateless. You pass one image in, and you get a list of masks back. Inside, it keeps exactly one image predictor for its whole life, built in the constructor. From automatic_mask_generator.py in the repository, read 28 October 2026:

self.predictor = SAM2ImagePredictor(
    model,
    max_hole_area=min_mask_region_area,
    max_sprinkle_area=min_mask_region_area,
)

One model, one private predictor, shared by every generate call the object ever serves.

A generate call walks the image in crops. For each crop, the _process_crop step runs the same three moves in order. It sets the crop into the predictor. It sweeps a grid of points through it in batches. Then it clears the predictor at the end of the crop:

self.predictor.set_image(cropped_im)   # start of the crop
for (points,) in batch_iterator(self.points_per_batch, points_for_image):
    ...
self.predictor.reset_predictor()       # end of the crop

The poison is the first thing set_image does. From sam2_image_predictor.py, its opening statement is a full reset of four fields. The fields live on the predictor object:

self._is_image_set = False
self._features = None
self._orig_hw = None
self._is_batch = False

After the reset, set_image runs the image encoder. It writes the new features into _features, and it sets the flag _is_image_set to True. The point sweep then reads those features, batch by batch. The batch size is points_per_batch, 64 by default.

Now overlap two requests on one generator. Request A is midway through the point sweep of its second crop. Request B enters generate, and calls set_image for its own first crop. The first thing B's call does is write _features = None. The next batch of A reads self._features["image_embed"]. That is a subscript on None. Python answers with the TypeError from the receipt. That is the whole crash. Not a graphics card fault, not a driver, not memory. Plain shared Python state, reset by one thread under the feet of another.

The window is not narrow. Every crop boundary is another reset and refill cycle. The sweep between them reads the shared fields many times over. Our crash receipt came from a CPU test host. The mutation is pure Python state, so the failure class carries to any device. We did not wait to collect a graphics card crash. The fix came first.

There is a second loud string in the same window, and you may meet it first. A predict that lands after a reset, before the flag goes up, raises RuntimeError: An image must be set with .set_image(...) before mask prediction. That string belongs to the bare image predictor. The full map of which SAM 2 object holds what state, and which are safe to share, is on the companion page Is SAM 2 thread safe. This page stays on the generator.

Nobody warns you

We looked for a writeup before writing this one. The cupboard is empty. Each of these was read on 28 October 2026:

• The repository README: zero matches for the word thread, read in full.
• The Hugging Face model cards for the SAM 2 and SAM 2.1 tiers: zero matches.
• The second build of the model inside the transformers library, with its own docs page: zero matches.
• The generated wiki pages over this source tree: a clean component table, no note on threads. One is even generated over the same source revision our receipts ran.

One artifact in the repository does face the question. Pull request 643, titled "Make image predictor thread-safe", opened on 6 May 2025. Its first line states the problem: "SAM2ImagePredictor is not thread-safe as it stores state variables after set_image() or set_image_batch(). This can be cumbersome to handle in a multi-threaded environment e.g. Web App." As of 28 October 2026 it has zero comments and zero reviews. It also targets the image predictor, not the mask generator. No pull request in the repository proposes a fix for the generator.

The demand behind the empty cupboard is measurable. Eighteen issues and pull requests in the repository mention thread in some form today. A search across all of GitHub for "sam2 thread" returns 934 results today. Serving questions gather around the code. One report tracks a segmentation loop dying inside a TorchServe worker (issue 537). Another tracks two predictors refusing to run faster than one (issue 652). The question gets asked everywhere, and answered nowhere. That gap is why this page exists.

Fix one: the lock, and what it costs

When the people who wrote the model serve it to many users from one process, they use a lock. The demo backend in the repository keeps one predictor, and wraps every model call in one inference_lock. We read the demo file on 28 October 2026, and counted seven call sites behind that one lock. One at a time is the shape the authors chose for themselves. It was also our first fix, on 2026-10-05.

The rule fits in one line: lock the whole generate call, not set_image.

import threading

lock = threading.Lock()

def serve(image):
    with lock:
        return generator.generate(image)

The whole call has to sit inside the lock. The reason is inside generate itself: it re-enters set_image on its own, once per crop. A lock around your own set_image line protects nothing. Its crops reset and refill the shared predictor, locked line or not. One unit, one lock, every caller in a line.

What the lock bought, and what it cost, in our own numbers from the same burst shape:

• Bought: the 26-request burst re-ran with zero failures of class 500. The TypeError became impossible. No second call could ever enter the shared predictor.
• Cost: one generate at a time, nothing overlapped. A generate under the lock ran about 2.5 seconds on our 24 GB RTX 3090 card. That puts the ceiling near 0.4 requests per second. Our arithmetic on a burst of ten: the tenth request waits around 25 seconds.
• The queue in front kept firing its clean 429s, each with a retry hint. It did its job. Behind the lock, there was nothing left to corrupt.

Correct, and slow. A one-at-a-time service leaves the card idle in the gaps between calls. It leaves every burst in a line. The fix held for exactly one day before we replaced it.

The failure you will not see

The TypeError is the good outcome. You get a crash, you find a page, you fix it. The bad outcome prints nothing.

Request A is midway through its point sweep. Request B runs a complete set_image between two of A's batches. The flag is up, because B's call finished. The features in the predictor are B's. A sweeps them without a whisper, and returns masks from B's crop, as a normal 200, to whoever sent A. A load test that only counts exceptions passes while the outputs are wrong. Count both classes. Otherwise the wrongness surfaces far from your logs, as a number that makes no sense. The companion page walks this silent class for every SAM 2 surface. For the generator, the defense is the same as for the crash. Never let two requests share one generator at the same time.

Fix two: the pool that replaced the lock

The safe serving shape for this generator is a pool. Build a fixed number of generators over one shared model. A request borrows an idle generator, runs its generate alone, and puts the generator back. No two requests ever share a generator. No request can reset a predictor under another one.

Sharing the model itself is safe, and the reason is short. A forward pass only reads the model. A PyTorch developer put the framework rule in one line on the PyTorch forums, in 2020: "all the objects in pytorch are thread safe to read. But are not to write into." The generator writes predictor state, never model weights. The pool puts the mutable part where exactly one request touches it. The shared part stays the read-only one.

Our receipts for the pool, from 2026-10-06, one day after the lock:

Two details made it hold. First, warm-up. One real generate runs through every pool slot before the service reports ready. No first request pays a compile stall inside live traffic. Second, pool width is a measured choice, not a knob to max out. The card's compute runs out well before its memory does. The maintainer reply on issue 652 says it in one line: "SAM2 is a very expensive model, so it's highly likely that you're saturating your GPU's compute capabilities, such that there's no more gains to be had from further parallelism." We stopped at two.

The full pool writeup, with the throughput accounting and the memory arithmetic, is its own page: the SAM 2 predictor pool.

A checklist for serving the mask generator

• One generator serves one request at a time. Enforce it with a lock or a pool, never with hope.
• Lock the whole generate call. generate owns set_image inside, once per crop. Locking your own set_image line protects nothing.
• Prefer the pool: a fixed count of generators over one model. Borrow, generate, return.
• Warm every slot with one real generate before taking traffic.
• Put a bounded queue in front. Four running and sixteen waiting held our burst at two clean rejections, each with a retry hint.
• Load test with a burst, and count both classes: the 500s and the wrong masks.
• Do not rebuild the model per request. One community demo app we read builds the model and a fresh predictor inside every request handler. It works, and it pays a full model build on every call.
• Keep the video predictor out of this shape. Video state lives per clip, in a dictionary you hold. That is a different design, and the companion page covers it.

If you would rather not own the lock

Everything above is the honest cost of running the open weights yourself: the lock or the pool, the warm-up, the burst tests, the silent class to guard against. None of it is a reason to avoid the model. SAM 2 ships under Apache 2.0 with its weights. The SAM 2 paper describes the streaming memory design behind it. It is a reason to price your own debugging time.

The hosted route here is the vision lane. It serves the sam2.1-base tier per image, and that row is on the public models list as of 28 October 2026. The larger tier has no row on the public list, and no date is attached. Each request is one image in, masks out, metered per image against a prepaid wallet. No shared generator exists on your side of that call. No lock, no race, no masks for the wrong image. Rates render on the pricing page, never typed by hand on pages like this one. A first call is three form fields and one file:

curl -X POST "https://api.ironstratum.com/v1/vision" \
  -H "Authorization: Bearer $KEY" \
  -F "model=sam2.1-base" \
  -F "file=@frame.jpg" \
  -F "response_format=json"

When a hosted call fits your loop, sign up, create a key, and run one frame through it. The neighboring pages carry the rest of the family. The per-component thread safety map, the predictor pool writeup, and the memory leak fix page. Also the automatic mask generation guide, and the segmentation and depth walkthrough for what masks feed once they exist.

Frequently asked questions

Is the SAM 2 automatic mask generator thread-safe?

No. One generator holds one private image predictor. Its state is rewritten on every generate call. In our 2026-10-05 wave of 26 requests, five crashed. All five carried TypeError, 'NoneType' object is not subscriptable, at _process_crop. Serve one request per generator at a time, with a lock or a pool.

What error does concurrent generate actually raise?

Two loud strings, and one quiet failure. The generator's own crash under load is TypeError, 'NoneType' object is not subscriptable, inside _process_crop. It fires after a concurrent set_image resets the features to None. A predict landing inside the reset window raises RuntimeError, an image must be set with set_image first. And a full set_image that finishes between your batches gives no error at all. It just gives masks of the wrong image.

Why does set_image make it unsafe?

set_image begins by resetting four fields on the predictor: _is_image_set, _features, _orig_hw, and _is_batch. It then computes new features, and fills them in. Another thread's reset between your fill and your read leaves _features as None. The next subscript on None is the TypeError. The state is plain Python, so a CPU and a graphics card fail alike.

Does a lock fix it?

Yes. The model authors' own demo serves this way, one lock around every model call. Lock the whole generate call, not set_image. generate resets and refills the predictor once per crop, inside itself. The measured cost on our 24 GB RTX 3090 card: about 2.5 seconds per generate, one at a time. The ceiling sits near 0.4 requests per second. In the same burst, zero failures of class 500.

What is the safe shape for serving?

A pool of generators over one shared model. Each request borrows an idle generator, generates alone, and returns it. Our pool of two ran the same 26-request burst at 24 clean, 2 turned away, 0 failed. Two requests at once finished in 2.228 seconds, against 1.502 for one. The pool page carries the full accounting.

Has Meta fixed this?

No. The only fix proposal in the repository, pull request 643, targets the image predictor, not the generator. It has sat at zero comments and zero reviews since 6 May 2025. The source on main still stores per-image state on the predictor object, read 28 October 2026, the same at the revision our receipts ran. The demo still wraps its calls in one lock. Plan around it, not for it.

Count both failure classes, not just the 500s. Lock the whole call, or pool the generators and warm every slot. Put a bounded queue in front, and burst test it. The per-component map lives on the companion page, and the pool receipts on the pool page. For the hosted route, the vision lane serves per image, with rates on the pricing page and a start at signup.

Last verified: 2026-10-28