The SAM 2 predictor pool: two generators, one model, double the ceiling

A SAM 2 predictor pool is a fixed set of mask generators over one shared model. Each request borrows a generator, runs its generate call alone, and returns the generator to the pool. We have served this way since 2026-10-06 with a pool of two. Two requests at once finish in 2.228 seconds, against 1.502 seconds for one alone. The 26-request burst runs with zero failures. Weights and steady video memory stay at one model's cost. This page gives the design, the receipts, and the recipe. It also names the cases where a single lane still wins. Every source was read on 28 October 2026.

Last verified: 2026-10-28

What the pool is

Three objects make up a served SAM 2 lane, and the pool is the relationship between them.

The model can be shared because serving only reads it. A PyTorch developer stated the framework rule on the PyTorch forums in 2020. The rule reads: "all the objects in pytorch are thread safe to read. But are not to write into." The generator cannot be shared. Its generate call rewrites a private predictor on every crop. That half of the story belongs to the companion page on the mask generator and thread safety. The crash receipts live there. This pool honors both halves. The read only thing is shared. The mutable things are not.

Its lifecycle has four steps, in order, at boot.

1. Build the model once. One copy of the weights enters memory and never leaves.
2. Build the generators over it, one per slot, and place them in a queue.
3. Warm every slot with one real generate, before the service reports ready. Our warm-up receipt on 2026-10-06: weights loaded in 1.7 seconds, all slots warm in 1.5 seconds.
4. Serve. A request takes a slot from the queue, generates alone, and puts the slot back. If every slot is busy, the request waits in process, in line, not in a crash.

The official SAM 2 repository ships none of this. Its demo backend builds one predictor and wraps every call in one lock. The whole demo serves one request at a time. That is a fine shape for a public demo. It is a slow shape for a service with more than one caller.

Why two generators lift the ceiling

Start from what one generate call actually does. It walks the image in crops. For each crop it sets the crop into a predictor and runs the encoder on the graphics card. It then sweeps a grid of points through the decoder and clears the predictor. Some of that work runs on the card. Some of it is plain Python on the CPU: cropping, batching the point grid, post-processing masks. One request leaves gaps in both places.

A report in the repository's own tracker, issue 768 (opened 2026-09-02), measured the same gap from the other side. A team streaming SAM 2 per frame found their pipeline "almost entirely launch-bound, not compute-bound". The card sat idle between kernel launches. Capturing the frame step into a CUDA graph bought a 5.9x speedup at 512 pixels. Their fix removed the gaps by fusing them. A pool fills the same gaps from the serving side. While one slot runs its Python crop loop, the other slot's encoder work keeps the card busy.

The arithmetic of the ceiling is short. One at a time, the generate stage tops out at one request per generate. With a pool of two, two generates run at once, so the stage's ceiling doubles. We measured the honest version of that claim on 2026-10-06. The card was our 24 GB RTX 3090. The receipts section carries every number. What the pool does not do is add compute. It removes the line.

The receipts, measured

Two bases appear below, and they are not the same measurement. The pool receipts are the lane alone: one model, one pool, controlled requests. The whole-card numbers are four lanes under a mixed load, three of them on the card. Each row names its basis.

Read the two slot rows twice, because they carry the doubling. Both slots were busy at the same time. Each one ran at its solo speed: 1.443 and 1.487 seconds, against 1.502 alone. Neither slot slowed the other down. That is the pool doing its job. Two full speed generates fit inside the wall time of one and a half.

The burst row is the same 26-request shape that broke the shared generator on 2026-10-05. That day's census, five crashes out of 26, belongs to the thread-safety page. Under the pool, the same shape ran with zero failures of any class. The queue turned away exactly two requests, each with a retry hint. That is the queue doing its job.

The memory accounting is the quiet half of the design. Our card held 23807 of 24576 MiB with all three GPU lanes resident. The mask lane's steady share is about 2.0 GiB, pool included. The weights load once. A second generator adds its own small predictor state and short lived encoder work, nothing else. Steady video memory held flat, measured at our width of two.

The whole-card ladder is the honest footnote. Under a mixed load, the mask lane's solo latency read 4359 milliseconds, against 2981 in the one-at-a-time era. That same update had moved a second model onto the card. The pool did not make the card faster, and a busier card eats into every lane's latency. The doubled ceiling is a property of the generate stage. Our lane receipts above prove it. The mixed ladder does not. When you measure your own pool, keep the bases apart the same way.

The recipe

This is the shape we run, in generic Python. It assumes you have built the model once at startup.

import queue, threading
import numpy as np, torch
from sam2.automatic_mask_generator import SAM2AutomaticMaskGenerator

POOL_WIDTH = 2
IN_FLIGHT = 4        # requests inside the endpoint at once
QUEUE_DEPTH = 16     # requests waiting beyond those

pool: queue.Queue = queue.Queue()
for slot in range(POOL_WIDTH):
    pool.put((slot, SAM2AutomaticMaskGenerator(shared_model)))

sem = threading.BoundedSemaphore(IN_FLIGHT)
waiters = {"n": 0}
guard = threading.Lock()

def serve(image: np.ndarray):
    with guard:
        waiters["n"] += 1
    got = sem.acquire(timeout=0.1)
    while not got:
        with guard:
            queued = waiters["n"] - IN_FLIGHT
        if queued > QUEUE_DEPTH:
            with guard:
                waiters["n"] -= 1
            raise QueueFull(retry_after_s=5)   # schematic: your framework's 429, Retry-After 5
        got = sem.acquire(timeout=0.5)
    with guard:
        waiters["n"] -= 1
    try:
        slot, generator = pool.get()           # waits in line if all busy
        try:
            with torch.inference_mode():
                return generator.generate(image)
        finally:
            pool.put((slot, generator))        # the slot always goes back
    finally:
        sem.release()

Warm every slot before you announce the service as ready, or the first customer pays the compile stall:

dummy = np.zeros((256, 256, 3), dtype=np.uint8)
for _ in range(POOL_WIDTH):
    slot, generator = pool.get()
    try:
        with torch.inference_mode():
            generator.generate(dummy)
    finally:
        pool.put((slot, generator))

The trap that defeats the whole design is one word long. We shipped the first version of this service with an async endpoint. Its generate call blocked the event loop. The service served one request at a time, no matter how wide the pool was, because no second request ever reached it. Our fix was one word: declare the handler as a plain def. The FastAPI documentation states the rule: a path operation function "with normal def instead of async def... is run in an external threadpool", so the blocking call runs off the event loop. If your pool shows one busy slot under load and a pile of waiting requests, check the endpoint first. Do not blame the pool.

Numbers to copy from our build, as starting points rather than laws: a pool width of 2, four requests in flight, sixteen waiting. The overflow answer carries a five second retry hint. Warm every slot at boot. The overflow boundary is the sum. The twenty first request of a burst gets the clean 429 answer, not a crash, not a timeout.

Pool width is a measured choice

Two earned its keep on our 24 GB RTX 3090 card. The receipts also say why more was not free. The card's compute gives out before its memory does. Two reports in the repository's tracker point the same way from two sides. One user ran two separate predictors in parallel and measured no speedup on their 24 GB card. A reply on issue 652 named compute saturation as the wall. That story is told in full on the thread-safety companion. The older SAM 1 tracker holds a warning too. A user running more than two mask generators in parallel watched generate times climb. They went from two or three seconds to between 60 and 200 seconds, issue 791, community reported. Beyond a small width, you are not queueing work, you are crowding it.

So widen on evidence, in this order. Burst your own service with the shape you expect in production. Read each slot's own latency while all slots are busy. If every slot still runs at its solo speed, the card has room, and one more slot may pay. If per slot latency climbs under load, the card is already full. A wider pool only deepens the line. Memory is rarely the wall. A wider pool adds predictor state and transient encoder work, not weights. Our steady share stayed flat at the width we run.

Multiprocessing is a different road, and the tracker has its failure on record. One user split SAM 2 inference across processes and hit memory that never returned. Leftover processes held the card, issue 430. A process pool also copies the weights per process unless you engineer around it. The in-process pool shares one model by construction, which is the whole point.

When the single lane is still right

The pool is not the only correct shape, and it is not always the cheap one. Serve one at a time behind a lock when any of these hold.

• Your traffic is a trickle. A few requests a minute never feel a ceiling near 0.4 requests per second. That is the cost of the locked lane, measured on our 24 GB RTX 3090 card.
• Your callers never overlap. A cron job, a single worker, a notebook: no second request exists to serve.
• You are offline and batch shaped. A folder of images with no requesters waiting is a batch loop's job, not a pool's. The automatic mask generation guide covers that mode.
• Your card is nearly full. The pool overlaps short lived encoder work, and on a card close to its limit, overlap is the thing you cannot afford.

The lock's cost is throughput, never correctness. Both shapes are safe. Both keep one request per generator at a time. The full lock recipe, with its measured ceiling, is on the thread-safety page. Moving from a lock to a pool later is a contained change. The lock comes out. The queue goes in. Nothing about the admission law around the endpoint moves.

Other roads to throughput

The pool is the serving side. Four model side roads change the math. Three of them stack with a pool. One closes the gap instead of filling it.

• Compile and half precision. The official segment-anything-fast repository from the PyTorch team, for the SAM 1 generation, ships a drop in replacement that sets eval mode, switches to bfloat16, and applies torch.compile with max autotune. Our lane compiles too, and the pool sits on top of that, not instead of it.
• Encode once, decode many. Pull request 117, open since August 2024, would let a serving endpoint cache the image embedding and re-prompt against it for under a second per mask call. It is not merged, so today that pattern means carrying the patch yourself.
• CUDA graphs. The same issue 768 that proved the launch gap proved the capture: 5.9x on the tiny tier at 512 pixels, shrinking to 1.89x at 1024 as real compute takes over. It removes the gap rather than filling it.
• A second build of the model. The transformers model card documents batched inference APIs for images and objects per image. Most services wrap the repository build instead, and it has no such batched surface, which is exactly why serving it becomes a pool question.

The larger tier, sam2.1-large, has no row on our public models list, and no date is attached. When your ceiling problem is model speed rather than request count, the tier question comes before the pool question.

The lane you do not have to build

Everything above is the honest cost of running the open weights yourself. The pool, the queue, the warm-up, the width tuning, the endpoint discipline. It is learnable, and this page is the map. It is also real maintenance, and now you own it. The vision lane serves sam2.1-base per image. It is the same tier our receipts ran on, metered per image against a prepaid wallet, and that row is on the public models list as of 28 October 2026. The count law is simple. Send N image files in one request, and the request bills N images. Rates render on the pricing page, never typed by hand on pages like this one. A first call is three form fields and two files:

curl -X POST "https://api.ironstratum.com/v1/vision" \
  -H "Authorization: Bearer $KEY" \
  -F "model=sam2.1-base" \
  -F "file=@frame1.jpg" \
  -F "file=@frame2.jpg" \
  -F "response_format=json"

When that trade reads well for your loop, sign up, create a key, and push two frames through. The neighboring pages carry the rest of the family. Start with the concurrency bug and the lock, then the per component safety map, the mask generation walkthrough, and the memory leak fix.

Frequently asked questions

What is a SAM 2 predictor pool?

A fixed set of mask generators over one shared model. Each incoming request borrows an idle generator, runs its generate call alone, and returns the generator. No two requests ever share a generator, so the thread safety bug never gets a chance. Several generates run at once, so the ceiling rises with the pool width.

How much throughput does a pool of two actually add?

The generate stage's ceiling doubles, from one generate at a time to two. The measured wall time: two concurrent requests finished in 2.228 seconds, against about 3.0 seconds in a line. Both slots held their solo speed while busy, at 1.443 and 1.487 seconds. On a whole card shared with other models, expect the mixed load, not the clean multiple. Measure your own card.

Does a predictor pool use twice the memory?

No. The weights load once and every generator reads the same copy. A second generator adds its own predictor state and short lived encoder work. Our measured steady share is about 2.0 GiB on a card holding 23807 of 24576 MiB across three GPU lanes, pool included.

Pool or lock: which should I serve with?

Both are correct. The lock serves one request at a time and costs about 0.4 requests per second on our 24 GB RTX 3090 card. That ceiling is measured on the thread-safety page. The pool keeps every request's safety and lifts the ceiling to the pool width. Serve with the lock when traffic is light or callers never overlap. Serve with the pool when bursts are real.

How wide should the pool be?

Start at two, then widen only on measurement. Watch each slot's own latency under a full load. Flat slots mean the card has room. Climbing slots mean compute is already saturated. Community reports back this up from both generations of the model. Parallel generators degraded past two on one H800 card. Two predictors gave no speedup on another 24 GB card.

Is multiprocessing a simpler version of the pool?

It is a different road with different costs. Each process carries its own copy of the weights unless you engineer sharing. The repository's tracker holds a report of card memory that never returned after multi-process runs. The in-process pool shares one model by construction and keeps the admission law in one place.

Is this shape documented anywhere else?

Not on the open web as of 28 October 2026. We checked two web search engines, the code host's tracker search, the repository, its demo source, the model card, and a major serverless platform's deployment guides. Nobody publishes a predictor pool for SAM 2. The receipts on this page are ours, dated 2026-10-06. The hosted per image lane is the route if you would rather not collect your own.

The pool is the boring, working answer. One model. A small queue of generators. Every slot warm. A bounded line at the door. The bug story and the lock recipe live on the thread-safety page. Per component safety maps sit on the companion page. The mask generation walkthrough has its own page. For the hosted route, the vision lane serves per image. Rates sit on the pricing page. The first step is signup.

Last verified: 2026-10-28