Depth Anything video/OOM errors: match the string, then pull the right lever

Depth Anything does not leak on video. It runs out of memory because the bill is frames times pixels times tier times dtype, and video multiplies three of those four at once. A single frame at 518 pixels fits nearly any card. Thirty-two of them in one window, on the large tier, in fp32, asks for 26.7 GB on an A100. This page quotes the exact error strings from the publishers' own trackers, names the memory law for each of the three Depth Anything lanes, and orders the four fixes by cost. Every source was read on 28 October 2026.

Last verified: 2026-10-28

The strings, exactly as they print

Three CUDA strings cover most of the reports, each copied from a named publisher issue read today. The first came from a 16 GB card running the large video model at its default size.

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 11.38 GiB. GPU

That is Video Depth Anything issue 37, opened 5 March 2025. The second looks the same and means something different: a 24 GB card running the small image model, one frame at a time.

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.53 GiB. GPU

That is Depth Anything V2 issue 68, opened 8 July 2024. The third is the loud one: roughly 700 photos through the streaming pipeline of Depth Anything 3 on a 32 GB V100.

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 669.28 GiB. GPU 0 has a total capacity of 31.73 GiB of which 17.47 GiB is free. Including non-PyTorch memory, this process has 14.26 GiB memory in use. Of the allocated memory 13.29 GiB is allocated by PyTorch, and 617.81 MiB is reserved by PyTorch but unallocated.

That is Depth Anything 3 issue 204, opened 13 January 2026. Match your screen against this table, then jump to the section it names.

One habit pays off everywhere: read what the line says the process holds and what it tried to add. The V100 held 14.26 GiB and asked for 669.28 more. That gap means the shape of the work exploded, not the card. The PyTorch memory notes explain every field, and the OpenVLA-OFT out of memory guide on this site owns the full read-the-line method.

Why video hits the wall when single images do not

There are three Depth Anything lanes for video, and each has a different memory law. Knowing your lane tells you which lever works.

The first lane is the oldest. You run Depth Anything V2 over frames in a loop, the way the repository's own run_video.py does: read one frame, run infer_image, write, move on. Video length does not touch VRAM here. The bill is one frame at the size you feed it. The V2 repository ships the loop with a default input size of 518.

Second is the video model. Video Depth Anything, CVPR 2025, does not feed frames one by one. Its source sets INFER_LEN = 32 and OVERLAP = 10, marked "infer settings, do not change". The model eats 32 frames per forward pass. VRAM tracks that one window: pixel size, tier, dtype. A two minute clip and a two hour clip cost the same VRAM, because the window slides. Length is not your lever here. The catch sits in system memory: the reader loads every frame into RAM up front.

Third is the newest. Depth Anything 3 treats frames as views of one scene, so frame count drives VRAM directly. The streaming variant caps that cost by processing views in chunks. Its published table, measured on an A100, is the honest arithmetic for this lane.

Wide frames are 504 by 154, tall are 504 by 378, and both tables come from the DA3-Streaming README. The same chunk nearly doubles its peak when frames grow taller. That is the resolution law in one table.

The video model publishes its own floor, from the same README, measured on one A100 with input of shape 1 by 32 by 518 by 518. That shape is exactly one inference window.

Read the large row twice. At the default 518 size, in fp32, the large tier wants 26.7 GB before your clip matters. A 24 GB card dies here with no video attached. The model also shrinks very wide input on its own, per an in-source comment: "we recommend to process video with ratio smaller than 16:9 due to memory limitation".

Fix one: tier down

The first lever is the model name. Both families ship three video-relevant tiers, and the tier sets the floor.

• Video Depth Anything: Small at 28.4M params, Base at 113.1M, Large at 381.8M.
• Depth Anything V2: Small at 24.8M, Base at 97.5M, Large at 335.3M.

The published window table above says what the drop buys. Small needs 7.3 GB for a window that needs 26.7 GB on Large, in fp32. That is a 3.7 times cut from one flag.

python3 run.py --input_video clip.mp4 --output_dir ./outputs --encoder vits

The --encoder flag picks the tier on both repositories. Swap vitl for vits and the same command fits an 8 GB card. The trade is edge detail and some accuracy; the benchmark tables that quantify it live on the Depth Anything 3 vs 2 guide, which owns that comparison. For video, Small is the tier for constrained cards.

Fix two: chunk the frames

If the tier is fixed, cut the frames per pass. On the DA3 streaming lane that is one config file the publisher's own numbers price.

The defaults in da3_streaming/configs/base_config.yaml are chunk_size: 120, overlap: 60, and loop_chunk_size: 20. Cut the chunk and the peak falls, row by row in the table above. Cutting from 120 to 30 saves 4.4 GB on wide frames and 9.6 GB on tall ones. The same README prints the cost: on KITTI, chunk 30 degrades to 19.39 ATE against 10.42 at chunk 120, and the sweet spot is chunk 90 at 9.38. Carry both numbers. A chunk is a quality decision, not just a memory dial.

The chunk boundary is where scenes stitch. A maintainer's reply on issue 99 gives the official recipe: "use chunk inference and then perform alignment chunk by chunk", solving a transformation from the pixel correspondences of overlapping frames. The default overlap of half a chunk already does this.

One reader of issue 204 made the same trade by hand: half the photo set, frames at 328 by 584, chunk 10 with overlap 5. The run completed on a 32 GB card. The streaming README also shows the frame-count lever in one line, its own suggested extraction command:

ffmpeg -i your_video.mp4 -vf "fps=5,scale=640:-1" ./extract_images/frame_%06d.png

Five frames per second, 640 pixels wide. That one command is the frame-count lever and the resolution lever together, from the publisher's own docs.

Fix three: resize the input

The input size is the lever with the best receipts, because a maintainer named it in public.

On the video model, the default --input_size is 518. A maintainer reply on issue 37 tells the 16 GB reader what to do: "you can try to reduce the --input_size to 378 or smaller, but there may be a loss of precision". The same reader then found their own ceiling: the large tier ran up to 514 on their card, and above that, "out of memory". 514 sits 4 pixels under the 518 default. A 16 GB card is that close to working: --encoder vitl --input_size 378 is the working command.

Two more size knobs sit beside it. --max_res, default 1280, caps the decoded video before inference. A maintainer on issue 5 explains it: input is resized so the longest side is 1280, "for saving memory for big and long video". Lower it and the reader holds less in RAM too. On DA3 the knob is process_res, default 504, method upper_bound_resize; the API doc's own example turns 1200 by 1600 into 378 by 504.

The V2 lane has one trap. The metric depth pointcloud script passes your image height straight into the model, at line 93 of depth_to_pointcloud.py: pred = depth_anything.infer_image(image, height). A tall frame becomes the network input. That is the 16.53 GiB string in issue 68, and the reply there names it: lower the input size, because that path follows your image height. The main script defaults to 518, and its README carries the temptation: raise the size "for even more fine-grained results". Every step up multiplies VRAM.

Fix four: split the batch

The deepest lever ships in the product. The video model's depth head used to process all 32 frames of the window as one batch. Pull 24, merged 26 February 2025 and present in main today, splits that batch into micro batches of four inside the head.

The pull's own claim: "On my machine, this change reduces Small model VRAM usage to 2.4 GB. The output video is exactly the same as before this change." No accuracy trade, because the split is the same math in smaller pieces. The constant micro_batch_size=4 lives in dpt_temporal.py today; a checkout older than February 2025 should update. Treat published peaks as ceilings and measure your own.

The dtype lever that is not optional

Every published number above moves again with dtype. The video model runs fp16 by default. The --fp32 flag opts you back up, and the README's own latency table shows what that costs: the large tier drops from 67 ms per window in fp32 to 14 ms in fp16, and VRAM falls from 26.7 GB to 23.6 GB. If you are passing --fp32 without a reason, stop.

The 669.28 GiB string has a dtype root cause. DA3 and its streaming pipeline default to BF16. The V100 does not support BF16 natively. A publisher reply on issue 204 states the result: on such cards the run "may lead to higher memory consumption and potential OOM" in fp32. The fp32 fallback multiplied one large chunk into an impossible request. Check what your card supports before blaming the model. Ampere's tensor cores added BF16, per NVIDIA's Ampere brief, and Hopper keeps it, so on Ampere and newer cards the default is correct.

When the error is not CUDA at all

Two failures print nothing from PyTorch.

The first is the ffmpeg lane. A Windows reader hit this pair before any model loaded, in issue 64:

[ERROR:0@0.018] global cap_ffmpeg_impl.hpp:1326 open Could not find decoder for codec_id=61
[ERROR:0@0.018] global cap_ffmpeg_impl.hpp:1375 open VIDEOIO/FFMPEG: Failed to initialize VideoCapture

When the decord reader is absent, as on that reader's Windows machine, the video reader in these pipelines is OpenCV. Your OpenCV build came without the ffmpeg decoder backend, or without the codec this file uses. Reinstall the full opencv-python package in the running environment, or re-encode the source to H.264. The issue is open with no maintainer answer, so treat the reinstall as the standard remedy, not a publisher promise. One more check from the same thread: that command passed a .png file where --input_video wanted a video.

The second is host RAM. The video model loads every frame into system memory before the first window runs, so a long clip at high resolution can die with VRAM idle and no CUDA line. Your levers are --max_len, which caps length, and --target_fps, which thins frames. The SAM 2 memory page on this site owns the full story of Python holding frames around a model. In short, the reader holds the frames, so the reader's limits are yours.

Why it runs slow, and what actually speeds it up

The same tables answer the slow question too.

On the video model, speed is tier and dtype. The published latency per window: Small at 9.1 ms in fp32 and 7.5 in fp16, Large at 67 and 14. The paper claims real time at 30 FPS for its smallest tier. On the DA3 streaming lane, the README measures 8.51 FPS on an A100 over 11,373 KITTI frames, against 2.91 for VGGT-Long and 3.15 for Pi-Long, and close to 10 FPS without the keyframe step. If you need faster than that today, the levers are resolution, tier, and fps, in that order.

Community reports add two levers the READMEs do not. A September 2026 thread on the mlx-vlm tracker, issue 2180, documents an MLX port of the same architecture. Two findings, quoted as that author's measurements: native input preprocessing measured about 2.6 times faster on that step, and fp16 weights 1.35 to 1.5 times faster end to end. On video the preprocessing cost multiplies by frame count. Those are community numbers on a port, not publisher benchmarks. Video Depth Anything's own streaming mode trades the other way: one frame at a time from cached state, with its README reporting the ScanNet d1 score drops from 0.926 to 0.836.

Prevention checklist

• Know your lane before you tune: per frame loop, 32 frame window, or view chunks. The lever list differs.
• Pick the tier for the card first. Small fits 8 GB cards; Large wants an A100 class card at defaults.
• Leave fp16 on. Pass --fp32 only when you can name why.
• Set --input_size to 518 or lower (514 is the reported 16 GB ceiling on the large tier) and keep --max_res at 1280 or lower; both bound the decode and the RAM hold.
• On DA3 streaming, start from the base config and cut chunk size in steps, watching the accuracy row next to the memory row.
• Extract at 5 fps and modest width when you control the source; the publisher's ffmpeg line is the template.
• Check your card's BF16 support before running DA3 on older datacenter cards.
• After any update, measure peak memory once with torch.cuda.max_memory_allocated().

The hosted lane for this model

Everything above is the honest cost of running these weights yourself. For the small tier, a hosted route already exists here. The platform's model data carries the da3-small row on the vision lane, and that row is on the public list as of 28 October 2026. The lane meters per image, so a video bills as its frames: send frames, the count is the frame count, stated in each response. One call carries a 25 MiB payload ceiling, so a clip goes as calls under the cap. Rates render on the pricing page, never typed by hand on pages like this one. A first call is three form fields and a frame:

curl -X POST "https://api.ironstratum.com/v1/vision" \
  -H "Authorization: Bearer $KEY" \
  -F "model=da3-small" \
  -F "file=@frame_0001.jpg" \
  -F "file=@frame_0002.jpg" \
  -F "response_format=json"

Stack more -F "file=@..." parts for more frames, keeping the payload under the cap. No window lives on your card. If that shape fits your loop, sign up, create a key, and point it at your first frame. Neighbors carry the rest: the video depth how-to guide walks the local pipeline, the DA3 small CPU tier page covers this tier without a GPU, the quickstart repeats these steps against this endpoint, and the SAM 2 memory and OpenVLA-OFT out of memory pages own the memory law for their lanes. For what depth feeds once it exists, read the segmentation and depth guide.

Frequently asked questions

Does a longer video always need more GPU memory?

No. In the per frame loop, VRAM covers one frame. In the video model, VRAM covers one sliding 32 frame window. Only the DA3 lane scales with frames, and only within a chunk. Total length scales system RAM, because the reader loads every frame up front.

What input size fits a 16 GB card?

On the video model with the large tier, the reported ceiling is 514, tested by the issue 37 reader. The maintainer's suggested value is 378 or smaller. With the small tier, defaults fit 16 GB with room to spare. On DA3, keep process_res at its 504 default and cut chunk size instead.

Why did my run try to allocate hundreds of gigabytes?

Your card almost certainly lacks BF16. DA3 and its streaming pipeline default to BF16; a V100 falls back to fp32, which multiplied one reader's chunk into a 669.28 GiB request. The publisher confirmed the cause on issue 204. Run DA3 on an Ampere or newer card, or shrink chunk and resolution until fp32 fits.

What does the ffmpeg error mean, and is it fixed?

It means OpenCV in your environment cannot decode your video file. The two lines from issue 64 name the missing decoder backend. Reinstalling the full opencv-python package, or re-encoding the source to H.264, clears it for most readers. The issue is still open with no maintainer answer as of 28 October 2026.

Does the hosted da3-small lane take whole videos?

Not in one request. The call takes frame files, meters per frame, and caps one request at 25 MiB. Send a clip as several calls of frames and keep each under the cap. The count you are billed is the frame count, stated in each response.

Match the string first. Then pull one lever at a time in cost order: tier, chunk, size, batch. Leave fp16 on unless your reason is written down. The hosted route is the vision lane, metered per frame, with rates on the pricing page and entry at signup.

Last verified: 2026-10-28