Video Depth Anything: from video file to depth, frame by frame

Video Depth Anything turns one video file into a depth value for every frame. This page walks the whole path: what the model takes, the exact call, how to read the outputs, and how to make it fast. The model reads clips in windows of 32 frames with a 10-frame overlap, runs in fp16 by default, and its small tier is 28.4M params under an Apache 2.0 license. On the hosted side, video depth becomes frames: you extract them, send them as image parts, and the bill is the frame count times the per-image rate.

Last verified: 2026-10-28

The search results for this task are the repo itself, the paper, and the demo. As of 28 October 2026, there is no third-party walkthrough. This page is that walkthrough, built from the official repo, the CVPR 2025 paper, the small tier's model card, and the publisher's own demo defaults. Every number below was read from those sources on 28 October 2026. Two sibling pages sit close by, and this page links them rather than repeating them. The DA3 import quickstart covers the newer API surface. The version comparison covers the family history.

What the model takes: one video path, decoded and resampled

The tool takes a path to a video file, not a folder of images. The decoder is decord when it is installed, with an OpenCV fallback if it is not. In practice that means any container those libraries read, and the repo's own examples are mp4 clips. If you start from frames instead of a clip, the ffmpeg command in the hosted section below is the bridge. The code reads the whole clip into memory as one array of frames, so very long clips want a length cap or a resample first.

Before the model sees a frame, three defaults reshape your video. The code resizes the long side to 1280 pixels. The model's working size, input_size, defaults to 518. And the frame rate can be resampled. You set a target fps, and the reader keeps every nth frame to hit it. One frame is always kept as a minimum. The full knob set, from the repo's own run script:

One shape rule hides in the inference code and bites wide video. If your frame is wider than 16 to 9, the working size shrinks to fit that ratio, because of memory pressure. The code comment says so plainly. A 21 to 9 cinematic clip gets a smaller working size than a 16 to 9 clip at the same setting. Crop or letterbox first if you want the full working size on ultra-wide footage.

The local run, start to finish

Three commands carry a clip from file to depth video on a machine with a CUDA GPU. From the repo:

git clone https://github.com/DepthAnything/Video-Depth-Anything
cd Video-Depth-Anything
pip install -r requirements.txt

The requirements pin the stack: torch 2.1.1, torchvision 0.16.1, decord for reading, xformers 0.0.23, and OpenEXR for one of the output formats. Then fetch weights. The publisher's script downloads the small and large tiers:

bash get_weights.sh

The base tier is not in that script; its checkpoint is linked in the repo's model table if you want it. Now the run itself, on the repo's own example clip:

python3 run.py --input_video ./assets/example_videos/davis_rollercoaster.mp4 \
    --output_dir ./outputs --encoder vitl --save_npz

Drop --encoder vitl to vits for the small tier and the fp16 default takes care of itself. Add --metric to switch to the metric-depth checkpoints, trained on the Virtual KITTI and IRS datasets. Add --grayscale for gray output instead of the color map, --save_exr for EXR frames alongside the default video. The device line picks CUDA when it is present and falls back to CPU when it is not, so the command runs anywhere, just slowly. The measured CPU band for the small tier of the newer DA3 model, which is heavier than this one per frame, is on the CPU tier page.

Why a long video stays consistent

Depth per frame is easy. Consistent depth across thousands of frames is the actual product. The repo's inference code shows the mechanism, and knowing it saves you a debugging afternoon.

The model reads windows of 32 frames. Each window overlaps the one before by 10 frames, so the pipeline advances 22 frames at a time. The overlap is not just for safety. Some frames inside each window are marked as key frames. The next window reuses the prior window's work on those frames as its anchor. Relative depth has no absolute scale, so each window's output can drift by a scale and shift from its neighbor. The code measures that drift on the shared frames and aligns every new window to what came before. A short clip never exercises this path. The paper's demo clip is 196 seconds and 4690 frames, held together by exactly this stitching.

First, the code pads the tail of your clip by repeating the last frame, then trims back to the true length. The final frames in your output are real frames, not padding. Second, metric mode skips the align step. Metric depth already has one scale, so the code fixes scale at one and shift at zero there.

The training side explains the smoothness. The paper trains on 730K labeled video frames plus 0.62 million unlabeled images. A loss term constrains how depth changes between frames, not just how each frame looks alone. That term is why edges wobble less than they do when you run a single-image model frame by frame.

Reading the outputs: two videos, one data file

One run writes several files into your output directory. Knowing which one to load is the difference between a visualization and data.

The depth you can watch is a picture, not data. The *_vis.mp4 file colors each frame with the inferno palette, or grayscale with the flag. The color scale spans the whole clip: the darkest value in the entire video maps to the low end, the brightest to the high end. That is why one frame lifted from this video cannot be compared with a frame from another run. A *_src.mp4 file with your resampled source frames rides along, so players keep the pairs in sync.

The data you compute with is the npz. With --save_npz, the run writes one compressed npz. It holds a single array under the key depths, shaped frames by height by width, in float32:

import numpy as np

data = np.load("outputs/davis_rollercoaster_depths.npz")
depths = data["depths"]        # shape [T, H, W], one map per frame
near = depths[0].max()          # larger values are nearer

Mind the value convention. This model inherits its target from Depth Anything V2. The paper's own scoring calls the output inverse depth, in the words "inferred video inverse depth". Larger values mean nearer surfaces. If your code assumes large means far, invert on your side before anything else.

The EXR flag writes one float file per frame with depth in the Z channel, which compositing tools read natively. Metric mode adds a per-frame point cloud in ply, built with the focal length arguments, which is why that mode imports Open3D.

The speed levers, ranked by payoff

Speed threads fill the issue trackers for this model, so here is every lever with its effect, ranked by what it buys.

The encoder tier is the big one. The repo's own table, measured on a single A100 with 32-frame input at 518:

The fp16 default is not a suggestion. It is the gap between 14 and 67 ms per frame on the large tier, 4.8 times, and between 6.8 and 23.6 GB of memory. The fp32 flag exists for correctness checks, not for throughput.

Frame rate is the multiplier nobody prices in. Halve the frames and you halve everything: compute, memory churn, and on the hosted route, the bill. The publisher's own demo defaults to 15 fps, not the source rate, and to 500 frames of length. Those defaults are the publisher telling you the practical band, capped at 1000 frames and 30 fps, with resolution from 480 to 1920 pixels. The ffmpeg filter that does this outside the repo, from the official filter docs, converts "by duplicating or dropping frames as necessary."

For context, the paper's rival table lists per-frame cost. ChronoDepth: 506 ms. DepthCrafter: 910 ms. DepthAnyVideo: 159 ms. NVDS: 204 ms. The V2 single-image model: 60 ms. This model's small tier: 9.1 ms. The abstract claims real-time at 30 FPS for the smallest tier.

Streaming exists, with a warning label. The repo also ships a streaming mode, marked experimental. It needs no extra training. It caches the temporal attention state and reads one frame at a time. It trades accuracy for memory: the repo reports the ScanNet d1 score dropping from 0.926 offline to 0.836 streaming. The newer DA3 line has its own streaming variant for ultra-long sequences under 12 GB, in the DA3 streaming notes.

One community lever is worth quoting, because it holds for any port. A thread on the mlx-vlm repo, issue 2180, reports two wins from tuning a sibling port of the same backbone. First, the input resize ran on the CPU, inside an imaging library, before the frames reached the GPU. Moving that step onto the GPU made it about 2.6 times faster at large image sizes. Second, an earlier fp16 verdict had come from a tooling bug that is now fixed. The retest came in 1.35 to 1.5 times faster across the whole run. The lesson holds anywhere: profile the resize, and do not trust an old fp16 verdict.

Video depth with no GPU: the hosted frame lane

The repo route needs a CUDA GPU with 6.8 GB free at the small tier's fp16 settings, and 23.6 GB at the large tier. If you do not own that, the hosted route on this platform takes the same job in a different shape. Not one video file, but its frames, sent as image parts to the vision route. The served depth row is da3-small, the small tier of Depth Anything 3, on the public list since 6 October 2026. Its model sheet is at da3-small. The row is young; try your own clips before committing a pipeline to it.

Step one is extraction, and the publisher's own streaming docs hand you the command:

mkdir frames
ffmpeg -i clip.mp4 -vf "fps=5,scale=640:-1" frames/frame_%06d.png

That line, from the DA3-Streaming guide, drops the frame rate to 5 per second and scales the long side to 640. Raise both for quality, lower both for cost. Step two is the call, with your key set once:

curl -sS $IRONSTRATUM_BASE_URL/v1/vision \
  -H "Authorization: Bearer $IRONSTRATUM_API_KEY" \
  -F "model=da3-small" \
  -F "file=@frames/frame_000001.png" \
  -F "file=@frames/frame_000002.png" \
  -F "file=@frames/frame_000003.png" \
  -F "response_format=json"

Every file part is one image, and the count of parts is the count you pay for. The answer carries the model's depth findings per image. A usage block comes back too, with the billed image count and the wallet debit at six decimals. A request the platform refuses before serving it bills zero images. One call to the model list endpoint confirms the row is live before you start.

The cost law for video is then arithmetic. Frames in, times the per-image rate, equals the bill:

A 60-second clip sends 1800 frames at 30 fps, 900 at 15, or 300 at 5. The rate for every listed row renders on the pricing page, never typed into a page like this one. The full budget walkthrough, with the worked math for this lane, is the API pricing guide.

Know the boundary before you build on it. The hosted lane treats every frame as its own image. You get per-frame depth at DA3 quality. You do not get the window-and-stitch temporal consistency; that machinery belongs to the local pipeline. For depth-aware compression, keyframe selection, or single-frame robot perception, per-frame output is the job. For flicker-free film work, run the repo route. An account on the signup page covers the hosted route; the first call above is the whole integration.

Five mistakes that bite video first

One: fp16 that fades to black. The highest-engaged failure thread in the repo, issue 92, is titled "Can only estimate for a few seconds." The reporter sees depth that "crashes after a few seconds until it becomes completely dark." The replies converge on fp16 overflow, infinity values turning into not-a-number, and one reporter states flatly that fp32 runs clean where fp16 fails. If long clips go dark on you, force --fp32 and compare.

Two: reading the color video as data. The vis mp4 is whole-clip normalized and palette-mapped. Load the npz for values. Every downstream surprise that starts with "the depth changed when I trimmed the clip" is this mistake.

Three: expecting real time from run.py. An early issue, number 3, asked for real-time streaming and the maintainers answered that the sliding window design does not support it. The experimental streaming mode is the honest answer today, with the measured quality drop above. For live camera loops, the hosted frame lane at a low frame rate, or a smaller model, is the realistic shape.

Four: feeding 4K ultra-wide source straight in. The long side cap resizes to 1280 anyway, and the aspect rule shrinks the working size further on wide than 16 to 9 footage. You pay for pixels the pipeline throws away. Pre-scale with ffmpeg and keep the width honest.

Five: picking the large tier for a commercial product without reading the license. The small tier is Apache 2.0. The base and large tiers are CC BY-NC 4.0, which is not a commercial grant. The repo says so in its license section, with a contact address for business terms. The license map across the whole family, including version 2 and version 3, is in the version comparison.

Frequently asked questions

What is Video Depth Anything?

A video depth model from ByteDance, published January 2025 as arXiv 2501.12375 and accepted as a CVPR 2025 highlight. It builds on Depth Anything V2 and swaps in a spatial-temporal head. It reads arbitrarily long clips, where earlier video methods stopped around 10 seconds.

How do I run it on a video?

Clone the repo, install requirements, run get_weights.sh, then call run.py with your clip path, an output directory, and an encoder of vits, vitb, or vitl. The commands above are the complete path.

How fast is it?

On a single A100 with 32-frame input at 518: 7.5 ms per frame at the small tier in fp16, 14 at the large tier. The paper claims real-time at 30 FPS for the smallest model. It reports rivals at 159 to 910 ms per frame.

Can it run in real time?

Not the standard run.py path: it processes fixed windows of 32 frames, so it is built for whole clips, not live streams. The repo's experimental streaming mode processes frame by frame with a reported accuracy drop, from 0.926 to 0.836 d1 on ScanNet.

Which encoder should I pick?

Pick vits for speed, 28.4M params, 6.8 GB in fp16, and the Apache 2.0 license. Pick vitl for the best quality at 23.6 GB and a non-commercial license. Pick vitb only when you need the middle and can carry its license terms; the weights script does not fetch it.

Does it output metric depth?

With --metric, yes: separate checkpoints trained on Virtual KITTI and IRS, plus per-frame point clouds. The default output is relative inverse depth, larger values nearer, with no absolute scale.

What does a minute of video cost on the hosted lane?

The frame count times the per-image rate: 1800 frames at 30 fps, 900 at 15, 300 at 5 for a 60-second clip. The current rate renders on the pricing page, and the worked budget math is on the pricing guide.

Sources read in full on 2026-10-28. The Video Depth Anything repository: README, run.py, video_depth.py, dc_utils.py, requirements.txt, get_weights.sh. The paper, including the v4 full text. The small tier model card and the project page. The official demo Space's published settings. The DA3-Streaming guide and the ffmpeg filter documentation. For the error family this page names but does not derive, see video depth out of memory.

Last verified: 2026-10-28