Depthbake

depthbake-readme.mp4

A raw depth map does not survive displacement. Depthbake fixes that part.

At the edge of a subject — hair, foliage, wire mesh, a shoulder against a distant wall — a single pixel contains both foreground and background. No single depth value is correct there, so a depth model returns a smooth middle one. Push that middle value through displacement and you get a sheet stretched between the two layers: the rubber-band artifact that every depth-parallax implementation shows on close inspection.

That is not a failure of depth estimation. It is an alpha problem being handled as a depth problem. Better upstream models do not fix it, because they are not asked to.

Depthbake detects those discontinuities, solves an alpha inside a narrow band around them using the photo itself as a guide, and then snaps every band pixel to either the foreground or the background depth. No intermediate depth values survive, so the stretched sheet has nothing to be built from — and a renderer can find and cut it. Cutting leaves a gap, which is not filled. Instead, each package declares how far the camera can move before that gap becomes visible, and the camera clamps to it.

npx depthbake-cli bake ./photos --out ./out --boundary
npx depthbake-cli compare ./out/raw/hero ./out/hero --out ab.png   # side by side
npx depthbake-cli measure ./out/hero                               # the numbers

The result is a portable, depth-aware asset for WebGL, three.js, and custom creative coding workflows: photo.jpg (plus optional AVIF/WebP candidates), depth.png, meta.json, and opt-in mask.png / normal.png. Drive it with pointer, scroll, shaders, lighting, particles, or your own runtime.

Live demo: depthbake.pages.dev

Boundary resolution

Run depthbake bake --boundary and the package gains three things:

• Resolved depth. Discontinuities are detected, dilated into a band, matted with a photo-guided edge-preserving filter, and each band pixel is committed to one layer.
• Coverage. mask.png's B channel records how much of its assigned layer each pixel covers (1 outside the band). GLSL_SNIPPETS.boundaryCut uses it, or blend with it directly if you prefer to soften rather than cut.
• A declared camera excursion. meta.safeExcursion says how far the camera can translate and orbit before a cut gap is exposed. DepthbakeCamera.setExcursion() clamps to it by default; { clampExcursion: false } opts out, and what happens outside the declared range is not guaranteed.

Measure it rather than eyeballing it. depthbake measure reports boundary offset (distance in pixels from each depth discontinuity to the nearest photo edge, median and p95), stretch ratio at a given camera translation, hole ratio, and package size. Run it against a fixed set of deliberately hostile subjects — see bench/ — including a flat-wall negative case, because over-detection is the failure mode that looks like success everywhere else.

Non-goals

Scope discipline is what makes the guarantee above meaningful.

• No generative disocclusion. What is behind the foreground is not in the image, and inpainting it is a different project with a different package size and inference cost. The gap left by a cut is bounded and declared, not filled.
• No general layered depth image. One foreground and one background per band, no variable layer count.
• No video. Temporal consistency is a separate problem.
• No production-quality bake in the browser. The browser path is preview quality by design; see the asymmetry below.
• No in-house or fine-tuned depth model. See docs/model-selection.md.

The limited camera excursion is a feature, not an apology: the asset itself carries the range over which it is guaranteed not to break. A hand-rolled parallax implementation has nothing equivalent.

Browser and CLI are not the same

Quality costs inference time, so "runs entirely in the browser" and "production quality" cannot both be true. Rather than meeting in a mediocre middle, the two paths are deliberately different, and the browser UI says so.

	Browser (src/)	CLI (packages/cli)
Role	Preview / trying parameters	Production asset generation
Depth model	Lightweight (small)	Same small model today; DepthEstimator is the seam a metric model drops into (docs/model-selection.md)
Matting	Opt-in checkbox, same CPU filter	Opt-in (--boundary)
Output	What it looks like	The package you ship

Because of this split, the inference layer is model-agnostic: DepthEstimator in packages/core/src/estimator.ts is the only surface a model swap has to fit through.

Packages

• Browser baker (the root app in this repo): drop in a photo, estimate depth in-browser (WebGPU/WASM), preview depth-aware effects, and export a .depthbake package. Everything runs locally; nothing is uploaded to a server. Export controls let you set photo/map resolution, cap the combined map PNG size, bundle AVIF/WebP/JPEG photo variants supported by the browser, and resolve boundaries.
• depthbake-cli: runs the same inference and packing logic in Node to batch-bake production packages from a folder of images. Distributed on npm as depthbake-cli.
• depthbake-runtime: a lightweight, renderer-agnostic loader that reads the package, selects the first decodable photo candidate, exposes GPU-ready map bitmaps, and recovers world-space positions for three.js, raw WebGL, Canvas2D, or custom shaders. Distributed on npm as depthbake-runtime.
• depthbake-three: a THREE.Loader that turns a package into ready-to-use three.js textures and a matching uniform bundle, plus a cover-fit DepthbakeCamera. It doesn't ship a default effect or motion system — write your own shader against asset.uniforms. Distributed on npm as depthbake-three.

What it is for

Depthbake is meant for creative developers and studios building image-led websites: brand hero visuals, exhibition sites, photographer portfolios, travel and hotel pages, music or film promos, and other cases where a still image needs to become an interactive WebGL surface rather than a baked video.

It is not just a parallax preset. The package gives you depth as creative input, so the same still image can drive:

• pointer or scroll parallax
• depth-based relighting
• particles and splats
• fog, blur, and color grading
• typography occlusion or reveal effects
• shader transitions and displacement

Five-minute start

npx depthbake-cli bake ./photos --out ./out

import * as THREE from "three";
import { DepthbakeLoader, DepthbakeCamera } from "depthbake-three";

const asset = await new DepthbakeLoader().loadAsync("/out/hero/");
const camera = new DepthbakeCamera(asset); // cover-fit THREE.PerspectiveCamera

// asset.photo / asset.depth are ready-to-use THREE.Texture instances, and
// asset.uniforms is a bundle your own shader can consume directly.
console.log(asset.meta, asset.uniforms);

See examples/hero for the full copy-paste template (pointer follow, idle autopilot, reduced-motion), and examples/three-scene, examples/depth-splats, examples/relight for complete three.js demos — all built on depthbake-three.

Repository layout

.
├── src/                 # Browser demo app (Vite, private)
├── examples/three-scene # three.js scene that loads a package into a custom shader
├── examples/depth-splats # depth-placed point cloud / splat demo
├── examples/relight     # depth + normal driven relighting demo
├── examples/hero         # copy-paste starter template (pointer follow, idle, reduced-motion)
├── bench/               # Fixed benchmark set for measurement (images not committed)
├── public/sample/source # A sample pre-baked package
└── packages/
    ├── core/            # Shared inference / normalization / upsampling / packing logic (private, used by both the viewer and the CLI)
    ├── cli/              → published as depthbake-cli
    ├── runtime/          → published as depthbake-runtime
    └── three/            → published as depthbake-three

packages/core is an internal package shared by the viewer app and the CLI; it is not published to npm (private: true). The CLI bundles it at build time.

Setup

pnpm install

Requires Node 20+ (the CLI depends on the native binaries of sharp and onnxruntime-node).

Running the browser demo

pnpm dev

On first run the depth estimation model (~25–50MB, onnx-community/depth-anything-v2-small) is downloaded in the browser. It uses WebGPU where available and falls back to WASM otherwise.

Using the CLI

npx depthbake-cli bake ./photos --out ./out

Or install it globally with npm install -g depthbake-cli. See packages/cli/README.md for details.

Measuring

depthbake measure ./out/hero --json          # boundary offset, stretch, hole ratio, size
depthbake compare ./out/raw/hero ./out/hero --out ab.png --translate 0.02

compare renders each package the way it is actually drawn — plain displacement for a raw package, the geometry cut for a resolved one — under one camera translation, and writes them side by side. Applying the cut to a raw depth map would delete the entire smeared band, so comparing them under identical rendering would not be the honest comparison.

Neither command loads a model; both read a baked package. See bench/README.md for how to run a full set and how to read the numbers.

Package format

The package the CLI writes and the runtime reads is specified in docs/package-format.md.

Model license

The default model onnx-community/depth-anything-v2-small is Apache-2.0 and permitted for commercial use. The Base/Large variants of Depth Anything V2 are CC-BY-NC-4.0 (non-commercial only), so always check each model card's license before switching models.

Tests & type-checking

pnpm test        # Unit tests in packages/*/test and src/*/*.test.ts
pnpm typecheck   # Type-check every workspace, including the root app

License

MIT