Camera intrinsics, normalised kernels, measured benchmarks - #4
Merged
Conversation
Closes #1: `--focal` treats the depth as metric distance from a pinhole camera, unprojects each pixel and returns the normals of that 3-D surface, so a plane keeps its orientation whatever its distance. Recovered against analytic planes to within 1e-4. That exposed a second problem: the Sobel and Scharr kernels had gains of 8 and 32, which the height field path hid inside `strength` but which made the metric form wrong. All three methods are now normalised, so a ramp of one unit per pixel reads as 1 and `strength` means the same thing whichever method is picked. Also: - The exported graph is pinned to IR version 10. ONNX Runtime 1.22 and earlier refuse anything above it, which the newest `onnx` emits by default, so the models would not load on a slightly older runtime. - `tools/benchmark.py`, and the README numbers now come from it: an M4 Pro and an i5-9600K with an RTX 3090, from 2 to 37 megapixels, portrait and landscape. - Answers #2 in the README: contour rings are the source's 8-bit steps amplified by the derivative, measured against the unquantised surface. - Depth estimation runs on CUDA when onnxruntime-gpu is installed: 32 ms on the 3090 against 231 ms on the M4 Pro CPU. - The browser demo is live at depth2normal.cobanov.dev.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1. Answers #2 in the README.
--focal, for metric depth (#1)The default treats a depth map as a height field, which is what a shading normal map wants.
--focal(in pixels) treats it as metric distance from a pinhole camera instead: each pixel is unprojected toP = ((u - cx) Z / f, (v - cy) Z / f, Z)and the normal is the cross product of the two surface tangents. A plane twice as far away then comes out with the same orientation.Tested against analytic planes: the recovered normal matches the true plane normal to within 1e-4 for all three gradient methods, and a very long focal length converges on the height field form with the relief scaled
f / Z.Normalised kernels
Implementing that exposed a real bug. The Sobel and Scharr kernels had gains of 8 and 32, so they were not estimating a derivative at all. The height field path hid this inside
strength, but the metric form needs the true slope, and a tilted plane came back at 0.63 of its correct orientation instead of 1.00.All three methods are now normalised: a ramp of one unit per pixel reads as 1, whichever method is picked, and
strengthfinally means the same thing across methods. Sobel and Scharr therefore produce a gentler relief than before at the samestrength;gaussian, the default, is unchanged.IR version 10
The exported graph carried IR version 13, which is what the newest
onnxemits. ONNX Runtime 1.22 and earlier refuse anything above 10, so the models failed to load on a slightly older runtime, which was found while benchmarking on the CUDA box. Pinned, with a test.Rings (#2)
Contour rings come from the source depth map's 8-bit steps, amplified by the derivative, not from the conversion. Measured against the same surface before quantisation, and written up in the README:
--sigma 0.5--sigma 1--sigma 3--sigma 1Benchmarks and the demo
tools/benchmark.pyproduces the README tables: an M4 Pro and an i5-9600K with an RTX 3090, 2 to 37 megapixels, portrait and landscape. Depth estimation now uses CUDA whenonnxruntime-gpuis installed: 32 ms against 231 ms on the M4 Pro CPU.The browser demo is deployed at depth2normal.cobanov.dev.
The README's version history section is gone; it describes what the tool is now.
87 tests, ruff clean.