Neural Texture Compression (NTC) is an algorithm designed to compress all PBR textures used for a single material together. It works best when the texture channels are correlated with each other, for example, detail in the albedo texture corresponds to detail in the normal texture. Up to 16 texture channels can be compressed into one NTC texture set. Typical PBR materials have 9-10 channels: 3x albedo, 3x normal, metalness, roughness, ambient occlusion, opacity.
During compression, the original texture data is transformed into a combination of weights for a small neural network (decoder) and a tensor of latents or features that are sampled and passed through the decoder to reconstruct the texture colors, as is illustrated below. The sampling and decoding processes are fast enough to use them directly in the shaders that normally sample the material textures, such as base pass pixel shaders or ray tracing hit shaders. However, the decoder produces unfiltered data for only one texel, and in order to get filtered textures, we suggest using NTC in combination with Stochastic Texture Filtering (STF). For renderers targeting lower-end hardware, we suggest implementing the "Inference on Load" mode where NTC textures are decompressed when the game or map is loaded, and transcoded to one of the block-compressed formats (BCn) at the same time. There is also an advanced "Inference on Feedback" mode that uses Sampler Feedback to find the set of texture tiles needed to render the current view and then decompresses only those tiles, storing them in a sparse tiled texture as BCn.
For more background information, please refer to the Random-Access Neural Compression of Material Textures page on the NVIDIA Research website.
NTC can be thought of as an adjustable quality/constant bitrate lossy compression scheme. This means that it will attempt to reconstruct the input images with minimal error while using a fixed amount of data specified as a compression time target. However, unlike block compression schemes (which have a fixed data rate for a given format) the per-texel memory footprint of a NTC-compressed texture bundle will vary based in the specified Latent Shape (which is the composite of the number of high- and low-resolution latent channels, the bit-depth of those high- and low-resolution channels, and the scale factor between them). Each latent shape corresponds to a given per-texel bitrate, and thus for a desired compression level a compatible latent shape can be selected which corresponds to desired bitrate as closely as possible. Furthermore, although NTC compression requires specifying a latent shape (and thus bitrate) it is possible to approximate a constant quality/variable bitrate approach by performing pre-analysis of the bundle to determine what formats are required to achieve a target quality level. See Adaptive Compression for more.
To demonstrate how NTC compares to other methods, consider a material defined by the following bundle of material textures:
- Albedo Color (RGB)
- Normal (XY)
- Roughness
- Metalness
- Ambient Occlusion Factor
Example material from MetalPlates013 from AmbientCG
Assuming 8 bits per channel and optimal channel packing, this corresponds to a bitrate of 64 bits/texel. In a contemporary pipeline this might be block-compressed into one BC7 texture for Albedo (8 bits/texel), one BC5 texture for normals (8 bits/texel), and a third BC7 texture for Roughness, Metalness, and Ambient Occlusion packed as separate channels (for another 8 bits/texel). For NTC we have found that many real-world texture bundles of this format can be compressed with results comparable to BCn (a PSNR of 40 to 50 dB) with a latent shape requiring about 5 bits/texel.
If we assume a 2k-by-2k texture resolution (and ignore the mip chains) we can compute the texture footprint of the bundle at various points in the data pipeline:
| Bundle Compression | Disk Size | PCI-E Traffic | VRAM Size |
|---|---|---|---|
| Raw Image | 32.00 MB | 32.00 MB | 32.00 MB |
| BCn Compressed | 12.00 MB | 12.00 MB | 12.00 MB |
| NTC-on-Load* | 2.50 MB | 2.50 MB | 12.00 MB |
| NTC-on-Sample | 2.50 MB | 2.50 MB | 2.50 MB |
*: Assumes transcoding to equivalent BCn formats at decompression time.
See the Settings and Quality Guide to learn more about various NTC settings and how they affect compression ratios and image quality.
Decompressing texels with NTC requires reading the latent data corresponding to a given texture coordinate and then performing an inference operation by running it through a small Multi-Layer Perceptron (MLP) network whose weights are determined during compression and stored as part of the compressed bundle. While this operation is modest relative to the massive networks employed by many other deep learning applications it still carries a significant computational cost relative to the average pixel shader commonly seen in 3D rendering applications. Fortunately, NTC is able to benefit from new Cooperative Vector extensions for Vulkan and Direct3D 12 which allow pixel shaders to leverage the same hardware acceleration used in large network inference. On Ada- and Blackwell-class GPUs this provides a 2-4x improvement in inference throughput over competing optimal implementations that do not utilize these new extensions.
In order to provide robust backwards compatibility, fallback implementations of the inference code using the DP4a instructions or regular integer math have also been provided. This will allow for the decompression code to be executed reliably on any platform that supports at least Direct3D 12 Shader Model 6; however, there will be substantial performance improvements on newer GPUs. See System Requirements for more details.
NTC texture decompression for DX12 applications, both on-load and on-sample, relies on a preview version of the Microsoft DirectX 12 Agility SDK, specifically,
1.721.0-preview. In order for the LinAlg (formerly Cooperative Vector) extensions to work, the application must enable theD3D12ExperimentalShaderModelsfeature, which requires that Windows is configured to be in the Developer Mode.A pre-release NVIDIA GPU driver version 620.12 or later is required for Shader Model 6.10 (LinAlg) functionality.
All non-CoopVec versions of DX12 decompression, as well as all Vulkan versions including CoopVec, are OK to use for shipping.
The DX12 LinAlg support is for testing purposes only. DO NOT SHIP ANY PRODUCTS USING IT.
The SDK exposes hardware-accelerated inference generically as "Cooperative Vector," but the concrete interface it compiles and runs depends on the graphics API selected at run time (--dx12 or --vk):
| Backend | Cooperative Vector interface |
|---|---|
Direct3D 12 (--dx12) |
Microsoft LinAlg (Shader Model 6.10, Agility SDK preview) |
Vulkan (--vk) |
VK_NV_cooperative_vector |
Either, with --no-coopVec |
DP4a / integer-math fallback |
On Direct3D 12 the Cooperative Vector shaders are compiled exclusively against the LinAlg (SM 6.10) interface; there is no separate legacy Cooperative Vector path for DX12. If the LinAlg preview is unavailable - for example, Windows is not in Developer Mode, or the driver is not a preview build - run with --no-coopVec to use the DP4a fallback, which works on any Shader Model 6 GPU. The older VK_NV_cooperative_vector interface remains available through the Vulkan backend (--vk).
See the Build Guide for instructions on compiling the SDK.
To experiment with how Neural Texture Compression performs with different latent shapes and texture bundles, follow the NTC Explorer Guide. A few sample material sets have been provided in this package under the ./assets/materials folder.
To see how NTC works on a sample 3D scene follow the instructions in the NTC Renderer Guide for pre-processing a GLTF scene. A sample scene has been provided under the ./assets/models folder.
| Component | Description | Source code |
|---|---|---|
| LibNTC | Library that implements neural texture compression, decompression, and BCn encoding | See Component |
ntc-cli |
Command-line tool for texture compression and decompression | tools/cli |
| NTC Explorer | App for interactive experimentation with neural texture compression and a viewer for NTC files | tools/explorer |
| NTC Renderer | Sample app that demonstrates how to render a GLTF model using NTC materials | samples/renderer |
| BCTest | Test app for evaluating the performance and quality of BCn encoders | support/tests/bctest |
ntc.py |
Python module for developing automation scripts that process materials using ntc-cli |
See Component |
test.py |
Script for basic functional testing of texture compression and decompression | See Component |
| Materials | Example materials for the CLI tool and Explorer | |
| FlightHelmet model | Example model for the Renderer sample |
Operating System:
- Windows 10/11 x64
- Windows 11 ARM64
- Linux x64
Graphics APIs:
- DirectX 12 - with preview Agility SDK for LinAlg support
- Vulkan 1.3
GPU for NTC decompression on load and transcoding to BCn:
- Minimum: Anything compatible with Shader Model 6 [*]
- Recommended: NVIDIA Turing (RTX 2000 series) and newer.
GPU for NTC inference on sample:
- Minimum: Anything compatible with Shader Model 6 (will be functional but very slow) [*]
- Recommended: NVIDIA Ada (RTX 4000 series) and newer.
GPU for NTC compression:
- Minimum: NVIDIA Turing (RTX 2000 series).
- Recommended: NVIDIA Ada (RTX 4000 series) and newer.
[*] The oldest GPUs that the NTC SDK functionality has been validated on are NVIDIA GTX 1000 series, AMD Radeon RX 6000 series, Intel Arc A series.
For Cooperative Vector support on NVIDIA GPUs, please use the NVIDIA Graphics Driver preview version 620.12 or newer for DX12 (Shader Model 6.10 / LinAlg), or at least version 570 for Vulkan. On Windows ARM64 the required driver comes with the ARM64 CUDA Toolkit (see the Windows ARM64 section below). The preview driver can be downloaded using the following link (requires an NVIDIA Developer Program account):
- GeForce GPUs: https://developer.nvidia.com/downloads/assets/secure/geforce-drivers/620.12_gameready_win11_win10-dch_64bit_international.exe
For a list of software components needed to build the SDK, please refer to the Build Guide.
- GDeflate decompression with
VK_NV_memory_decompressionwith the NVIDIA 590.26 Developer Preview driver crashes the GPU unless Nsight Aftermath is enabled. - Running NTC SDK apps in D3D12 mode on non-Developer Preview NVIDIA drivers may result in messages like
ERROR: D3D12CreateDevice failed, error code = 0x80070057. To work around that, add--no-coopVec. - DX12 LinAlg inference is incompatible with the current version of AMD developer preview drivers (26.10.07.02).
- Inference on Feedback mode in the Rendering sample is broken on AMD GPUs.
NTC SDK supports Windows x64, Windows ARM64, and Linux x64 targets.
Building the NTC SDK on Windows requires the following components:
- Visual Studio 2022 (at least the build tools)
- Windows SDK (tested with 10.0.26100.0)
- CMake (tested with v3.31)
- CUDA SDK (tested with v13.3)
Follow the usual way of building CMake projects on Windows:
- Clone the project recursively:
git clone --recursive https://github.com/NVIDIA-RTX/RTXNTC.git
- Building using the "x64 Native Tools Command Prompt for VS 2022":
cd RTXNTC mkdir build cd build cmake .. cmake --build .
- Building using CMake GUI:
- Set "Where is the source code" to the
RTXNTCfolder - Set "Where to build the binaries" to
RTXNTC/build - Configure a solution using
Visual Studio 2022tools for thex64platform. - Generate and open the solution in Visual Studio.
- Build.
- Set "Where is the source code" to the
Visual Studio Code with CMake Tools extension and Ninja build system works fine, too.
Windows ARM64 (for example, the NVIDIA RTX Spark) is supported as well. The requirements are the same as for Windows x64, with the following differences:
- The Visual Studio 2022 installation must include the ARM64 C++ build tools.
- A CUDA Toolkit build for Windows ARM64 is required. At the time of writing this is only available as a developer preview: download CUDA 13.4 for Windows ARM64 from the CUDA 13.4 download archive. It also requires an NVIDIA R616 or newer driver.
- Both the Visual Studio generator and Ninja are supported. To generate and build a Visual Studio solution (
build/ntc.sln), from a regular command prompt:Ninja works as well, for example from the "ARM64 Native Tools Command Prompt for VS 2022":cd RTXNTC cmake -S . -B build -G "Visual Studio 17 2022" -A ARM64 cmake --build build --config Release
cmake -S . -B build-ninja -G Ninja cmake --build build-ninja
DirectX 12 (including Cooperative Vector), Vulkan, and CUDA-based compression all work on Windows ARM64. DLSS and OptiX have no Windows ARM64 support and are disabled automatically.
Building the NTC SDK on Linux requires the following components:
- C++ compiler (tested with GCC 12.2 and Clang 16.0)
- CMake (tested with v3.31)
- CUDA SDK (tested with v12.4)
- Some development packages, approximately:
sudo apt-get update sudo apt-get install build-essential cmake libx11-dev libxrandr-dev libxinerama-dev libxcursor-dev libxi-dev
Follow the usual way of building CMake projects on Linux:
- Clone the project recursively:
git clone --recursive https://github.com/NVIDIA-RTX/RTXNTC.git
- Create the build folder, configure and build:
mkdir build && cd build cmake .. make -j
The SDK provides several tools for incorporating NTC into a content pipeline and engine runtime.
The LibNTC library provides all functionality necessary to compress, serialize/deserialize, and decompress texture sets using whatever hardware acceleration is available. It also provides implementation of a GPU-based BCn encoder to allow for transcoding decompressed textures into block-compressed format at load time, as well as a shader library demonstrating how to decompress bundle texels directly in pixel shaders at sample time.
Additionally, the ntc-cli tool can be included as part of any script-based pipelines if that is preferable. This tool is based on LibNTC as well and can perform compression and decompression tasks as desired. For more information on syntax please consult the NTC Command-Line Tool Documentation. Texture bundle information can be specified directly from the command line; however, the ntc-cli tool can also be given a Bundle Manifest File which encodes image file path, semantic and formatting information, and desired transcoding formats into one conveient serialized object. Your content pipeline can either track this information using its own database or used the provided manifest format as desired.
Further details about specific usages of LibNTC can be found divided by topic in the following guides, which walk through a proposed pipeline for a typical NTC application:
- Library Initialization
- Compression
- Decompression
- On Load
- On Sample
- On Feedback
- See the Inference on Feedback section in the Renderer's Readme file.
The NTC Explorer and NTC Renderer samples demonstrate using LibNTC for compression and decompression and can be used as a further reference for what an integrated content workflow might look like.
Please use GitHub issues or email rtxntc-sdk-support@nvidia.com for developer support.