VolStream

VolStream is a comprehensive real-time volumetric video streaming solution that captures, processes, encodes and streams, as well as receives, decodes and renders V3C content generated from multiple RGB-D cameras. The system is based on Visual volumetric video-based coding (V3C, ISO/IEC 23090-5) using MPEG Immersive Video (MIV, ISO/IEC 23090-12) application. V3C provides the high-level syntax shared by different volumetric video coding standards. MIV defines the tools and application that the system leverages. Furthermore the system implements a V3C RTP payload format to allow streaming of volumetric data at extremely low latency.

Blog posts and articles

• https://www.nokia.com/blog/nokia-is-bringing-volumetric-video-services-closer-to-consumers/

• https://www.nokia.com/blog/3d-live-communication-becomes-part-of-everyday-life-with-volumetric-video/

• https://www.nokia.com/blog/how-volumetric-video-is-making-3d-rendered-video-communication-a-reality/

• https://www.tvbeurope.com/media-consumption/why-volumetric-3d-is-finally-ready-for-broadcast

Research papers

• https://dl.acm.org/doi/fullHtml/10.1145/3678726.3678754

• https://www.researchgate.net/publication/379136433_Real-time_Delivery_of_Visual_Volumetric_Video-based_Coding_Data

Presentations

• https://gstconf.ubicast.tv/videos/real-time-volumetric-video-streaming/

Repository Structure

This repository contains multiple components that together form a complete volumetric video solution:

gst-plugins-volcam/

GStreamer plugins for interfacing with Orbbec and Azure Kinect (K4A) RGB-D cameras.

Features:

• Native camera integration via GStreamer elements
• Support for Orbbec Femto Bolt (tested model)
• Legacy Azure Kinect DK support
• Direct camera data capture for streaming pipelines

Platform: Linux (Ubuntu 24.04 LTS recommended)

Main dependencies: Meson 1.10.0+, Ninja, CMake, GStreamer 1.20.3+, OrbbecSDK v2, K4A libraries, libsoundio

gst-plugins-volstream/

Core GStreamer plugins providing end-to-end volumetric video streaming functionality.

Purpose: Contains all necessary GStreamer elements to build a complete volumetric video streaming solution, including capture preprocessing, encoding coordination, network transport, and rendering preparation.

Platform: Linux (Ubuntu 24.04 LTS recommended)

Main dependencies: Meson 1.10.0+, Ninja, CMake, GStreamer 1.20.3+, cglm (3D math library), cJSON

gst-plugins-tmiv/

GStreamer plugins for TMIV (MPEG Immersive Video reference software).

Purpose: Provides GStreamer integration for TMIV to parse V3C bitstreams, enabling standard-compliant immersive video processing in streaming pipelines.

Platform: Linux (Ubuntu 24.04 LTS recommended)

Main dependencies: Meson 1.10.0+, Ninja, CMake, GStreamer 1.20.3+, TMIV reference software, fmt

client/

Unity-based client application for receiving and rendering volumetric video streams.

Purpose: Provides Unity plugins for parsing, decoding and rendering V3C bitstreams. Allows you to build a client application for Windows.

Platform: Windows

Main dependencies: GStreamer 1.0+, gst-unity-bridge, Unity, Visual Studio for plugin compilation

test-data/

Sample data for validating and testing the volumetric video streaming pipeline.

Contents:

• camera/ - Test data for GStreamer pipeline streaming using gst-plugins-volstream
  • ARGB64 files (7z compressed) - Raw camera data recorded from four ORBBEC Femto Bolt cameras
  • rendering_info.json - Camera intrinsic/extrinsic parameters and calibration data
• local/ - Test data for Unity client local playback
  • MP4 file - Encoded volumetric video containing data from all four cameras
  • volumetric_video_info.json - Camera parameters and setup configuration

Purpose: Enables testing and development without requiring physical camera hardware by simulating a four-camera volumetric capture setup.

License

Licensed under the BSD 3 Clause Clear License. See details in LICENSE.md.

Getting Started

• Fetch the necessary dependencies for the client application through this command;

git submodule update --init

• Refer to the README.md file in each component's directory for detailed build instructions, dependencies, and usage guidelines.
• After building and installing all the components, test-data can be used to validate the streaming as well as to test the rendering on the client.

Streaming and Playback

Update the filesrc location in pipelines according to your test-data location.

Sender

• Launch following GStreamer pipeline:

sudo gst-launch-1.0 filesrc location=/home/volstream/Downloads/open_src/camera_0_CL8M84101DM.argb64 ! rawvideoparse  width=1280 height=720 format=GST_VIDEO_FORMAT_Y444_16LE framerate=30/1 ! queue ! videoconvert  ! obdemux name=d0  mixer name=m  rendering-info-filepath=/home/volstream/Downloads/open_src/rendering_info.json  atlas-encap name=at d0.src_depth ! queue ! glupload ! glcolorconvert ! capsfilter caps='video/x-raw(memory:GLMemory), format=(string)ARGB64'  ! depth-process device-index=0 effect=1 rendering-info-filepath=/home/volstream/Downloads/open_src/rendering_info.json ! tee name = t0 ! queue ! m.   t0. ! queue !  glcolorconvert ! capsfilter caps='video/x-raw(memory:GLMemory),format=(string)RGBA' ! gldownload ! videoconvertscale method=6  sharpen=1 sharpness=1.5 ! capsfilter caps='video/x-raw,width=128,height=72,format=(string)RGBA' ! queue !  videocrop left=16 right=16 ! queue ! patch-gen ! queue ! at.     d0.src_color ! queue  ! videoconvert ! glupload ! m. filesrc location=/home/volstream/Downloads/open_src/camera_1_CL8K14101H0.argb64 ! rawvideoparse  width=1280 height=720 format=GST_VIDEO_FORMAT_Y444_16LE framerate=30/1 ! queue ! videoconvert  ! obdemux name=d1  d1.src_depth ! queue ! glupload ! glcolorconvert ! capsfilter caps='video/x-raw(memory:GLMemory), format=(string)ARGB64'  ! depth-process device-index=1 effect=1 rendering-info-filepath=/home/volstream/Downloads/open_src/rendering_info.json ! tee name = t1 ! queue ! m.   t1. ! queue !  glcolorconvert ! capsfilter caps='video/x-raw(memory:GLMemory),format=(string)RGBA' ! gldownload ! videoconvertscale method=6  sharpen=1 sharpness=1.5 ! capsfilter caps='video/x-raw,width=128,height=72,format=(string)RGBA' ! queue !  videocrop left=16 right=16 ! queue ! patch-gen ! queue ! at.  d1.src_color ! queue  ! videoconvert ! glupload ! m. filesrc location=/home/volstream/Downloads/open_src/camera_2_CL8M84100RC.argb64 ! rawvideoparse  width=1280 height=720 format=GST_VIDEO_FORMAT_Y444_16LE framerate=30/1 ! queue ! videoconvert  ! obdemux name=d2   d2.src_depth ! queue ! glupload ! glcolorconvert ! capsfilter caps='video/x-raw(memory:GLMemory), format=(string)ARGB64'  ! depth-process device-index=2 effect=1 rendering-info-filepath=/home/volstream/Downloads/open_src/rendering_info.json ! tee name = t2 ! queue ! m.   t2. ! queue !  glcolorconvert ! capsfilter caps='video/x-raw(memory:GLMemory),format=(string)RGBA' ! gldownload ! videoconvertscale method=6  sharpen=1 sharpness=1.5 ! capsfilter caps='video/x-raw,width=128,height=72,format=(string)RGBA' ! queue !  videocrop left=16 right=16 ! queue ! patch-gen ! queue ! at. d2.src_color ! queue  ! videoconvert ! glupload ! m.  filesrc location=/home/volstream/Downloads/open_src/camera_3_CL86541004R.argb64 ! rawvideoparse  width=1280 height=720 format=GST_VIDEO_FORMAT_Y444_16LE framerate=30/1  ! queue ! videoconvert  ! obdemux name=d3  d3.src_depth ! queue ! glupload ! glcolorconvert ! capsfilter caps='video/x-raw(memory:GLMemory), format=(string)ARGB64'  ! depth-process device-index=3 effect=1 rendering-info-filepath=/home/volstream/Downloads/open_src/rendering_info.json ! tee name = t3 ! queue ! m.   t3. ! queue !  glcolorconvert ! capsfilter caps='video/x-raw(memory:GLMemory),format=(string)RGBA' ! gldownload ! videoconvertscale method=6  sharpen=1 sharpness=1.5 ! capsfilter caps='video/x-raw,width=128,height=72,format=(string)RGBA' ! queue !  videocrop left=16 right=16 ! queue ! patch-gen ! queue ! at.  d3.src_color ! queue  ! videoconvert ! glupload ! m.   m. ! queue !  glcolorconvert ! capsfilter caps='video/x-raw(memory:GLMemory),width=(int)1920,height=(int)2880,format=(string)BGR10A2_LE' ! nvh265enc ! queue ! h265parse ! rtph265pay ! udpsink host=127.0.0.1  port=5000  at. ! queue  ! rtpatlaspay ! udpsink host=127.0.0.1 port=5100

• File source based volumetric video sender (with atlas data) pipeline graph

Receiver

• Launch following GStreamer pipeline:

sudo gst-launch-1.0 udpsrc caps="application/x-rtp,media=(string)video,clock-rate=(int)90000,encoding-name=(string)H265,payload=(int)96" port=5000 buffer-size=1524288 ! queue2 ! rtpjitterbuffer latency=500 ! rtph265depay ! h265parse  ! queue2 ! nvh265dec ! queue2 ! videoconvert ! glupload ! renderer  rendering-info-filepath=/home/volstream/Downloads/open_src/rendering_info.json ! glimagesink

• File source based volumetric video receiver pipeline graph

Receiver with atlas data

• Launch following GStreamer pipeline:

sudo gst-launch-1.0 udpsrc caps="application/x-rtp,media=(string)application,clock-rate=(int)90000,encoding-name=(string)v3c,payload=(int)96" port=5100 buffer-size=1524288 ! queue2 ! rtpjitterbuffer latency=500 ! rtpatlasdepay ! atlas-decap  ! queue2 ! patch-dec camera-count=4 ! queue2 ! videoconvert !  queue  ! autovideosink

• File source based volumetric video receiver (with atlas data) pipeline graph

Local playback

• Client application uses test-data, decodes it and finally renders it to achieve an immersive experience as depicted below.