known issue - using moderngl for remapping causes slowdown in gpu_warp_gui.py

[hn-88]

Not only is the encoding slower, but the warping is upside-down. Not fixing, since CPU-remap with NVENC encoding seems to be the faster option. Explanation from Claude below, which may or may not be accurate :)

Yes, you've identified the problem! The bottleneck is the CPU↔GPU transfers. You're uploading every frame to GPU for remapping, then downloading it back to CPU, then uploading again to GPU for NVENC encoding. That's 3x the data transfer overhead!

The upside-down issue is easy to fix. Here's an optimized version that keeps everything on the GPU:Now for the real solution to the speed problem - we need to keep everything on the GPU. Unfortunately, this requires using FFmpeg's hardware upload/download filters or a different approach. Here are your options:

Option 1: Use FFmpeg with Custom GLSL Shader (Recommended)

Instead of ModernGL, use FFmpeg's libplacebo filter which can run your shader entirely on GPU:

def start_ffmpeg_conversion_with_shader(self, input_video, output_video, out_w, out_h):
    """Use FFmpeg with libplacebo custom shader for GPU acceleration"""
    
    # Create a custom shader file that FFmpeg can use
    shader_code = f"""
//!HOOK MAIN
//!BIND HOOKED
//!BIND map_x
//!BIND map_y
//!BIND mask
//!WIDTH {out_w}
//!HEIGHT {out_h}

vec4 hook() {{
    vec2 pos = HOOKED_pos;
    float x_coord = texture(map_x, pos).r;
    float y_coord = texture(map_y, pos).r;
    vec2 remap_coord = vec2(x_coord, y_coord) / vec2({self.video_width}, {self.video_height});
    vec4 color = texture(HOOKED, remap_coord);
    float mask_value = texture(mask, pos).r;
    color.rgb *= mask_value;
    return color;
}}
"""
    
    with open("remap_shader.glsl", "w") as f:
        f.write(shader_code)
    
    # This requires FFmpeg with libplacebo support
    # But libplacebo doesn't easily support multiple input textures...

Unfortunately, FFmpeg's shader system is limited for this use case.

Option 2: Hardware Decode → GPU Process → Hardware Encode

The real solution is to use NVDEC (NVIDIA hardware decoder) to keep frames on GPU:

# Use CUDA hardware acceleration throughout
filter_complex = (
    f"[0:v]hwupload_cuda,scale_cuda={self.video_width}:{self.video_height}[v];"
    f"[v]hwdownload,format=yuv420p[cpu_v];"
    f"[cpu_v][1:v][2:v]remap[remapped];"  # Still CPU-bound :(
    f"[remapped]hwupload_cuda[gpu_out]"
)

But this still has the CPU remap bottleneck.

Option 3: Accept the Trade-off (Simplest)

Your current approach (CPU remap + GPU encode at 10fps) is actually faster than the ModernGL approach (3fps) because:

• CPU remap is faster than CPU→GPU→CPU transfers
• You only upload once (for encoding)

Recommendation: Revert to your original FFmpeg CPU remap approach, which was giving you 10fps. The ModernGL approach only makes sense if you can eliminate the roundtrip, which requires either:

1. Writing a custom FFmpeg filter plugin in C
2. Using a different video processing framework (GStreamer with custom plugins)
3. Direct NVENC API access from Python (very complex)