Sound Separation: Comparative Analysis of DSP and Deep Learning Architectures

This repository contains the implementation and evaluation of a complete pipeline for music source separation. Developed for the "Selected Topics in Music and Acoustic Engineering" course at Politecnico di Milano (2025-2026), this project critically compares traditional Digital Signal Processing (DSP) methodologies against modern Deep Neural Networks (DNNs).

The core objective is to decompose professionally mixed stereo recordings into their constituent sources: vocals, drums, bass, and accompaniment (other).

📊 Dataset

The models are trained and evaluated on the MUSDB18 dataset.

• Comprises 150 full-length, professionally produced music tracks (approx. 10 hours).
• Provides both the full stereo mix and isolated stems, enabling supervised learning and objective baseline evaluation.
• Poses realistic challenges, including heavy frequency overlap, production effects (reverb/delay), and highly correlated temporal structures.

⚙️ System Architectures & Methodologies

Our comparative framework investigates four distinct structural approaches to the separation problem:

1. Unsupervised DSP (Baseline)

• HPSS (Harmonic-Percussive Source Separation): Exploits geometric duality in the STFT magnitude spectrogram to isolate vertical transients (drums) and horizontal stationary lines (harmonic components) using median filters.
• Blind NMF (Non-negative Matrix Factorization): Decomposes the spectrogram into a linear combination of basis vectors and activation matrices using multiplicative update rules, initialized via NNDSVD.

2. Supervised NMF

Addresses the multi-source ambiguity of blind NMF by splitting the process into learning and inference phases. Target-specific dictionaries are pre-trained on isolated stems to enforce strict acoustic profiles during mixture factorization.

3. Deep Learning: Custom U-Net

Custom U-Net

A convolutional encoder-decoder architecture treating the magnitude spectrogram as a 2D image.

• Skip Connections: Bypasses the bottleneck to preserve fine, high-frequency spectral details and mitigate vanishing gradients.
• Loss Function: Optimized using LogL2 loss.

4. Transfer Learning & Stem Shuffling (Open-Unmix)

Utilizes the state-of-the-art Open-Unmix (UMX) BiLSTM architecture as a foundational base.

• Stem Shuffling Augmentation: A custom PyTorch Dataset dynamically generates artificial mixtures by superimposing asynchronous stems from independent tracks. This completely eradicates temporal covariance, forcing the network to decouple sources semantically rather than relying on contextual rhythm cues.
• Fine-Tuning: Applied specifically to the UMX drums target, freezing deep feature extractors and updating only the final fully connected layers.

📈 Evaluation & Results

Models were evaluated using standard BSSEval metrics (SDR, SIR, SAR, ISR) via the museval library.

Key Findings

• The Limits of DSP: Unsupervised methods fundamentally failed on dense commercial mixes. Blind NMF yielded a negative SDR (-1.756 dB) for drums, proving its inability to resolve overlapping frequency components.
• Linear Bottlenecks: While Supervised NMF significantly improved vocal extraction (SDR 2.772 dB), its fixed-spectra assumption fails to capture non-linear dynamic shifts, acting as a hard performance ceiling.
• Deep Learning Superiority: The custom U-Net achieved robust separation, proving the validity of convolutional spatial mapping.
• Transfer Learning Success: The Stem Shuffling protocol on the UMX drums model successfully isolated percussive transients. By destroying rhythmic context during training, the model increased its Signal-to-Interference Ratio (SIR) from 11.866 dB to 12.677 dB, trading a marginal amount of mathematical distortion for a purer semantic extraction.

🚀 Critical Gaps & Future Trajectories

To surpass current architectural bottlenecks, future iterations must address these root causes:

1. Waveform-Domain Processing: STFT-based magnitude masking is heavily constrained by the phase-retrieval bottleneck (reusing the mixture's phase). Transitioning to 1D convolutions (e.g., Demucs, Wave-U-Net) is required to estimate phase implicitly.
2. Multi-Resolution STFTs: Instead of single-resolution compromises, parallel encoders must process high-temporal-resolution data for transients and high-frequency-resolution data for pitched harmonics.
3. Extreme Decorrelation: The success of Stem Shuffling proves that intelligent data engineering drives performance. Expand augmentation (pitch shifting, dynamic re-reverberation) to systematically eliminate mixing-engineer bias from the training distribution.