Separation and Count Estimation for Audio Sources Overlapping in Time and Frequency

Ph.D. Presentation

Fabian-Robert Stöter

September 19th, 2019

How many speakers do you hear?

Example A

Example B

Can you focus on the individual instruments?

Example A

Example B

Overlap in Time and Frequency



The Big Picture

Signal Processing

  • Can we obtain $\mathbf{s}_j$ from $\mathbf{x}$?
  • Source Separation

Signal Analysis

  • Can we find $k$ from $\mathbf{x}$?
  • Source Count Estimation

Applications

Separation

  • Active Listening
  • Hearing Aids
  • Frontend for many other signal processing methods

Count Estimation

  • First step to address the separation
  • Crowd Surveillance
  • Wildlife Monitoring

Source Separation

Scenario: Instruments playing in Unison

Trumpet

Scenario: Instruments playing in Unison

Cello

Scenario: Instruments playing in Unison

Cello

Scenario: Instruments playing in Unison

Cello

Instruments playing in Unison

Trumpet + Cello

Why is it Difficult?

  • Unison has extreme overlap
  • Toy example to study modulations
  • Time-Frequency representations (e.g. STFT) are not sufficient


💡Can we utilize modulations for separation of unison mixtures?



Contributions

  • Known Modulation
  • Unknown Modulation

Known Modulation

$F_{0}$ informed separation

  • Spectral Comb Filter
  • Informed non-negative matrix factorization (NMF)

Contribution

  • Use Time Warping
  • Filter in time domain
    • less artifacts
  • required: accurate F0

Proposed: Multi Sensor Recordings for Cello (MUSERC)

Unknown Modulation

Joint modulation estimation and separation

  • Source/filter model
  • HR-NMF (high resolution NMF)
  • AM spectrogram factorization

Contributions

  • Novel Representation
  • Suitable Separation Model

Common Fate: groups based on their common motion over time.

Common Fate: groups based on their common motion over time.

Common Fate Transform (CFT)

  • CFT is computed using complex STFT
    • Easily invertible
  • Adequate for 2D-stationary texture
  • Separation using Non-Negative Tensor Factorization

Demo: Sax + Flute

Demo: Viola + Flute

Extending CFT for Music Separation

  • Common fate transform + Deep Neural Networks
  • Model from Uhlich 2015
  • Improvements compared to STFT
  • Participated in SiSEC 2016

Source Count Estimation

Strategies to Count



Research Questions

  • What are the limitations of subitizing audio sources?
  • Can we build a machine to address count estimation?
  • Would a machine be subject to the same limitations?

Study on Music

What is the number of instruments in music?

  • 12 music stimuli
  • Unison for $k=2$
  • Musicians vs. Non-Musicians
  • lab ($n=40$) vs web ($n=1168$)

🔈

🔊

Study on Speech

"What is the maximum number of concurrent speakers?"

  • 100 English stimuli (5s)
  • $k_{max}$ unknown (=blind) $n=20$
  • $k_{max}$ known (=informed) $n=20$
  • Confirms earlier study in japanese

Music and Speech: "One-Two-Three-Many"

Research Questions

  • What are the limitations of subitizing audio sources?
  • Can we build a machine to address count estimation?
  • Would a machine be subject to the same limitations?

Task Definition

  • $k$ = Maximum number of concurrent speaker in excerpt L
  • Evaluation metric: mean absolute (count) error

Data-Driven Count Estimation

  • Synthetically overlapped mixtures using LibriSpeech
  • 20.020 training items (55h) ground truth labels
    • 1820 samples for $[0 ... 10]$ speakers
    • $k=0$ from TUT Acoustic Scenes dataset

Design Decisions

Complexity

  • CRNN: Best tradeoff Performance/Complexity
  • F-CRNN: Suitable for mobile applications

CountNet: STFT → CRNN → Classification

Results on Test Set

  • Excellent voice activity detection
  • Overestimation between $2 < k < 6$
  • Error rarely larger than 2

Comparison to Baselines

Train: LibriSpeech
Test: LibriSpeech

  • MFCC7: K-Means on ${MFCC}[7]$
  • SVM Classification on ${MFCC}[0:19]$
  • CRNN: DNN LSTM+CNN on STFT

Generalization Performance CountNet

CountNet Demo

Research Questions

  • What are the limitations of subitizing audio sources?
  • Can we build a machine to address count estimation?
  • Would a machine be subject to the same limitations?

CountNet vs. Human

  • Underestimation
  • Overestimation
  • One-Two-Three-Many
  • One-...-$k_{max}$

Understanding CountNet

Convolutional Filters

Does CountNet recognize phonemes?

Understanding CountNet

Ablation Study

  • Using phonetic annotation in TIMIT dataset
  • Measure speaking rate
  • Syllables per second => Modulation of Speech

Understanding CountNet

  • Retrained model on TIMIT
  • Select k=6 items (balanced)
  • Errors only underestimation
  • Speaking rate has significant influence on the error
  • CountNet used speech modulations

Conclusion

  • Focus on overlapping part instead of non-overlapping part
  • Proposed methods for separating unison mixtures
  • Extended Common Fate for music scenario
  • Speaker count estimation using Deep Learning
  • Studies showed CountNet outperformed humans
  • Modulations play an important role audio tasks

Further Contributions

Scientific

  • F0 Estimation using Warping
  • Overview articles on music separation

Evaluation campaigns

  • Participations in SiSEC 2015, 2016
  • SiSEC Organization 2016, 2018

Reproducibility

  • Unison and Cello Dataset
  • DSD100, MUSDB18 Dataset
  • commonfate, musdb, museval
  • SiSEC Websites, webMUSHRA
  • Experiment https://denumerate.app
Logo
Separation and Count Estimation for Audio Sources
Overlapping in Time and Frequency