kyungyunlee / ismir2018-revisiting-svd Goto Github PK

Revisiting Singing Voice Detection : a Quantitative Review and the Future Outlook

Python 100.00%

singing-voice machine-learning deep-learning music-information-retrieval ismir

ismir2018-revisiting-svd's Introduction

Revisiting Singing Voice Detection : a quantitative review and the future outlook

This repo contains code for the paper "Revisiting Singing Voice Detection: a Quantitative Review and the Future Outlook" by Kyungyun Lee, Keunwoo Choi and Juhan Nam at the 19th International Society for Music Information Retrieval Conference (ISMIR) 2018. [pdf, blog post]

Requirements

specified in requirements.txt

Public Dataset

Jamendo with the same labeling, train/valid/test set split as described in the website.
MedleyDB
We used 61 songs that contain vocals, which can be found in medleydb_vocal_songs.txt.
Note : MedleyDB does not provide vocal annotations, so we generated labels using the provided instrument activation annotation.
Download the songs, change path, and run python medley_voice_label.py to generate labels for the 61 songs.

Dataset for stress testing (section 5)

To generate dataset, run

python vibrato_data_gen.py for vibrato test in section 5.1.
python snr_data_gen.py for SNR test in section 5.2. (Requires modification for path to MedleyDB vocal containing songs.)

Reproduction of singing voice detection models (section 3)

There are 3 reproduced models in the following folders :

lehner_randomforest [1]
schluter_cnn [2]
leglaive_lstm [3]
Note : Set paths for datasets in each config files within the model folders

Commandline arguments are :

--model_name : whatever name you set it during training, and will be saved in ./weights/ folder.
--dataset : one of {"jamendo", "vibrato", "snr"}. New dataset can be added with modification in load_data.py (might add RWC pop).

In each model folder, audio processor to preprocess data must be run before playing around with the model.

python audio_processor.py --dataset "jamendo" in CNN and RNN model with {"jamendo", "vibrato", "snr"}
python vocal_var.py --dataset "jamendo"" in randomforest model with {"jamendo", "vibrato", "snr"}
Note : This file for randomforest computes vocal variance and concatenates them with the features extracted from the matlab code provided by the authors of [1]. So, this file only provides functions for computing the vocal variance. Either you can add onto this file to compute other features or you can find the matlab code ;)

To train models, run the following in each model folder

python main.py --model_name "mynewmodel"

To run pretrained models (models are provided in ./weights/ folder), run the following in each model folder

python test.py --model_name "mynewmodel" --dataset "jamendo"

References

[1] Bernhard Lehner, Gerhard Widmer, and Reinhard Sonnleitner. "On the reduction of false positives in singing voice detection." pdf
[2] Jan Schlueter and Thomas Grill. "Exploring data augmentation for improved singing voice detection with neural networks." pdf
[3] Simon Leglaive, Romain Hennequin, and Roland Badeau. "Singing voice detection with deep recurrent neural network." pdf

TO DO (2018.06)

Upload notebook file for model analysis and audacity compatible label generation.

ismir2018-revisiting-svd's People

Contributors

Stargazers

Watchers

Forkers

locussam waithope lvaleriu guiwenming linhung0319 vichoko ashispati shradhit lilujunai

ismir2018-revisiting-svd's Issues

Help: which probability to chose in y_pred segment

I want to use this for my voice detection(RNN-VD).

But i don't know what y_pred means.

I know y_pred is (segment,RNN_INPUT_SIZE,1), but in one segment there's RNN_INPUT_SIZE probabilities and i don't know which to choose.

can you help?

Link is broken

The link to Jamendo dataset redirects to a weird Chinese webpage

SVD features processing

Hi @kyungyunlee ,
thanks for your repo and your ideas.
I am new in the field of audio processing and I would like to know how features are treated in the preprocessing phase in the specific case of CNN training for SVD, and how prediction can be associated to input features in order to apply mask ( 0 if no_voice , 1 if voice) and recreate audio of same length where it can be possible to hear voice if prediction is VOICE, and no audio if prediction is NO_VOICE.

SR = 22050
FRAME_LEN = 1024
HOP_LENGTH = 315
CNN_INPUT_SIZE = 115 # 1.6 sec
CNN_OVERLAP = 5 # Hopsize of 5 for training, 1 for inference
N_MELS = 80
CUTOFF = 8000 # fmax = 8kHz

Step: Load an audio file as a floating point time series.

y, _ = librosa.load("audio.mp3", sr=22050)

From this I get 4280832 samples.

Step: Mel Spectrogram

x = log_melgram(y, SR=22050, FRAME_LEN=1024, HOP_LENGTH=315, N_MELS=80, 27.5, 8000)

Size of x results (80, 13590).

Step: Segmentation

for i in range(0, x.shape[1] - CNN_INPUT_SIZE, 1):
    x_segment = x[:, i: i + CNN_INPUT_SIZE]
    # pick the center frame label 
    total_x.append(x_segment)

# Normalization steps
X = (total_x - mean) / std
X = np.expand_dims(X, axis=3)

After this step total_x has shape (13475, 80, 115,1)

So these are the main steps in order to get X that is fed to the network.
What is not so clear to me is the transition between Step 2 and Step 3 , so why (80, 13590) dimension becomes (13475, 80, 115) and what is its meaning.
That is the keypoint I think also to understand how return back to audio where I can apply SVD prediction of the network and build an audio with SVD and with the same original shape.
VAD prediction has shape (13475, 1).

Thank you very much

Doubt: Double Stage HPSS calculated over first P component

I've been thinking a lot about this code fragment in
https://github.com/kyungyunlee/ismir2018-revisiting-svd/blob/master/leglaive_lstm/audio_processor.py
in function process_single_audio (Compute double stage HPSS for the given audio file) in lines 24-33:

    audio_src, _ = librosa.load(audio_file, sr=SR)
    # Normalize audio signal
    audio_src = librosa.util.normalize(audio_src)
    # first HPSS
    D_harmonic, D_percussive = ono_hpss(audio_src, N_FFT1, N_HOP1)
    # second HPSS
    D2_harmonic, D2_percussive = ono_hpss(D_percussive, N_FFT2, N_HOP2)

    assert D2_harmonic.shape == D2_percussive.shape
    print(D2_harmonic.shape, D2_percussive.shape)

The D2_harmonic and D2_percussive are calculated from the D_percussive component.

Is this right? I'm currently checking the original paper and i will keep you updated if i discover something.

This seems kinda odd, since my intuition says that the harmonic component has more importance to voice activity detection.