This Library is an adaption of the DDSP to work on timbre transfer on singing voices (forked on 18.6.2020). Check out the "using_vocoder" branch for the latest changes. The corresponding audio files and some more information about the changes to the ddsp library can be found at the online supplement For training with singing voices we recommend the datasets from Wilkinson et al. or the NUS-48

The final and complete architecture can be seen in Figure 1:

The project is ready to be used in Google Colab using the "notebooks" from the same-named folder:

This notebook you downloads any given list of songs from youtube, performs audio source separation (using spleeter) to extract the voices from the songs and if desired, removes all the silences and adds the snippets together to one long audio file. The output is automatically saved in your google drive in a folder with the name defined at "folder_name"

Contains all necessary steps to set up the environment to train the autoencoder on any unsupervised singing data saved on the connected google drive and save the resulting model to be used.

When provided with the model (or multiple intermediate model save points) and the audio to be transformed this notebook saves the output of each model save point to the google drive to be compared to each other and evaluate the transformation quality.

Same as the train_no_vocoder.ipynb file but takes the "alternative audio" as input to the z-encoder to better generalize the input voices and rid them of the singer specific timbre. Can be used for other purposes as well in case backtranslation would be implemented or other effects added to the voice. The notebook actually clones the repo from the "vocoder" branch, not from the master.

Pendant to "transfer_no_vocoder.ipynb" again taking the alternative audio to use the vocoded input.

Can be used instead of "transfer_no_vocoder.ipynb" to generate one output transformation with the improved mixing technique of the filtered noise and the harmonic synthesizer.