sdatkinson / neural-amp-modeler Goto Github PK

Hi Steven,

Would love to chat to you about this project and your iPlug2 NAM plugin :-)

Best,

Oli

From lessons from easy mode

Thanks for making the colab notebook available!

It would be really useful to have a section at the end of the notebook to run the trained model on an input .wav file and produce an output .wav for downloading. The input .wav could just be the validation .wav source, or even better an arbitrary uploaded .wav file.

Containing timestamping function and other things

There's a bug in v1_1_0.wav.

This causes delay calibration to be incorrect due to hard-coded spike locations, and it's not clear where the actual "spikes" should be.

Will remove support for input v1_1_0.wav.

Implement and report the "Error-signal ratio" (ESR) metric of Wright et al., 2020 and report it as the validation metric

Try weight pruning: Train a bigger net, then prune back to the desired size. Can this get better fits?

DC offset seems to contribute a lot to the loss. Incorporate term from Eq. (19) of https://www.mdpi.com/2076-3417/10/3/766/htm

This parameter can be obtained by looking at the .receptive_field property of the model being trained.

For desktop installs, a GUI version of the trainer would be nice.

According to PyTorch as of writing this issue, the way to install PyTorch with GPU support via conda is

conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia.

This is different from the current environment (apart from the unneeded torchvision torchaudio packages).

There should be two environments--one for CPU-only computers and one for those with a GPU.

Forgot it

Implement amp & cab modeling.

Consider using a causal Wiener filter to estimate the cab quickly.

Enable one-command pip install

When switching between different presets made in Cubase12 the Plugin crashes the entire DAW.

Steps to reproduce:

1. Add NAM to a track, load a model and save that as a VST3 preset
2. Repeat step 1) multiple times, so you have some different preset to test
3. Load those presets one after another, or use the Arrows in Cubase
4. At some point the screen freezes and the application crashes

PS: I found, that if I save presets without a model loaded, and switch between them, the crash does not happen.

Info:
WIN11 - Cubase Pro 12 - i7 8700K - 16 gb RAM

Implement the WaveNet architecture from https://arxiv.org/abs/1609.03499

Several people have reported issues trying to train models of signal chains that include a noise gate. I'm going to assume that they're trying to use "easy mode" and that any fix should go there.

Functionality implemented for parametric models' .export() isn't accessible by bin/export.py

Implement the first-order pre-emphasis filter from Eq. (11) of https://www.mdpi.com/2076-3417/10/3/766/ and incorporate as a loss term.

Would also be good to include a loss weight (to be optimized w/ HPO) so that it's not necessarily as important as getting the actual MSE minimized.

Just a small quality of life update if possible: add 2 buttons to scroll through IRs within the current IR's folder.

Thank you! \m/

NAM uses an out-of-date version of wavio (version <=0.0.4). It should be updated to use the latest version.

The ESR loss is wrong for batches.

A dictionary mapping the data indices to the datasets from which they belong would speed up things, and should not be unduly large (even 1B data--over 3 years of audio!--would be only a few gigs to store).

The standard model is great, but lighter models would work a lot of the time and could save on DSP by quite a bit. Let's get some options in for lighter models.

Would be fun/possible lighter-weight model

Currently locked it into 48kHz/24-bit just to make sure I don't make mistakes, but this shouldn't need to be the case.

Try an architecture where the outputs of each convolutional layer have a skip connection to the output. (cf Figure 1 of Wright et al., 2020).

Is it possible to compile VST plugin with loading option. Not to compile new plugins every time.

Architecture where the input signal has a direct connection into each convolutional layer.
Complementary to #7 , sort of like highway nets)...

I'd like to see the models be more accurate with low-amplitude outputs.

Experiment with weighting data points that are closer to zero more heavily than high-amplitude?

We at aiXdsp saw your youtube videos and are very very interested! We could package this as a plugin, very likely, or do whatever else you might want to with it.

But really, we want to talk to YOU, you seem like a very visionary developer.

I can be reached anytime at [email protected]

Report training progress w/ TensorBoard

A second parametric model option.

Another skip connection

Add the ability to export a model i the format used by the ORT.

E.g. from PyTorch documentation.

Should be sufficient to get a .onnx file and handle the rest elsewhere.

model.load_state_dict(torch.load(args.params))

gets

IsADirectoryError: [Errno 21] Is a directory: 'data/emissary'

Do these model checkpoints need to be updated for the pycharm version? Do you happen to have the samples?

It'd be great to have TensorBoard for watching the loss curves in the Colab :)

The current model export outputs a folder holding a config.json and a weights.npy. The advantage of this is that the weights can be efficiently stored while the configuration is human-readable.

The drawbacks are that (1) these are potentially tricky to hold onto (you need to compress the directory to a file to share it anyways) and (2) the directory picker in iPlug2 seems more cumbersome to use than the file picker.

The steps to moving to a single-file export are:

1. Define the file format and implement it in this repo
2. Implement the loader in the plugin.
3. Implement a converter so that folks can update their old models to the new format. (A distributable exectuable with a GUI would be very useful here so that users find it approachable).

Using a non-unity head_scale was observed to improve WaveNet training. Using a default of 0.02 seems to work better in most cases and should be adopted as the default for the "easy mode" notebook.

If I want to be reeeaaally accurate about the delay between input and output, it'd be nice to be able to do non-integer delays. Interpolation could be via cubic interpolation to start, and could be an option farther out.

WaveNet is supposed to have separate 1x1 convolutions for the output pathway and the skip connection in a block.

The implementation in this repo only has it for the skip connection pathway.

Including that other 1x1 should substantially improve the model.

Multiple issues have been reported that seem to trace back to poorly-formed data (not being in mono, not being the same length). Increase logging while loading data so that users can easily diagnose issues with their data.

Aka knobs on the modeled amps

On the main readme.md the link for iplug is outdated, change to this: https://github.com/sdatkinson/NeuralAmpModelerPlugin

Input & output files are good for debugging, but don't always need to be exported. Flip this to off by default.

If I reamp a training signal, but the output from my interface is louder than unity gain, then that could allow me to model how the amp reacts for inputs beyond +/- 1. This could be useful for getting a realistic response when e.g. the input is hit with a really hot boost. We'd also then want to multiply up the signal as it's being fed to the amp to reflect what the amp "really saw."

This might be incorporated as an optional parameter in the data set configuration JSON.

This would be easier for people to use if there were just a Colab notebook to click through!

Hello,
Thanks for sharing this project! I'm very eager to get training and testing this out, but I'm a bit of a newb and am running into a Tensorflow 2 compatibility issue (Session isn't available in Tensorflow 2.) I'm attempting to stumble through converting the code to the new eager format, but I have no idea what I'm doing. Have you converted this to Tensorflow 2, or would you be willing to update?

I cannot seem to get the script to recognize my GPU. I have a GTX 1660 - Running on Windows 10.

If delay=0 here, then x and y are reduced to length zero.

Workaround is to use delay=None, but surprising behavior.