How did you train the model? could you open source the training code? about online_amt HOT 3 CLOSED

Hi, thank you for having an interest in my work.
The auto-regressive LSTM model has to take the output of the previous frame y_{t-1} (not the previous batch, exactly to say) to predict the output of the current frame y_t. Each sample in a mini-batch is independent, and all sampled from different performances. The batch sampling is exactly the same as the code by Dr. Jongwook Kim (https://github.com/jongwook/onsets-and-frames). Of course, as you said, passing hidden states to the next step can be helpful to train long-term dependency that exceeds the length a single sample in a batch. However, in our model, the LSTM module doesn't need to train that long dependency. The length of the sample in a batch is about 20 secs, which is long enough for training a transcription model.

The only difference is that to train the auto-regressive model, we also gave the GT label in a one-time-frame-shifted version as an input for the training. This shifted-GT label is directly given to the last auto-regressive LSTM, so that the module predicts the current output based on the GT previous output. Since our model uses multi-state labeling, the data pre-processing and loss calculation has to be slightly modified too. For the validation, I recommend predicting output frame-by-frame so that the model's prediction on the previous frame is fed into the prediction on the current frame.

Unfortunately, the code I've programmed for the training was based on my previous colleagues' modification of Jongwook's code. So I think I don't have enough rights to open the training code. But basically, the main pipeline is directly forked from Jongwook's code.

from online_amt.

Thank you! I saw your lm_model_step func in AR_Transcriber in models.py, the prediction step passes the LSTM state pair (h, c). That is to say, the training procedure just use 20s samples to train, the samples are all shuffled and the lstm state is only transfer in single sample. When inference, we can pass the state of LSTM of N step(N frame) to the N+1 step to initial the LSTM state for more precise prediction?

from online_amt.

Yes that's exactly right. Not only passing the hidden state of the previous frame, but an auto-regressive model also has to pass the output of the previous frame, which is an argmax class of the LSTM output with a linear layer. Passing the (h,c) to the next frame will be naturally calculated in every LSTM regardless of training or inference, even though one does not clarify it in PyTorch or TensorFlow. The reason why I wrote the lm_model_step function explicitly is to calculate the output argmax class for each time frame and then feed it into the next frame, which does not occur in ordinary non-auto-regressive LSTM.

from online_amt.