Training gets stuck after epoch 50 is complete about few-shot-vid2vid HOT 6 CLOSED

1 GPU using batchsize 2 yes?

from few-shot-vid2vid.

Try with 1 GPU- I think you've got an error that you can't see that you're hung up on.

The model hyper parameters seem to tweak themselves as training proceeds- it reports things like adjustments to the learning rate and 'training sequence length.' I wonder if one of these automatic adjustments causes an issue- I'd have said you were running out of memory but you said you dropped the batch size already.

from few-shot-vid2vid.

Did you have a performance difference between 1x VideoCard vs 2x VideoCards?
I have no difference at all, the same performance.
2x GTX1080ti

from few-shot-vid2vid.

I haven't trained using just one GPU, I used both through out, it's only when I run the test scripts that I run on the second GPU specifically, adding the finetune argument causes the first GPU to dump the core...

Are you using the latest source?

from few-shot-vid2vid.

Yes. The same story with the recent and previous versions. The only difference while using multiple gpu, I can double --batchSize (8 against 4). But it doesn't affect performance

from few-shot-vid2vid.

This repo is now deprecated. Please refer to the new Imaginaire repo: https://github.com/NVlabs/imaginaire.

from few-shot-vid2vid.