OOM Resource Exhausted after one epoch about cartoongan-tensorflow HOT 9 CLOSED

do u have found something about this?

from cartoongan-tensorflow.

Yes. I commented this block of code in file train.py, starting from line 624 and the training proceeds without OOM error.
Of course I don't realize the consequences on the elimination of the validation fake batch on the entire training.

with summary_writer.as_default(): if not self.disable_sampling: val_fake_batch = tf.cast( (generator(val_real_batch, training=False) + 1) * 127.5, tf.uint8) img = np.expand_dims(self._save_generated_images( val_fake_batch, image_name=("gan_val_generated_images_at_epoch_" f"{epoch_idx}_step_{step}.png")), 0, ) tf.summary.image('gan_val_generated_images', img, step=epoch)

from cartoongan-tensorflow.

ok, thx for ur anwser, i think may the summary_writer.as_default() is the cause, i'll try this solution.

from cartoongan-tensorflow.

Why do you think summary_writer.as_default() is the cause?

from cartoongan-tensorflow.

@giuliacern, removing the LOCs you mentioned didn't help me. Instead I reduced the number of batch_size, and it did the trick!

from cartoongan-tensorflow.

Where were you running the experiments?

from cartoongan-tensorflow.

Windows 7
Spyder 3.3.6
Python 3.7.4

from cartoongan-tensorflow.

@giuliacern, removing the LOCs you mentioned didn't help me. Instead I reduced the number of batch_size, and it did the trick!

But would reducing batch_size downgrade the model performance?

from cartoongan-tensorflow.

We suggest using gpu with more than 11GB memory. If you encounter OOM issues, try reducing batch_size, input_size, or d_base_filters and base_filters in train.py.

We don't have unlimited computing power so apparently we cannot try all possible hyper-parameter combinations; therefore we cannot guarantee the model performance if you change any of the default hyper-parameters. It's very likely that you get worse model performance when you are forced to change any of the hyper-parameters due to lack of memory, and we are sorry that we cannot do much with this issue.

from cartoongan-tensorflow.