Model stops training with variable-size dataset about tf-keras HOT 6 OPEN

Waiting for triage.
Summary:
When the dataset has a different number of samples from epoch to epoch (the batch size are the same, the number of steps are different), the training will stop at a epoch, whose number of steps is different from the first epoch.

from tf-keras.

@sachinprasadhs, I was able to reproduce the issue on Colab using TF2.7 and tf-nightly(2.8.0-dev20211201). Please find the gist here for reference.Thanks!

from tf-keras.

You can use the solution mentioned here to avoid warning and continue training.

from tf-keras.

I know a workaround, but that is very ugly:

• measure dataset length by hands (i use bucketing, so len(dataset) is None and measuring length in my case takes around 30 min)
• dataset = dataset.repeat(2).take(measured_len)

This is a bad solution. In my opinion model should continue training until reaches num_epochs even if some epoch has less batches then first one.
Displaying the number of steps remaining within an epoch is not as important as the completion of all epochs.

from tf-keras.

Thanks for reporting the issue - one solution is to use a steps_per_epoch that's large enough for the number of data in all epochs, and have the termination of an epoch rely on exhaustion of data (OutOfRangeError). Can you check if this works?

from tf-keras.

Got same issue when implementing word2vec model. Dataset size changes from epoch to epoch due to:

• randomness in skipgram/cbow context size
• randomness in downsampling with threshold

Single estimation number of batches takes around 4 hours (very large dataset).
And this size can changes with +- 20% from epoch to epoch.

So setting steps_per_epoch is not a good option.
It would be great if keras.Model will always look at OutOfRangeError itself.

from tf-keras.