Could you please give more details?
What custom does the dataset look like? What is the resolution of the images, how it was created?
What's your config?
How many GPUs?
Windows|Linux?
Maybe you can create a minimal example of the dataset with ~8-16 images that causes the same problem?

''Aborted'' - is that all the error reported, or there is something more to it?

Could you please confirm, that before failing at the line you mentioned, the dataset is not empty and has the expected size?

from alae.

Hi,
The custom dataset is gray scale image of size (128,128). I created tfrecords for the samples.
My config file looks like this
DATASET:
PART_COUNT: 1
SIZE: 12000
#PATH: /data/datasets/mnist/tfrecords/mnist-r%02d.tfrecords.%03d
SAMPLES_PATH: no_path
PATH: /home/Documents/ALAE-master/data/datasets/test/tfrecords/test-r%02d.tfrecords.%03d
#PATH_TEST: /home/Documents/ALAE-master/data/datasets/test/tfrecords/test-r%02d.test_tfrecords.%03d
MAX_RESOLUTION_LEVEL: 7

MODEL:
LATENT_SPACE_SIZE: 256
LAYER_COUNT: 6
MAX_CHANNEL_COUNT: 256
START_CHANNEL_COUNT: 64
DLATENT_AVG_BETA: 0.995
MAPPING_LAYERS: 8
OUTPUT_DIR: results
CHANNELS: 1
TRAIN:
BASE_LEARNING_RATE: 0.002
EPOCHS_PER_LOD: 6
LEARNING_DECAY_RATE: 0.1
LEARNING_DECAY_STEPS: []
TRAIN_EPOCHS: 80

4 8 16 32 64 128 256 512 1024

LOD_2_BATCH_8GPU: [512, 256, 128, 64, 32, 32, 32, 32, 32]
LOD_2_BATCH_4GPU: [512, 256, 128, 64, 32, 32, 32, 32, 16]
LOD_2_BATCH_2GPU: [128, 128, 128, 64, 32, 32, 16]
LOD_2_BATCH_1GPU: [128, 128, 128, 64, 32, 16]

LEARNING_RATES: [0.0015, 0.0015, 0.0015, 0.0015, 0.0015, 0.0015, 0.002, 0.003, 0.003]

I am running this on Linux with 1 GPU
Aborted is the only error message I am getting.
Yes i confirmed the dataset length before the mentioned line.
Also the same dataset with resolution 32 works without errors when used with mnist.yaml

from alae.

Does it fail only later during training when reaches resolutions > 32 or it fails immediately? Could you share the whole log on pastebin?
You said that it works fine at resolution 32 with mnist.yaml. Do you use precisely the same files as dataset, or you created them separately? It should be the case, that first 4 tfrecord files (that correspond to 4x4, 8x8, 16x16, 32x32) should be the same. So you should be able to run mnist.yaml on the dataset created for resolution 128.
So, if it works fine with mnist.yaml with the 128 dataset, I would just change MAX_RESOLUTION_LEVEL to 7 and LAYER_COUNT to 6 and see what happens.

from alae.

I have entered the folds size wrong in my case. When I corrected it, everything works fine

from alae.

I have entered the folds size wrong in my case. When I corrected it, everything works fine

what does that mean ? I am stuck at the aborted issue, while running on celeba 128 :(
Any advise ?

from alae.