GeForce GTX 1070
ubuntu 16.04
python 2.7
tensorflow 0.12.0
cuda 8.0
cudnn 5.1
I'm currently trying to train the StackGAN network.
After download the birds caption data and image data, I pre-process images using python misc/preprocess_birds.py"
.
It works and produces 38images.pickle and 304images.pickle.
But when I'm training Stage-I using python stageI/run_exp.py --cfg stageI/cfg/birds.yml --gpu 0
,some problems happened.
The console output :
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.so.8.0 locally I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.so.5 locally I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.so.8.0 locally I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.so.1 locally I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.so.8.0 locally Using config: {'CONFIG_NAME': 'stageI', 'DATASET_NAME': 'birds', 'EMBEDDING_TYPE': 'cnn-rnn', 'GAN': {'DF_DIM': 64, 'EMBEDDING_DIM': 128, 'GF_DIM': 128, 'NETWORK_TYPE': 'default'}, 'GPU_ID': 0, 'TEST': {'BATCH_SIZE': 64, 'CAPTION_PATH': '', 'HR_IMSIZE': 256, 'LR_IMSIZE': 64, 'NUM_COPY': 16, 'PRETRAINED_MODEL': ''}, 'TRAIN': {'BATCH_SIZE': 64, 'B_WRONG': True, 'COEFF': {'KL': 2.0}, 'COND_AUGMENTATION': True, 'DISCRIMINATOR_LR': 0.0002, 'FINETUNE_LR': False, 'FLAG': True, 'FT_LR_RETIO': 0.1, 'GENERATOR_LR': 0.0002, 'LR_DECAY_EPOCH': 50, 'MAX_EPOCH': 600, 'NUM_COPY': 4, 'NUM_EMBEDDING': 4, 'PRETRAINED_EPOCH': 600, 'PRETRAINED_MODEL': '', 'SNAPSHOT_INTERVAL': 2000}, 'Z_DIM': 100} images: (2933, 76, 76, 3) embeddings: (2933, 10, 1024) list_filenames: 2933 001.Black_footed_Albatross/Black_Footed_Albatross_0046_18 images: (8855, 76, 76, 3) embeddings: (8855, 10, 1024) list_filenames: 8855 002.Laysan_Albatross/Laysan_Albatross_0002_1027 train(self): <stageI.trainer.CondGANTrainer object at 0x7fc3bad2a610> W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations. I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:910] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties: name: GeForce GTX 1070 major: 6 minor: 1 memoryClockRate (GHz) 1.7845 pciBusID 0000:03:00.0 Total memory: 7.92GiB Free memory: 7.56GiB I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0: Y I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1070, pci bus id: 0000:03:00.0) success Created model with fresh parameters. epoch #0| 0%| |ETA: --:--:--
Then my computer will auto reboot when the progress bar is still 0%.
I don't know if this is a problem with computer equipment.
What should I do now? Can you help me?
Thank you!!!