jun-tam / 3d-seismic-image-fault-segmentation Goto Github PK

C:\Users\Windground\PycharmProjects\3D-Seismic-Image-Fault-Segmentation\venv\Scripts\python.exe C:/Users/Windground/PycharmProjects/3D-Seismic-Image-Fault-Segmentation/fault_segmentation.py
2021-01-12 12:18:58.345300: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
Creating CNN Model
2021-01-12 12:19:07.671807: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-01-12 12:19:07.679657: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library nvcuda.dll
2021-01-12 12:19:07.741949: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce RTX 2060 computeCapability: 7.5
coreClock: 1.56GHz coreCount: 30 deviceMemorySize: 6.00GiB deviceMemoryBandwidth: 245.91GiB/s
2021-01-12 12:19:07.742217: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-01-12 12:19:07.783808: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-01-12 12:19:07.783958: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-01-12 12:19:07.804594: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-01-12 12:19:07.810753: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-01-12 12:19:07.862311: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-01-12 12:19:07.878932: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-01-12 12:19:07.882485: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2021-01-12 12:19:07.882848: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-01-12 12:19:07.884205: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-01-12 12:19:07.885514: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce RTX 2060 computeCapability: 7.5
coreClock: 1.56GHz coreCount: 30 deviceMemorySize: 6.00GiB deviceMemoryBandwidth: 245.91GiB/s
2021-01-12 12:19:07.885959: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-01-12 12:19:07.886166: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-01-12 12:19:07.886378: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-01-12 12:19:07.886549: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-01-12 12:19:07.886703: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-01-12 12:19:07.886854: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-01-12 12:19:07.887007: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-01-12 12:19:07.887160: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2021-01-12 12:19:07.887348: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-01-12 12:19:09.305698: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-01-12 12:19:09.305927: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2021-01-12 12:19:09.306029: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2021-01-12 12:19:09.307335: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2021-01-12 12:19:09.307555: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4720 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2060, pci bus id: 0000:01:00.0, compute capability: 7.5)
2021-01-12 12:19:09.309999: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
WARNING:tensorflow:`write_grads` will be ignored in TensorFlow 2.0 for the `TensorBoard` Callback.
WARNING:tensorflow:`batch_size` is no longer needed in the `TensorBoard` Callback and will be ignored in TensorFlow 2.0.
2021-01-12 12:19:09.523762: I tensorflow/core/profiler/lib/profiler_session.cc:136] Profiler session initializing.
2021-01-12 12:19:09.523869: I tensorflow/core/profiler/lib/profiler_session.cc:155] Profiler session started.
2021-01-12 12:19:09.524681: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1365] Profiler found 1 GPUs

Model Fitting
2021-01-12 12:19:09.526305: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cupti64_110.dll'; dlerror: cupti64_110.dll not found
2021-01-12 12:19:09.527227: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cupti.dll'; dlerror: cupti.dll not found
2021-01-12 12:19:09.527605: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1415] function cupti_interface_->Subscribe( &subscriber_, (CUpti_CallbackFunc)ApiCallback, this)failed with error CUPTI could not be loaded or symbol could not be found.
2021-01-12 12:19:09.527866: I tensorflow/core/profiler/lib/profiler_session.cc:172] Profiler session tear down.
2021-01-12 12:19:09.527992: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1496] function cupti_interface_->Finalize()failed with error CUPTI could not be loaded or symbol could not be found.
C:\Users\Windground\PycharmProjects\3D-Seismic-Image-Fault-Segmentation\venv\lib\site-packages\tensorflow\python\keras\engine\training.py:1844: UserWarning: `Model.fit_generator` is deprecated and will be removed in a future version. Please use `Model.fit`, which supports generators.
  warnings.warn('`Model.fit_generator` is deprecated and '
2021-01-12 12:19:09.648297: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
Epoch 1/25
2021-01-12 12:19:11.873786: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2021-01-12 12:19:14.829302: I tensorflow/core/platform/windows/subprocess.cc:308] SubProcess ended with return code: 0

2021-01-12 12:19:14.869995: I tensorflow/core/platform/windows/subprocess.cc:308] SubProcess ended with return code: 0

2021-01-12 12:19:15.182234: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-01-12 12:19:16.341179: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-01-12 12:19:35.055075: W tensorflow/core/common_runtime/bfc_allocator.cc:433] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.12GiB (rounded to 1207959552)requested by op model/concatenate_2/concat
Current allocation summary follows.
2021-01-12 12:19:35.056074: I tensorflow/core/common_runtime/bfc_allocator.cc:972] BFCAllocator dump for GPU_0_bfc
2021-01-12 12:19:35.056533: I tensorflow/core/common_runtime/bfc_allocator.cc:979] Bin (256): 	Total Chunks: 83, Chunks in use: 82. 20.8KiB allocated for chunks. 20.5KiB in use in bin. 5.6KiB client-requested in use in bin.
2021-01-12 12:19:35.057383: I tensorflow/core/common_runtime/bfc_allocator.cc:979] Bin (512): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-01-12 12:19:35.058183: I tensorflow/core/common_runtime/bfc_allocator.cc:979] Bin (1024): 	Total Chunks: 4, Chunks in use: 4. 6.5KiB allocated for chunks. 6.5KiB in use in bin. 6.1KiB client-requested in use in bin.
2021-01-12 12:19:35.059025: I tensorflow/core/common_runtime/bfc_allocator.cc:979] Bin (2048): 	Total Chunks: 6, Chunks in use: 6. 12.0KiB allocated for chunks. 12.0KiB in use in bin. 12.0KiB client-requested in use in bin.
2021-01-12 12:19:35.059862: I tensorflow/core/common_runtime/bfc_allocator.cc:979] Bin (4096): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-01-12 12:19:35.060128: I tensorflow/core/common_runtime/bfc_allocator.cc:979] Bin (8192): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-01-12 12:19:35.060295: I tensorflow/core/common_runtime/bfc_allocator.cc:979] Bin (16384): 	Total Chunks: 5, Chunks in use: 5. 135.0KiB allocated for chunks. 135.0KiB in use in bin. 135.0KiB client-requested in use in bin.
2021-01-12 12:19:35.060476: I tensorflow/core/common_runtime/bfc_allocator.cc:979] Bin (32768): 	Total Chunks: 3, Chunks in use: 3. 156.8KiB allocated for chunks. 156.8KiB in use in bin. 135.0KiB client-requested in use in bin.
2021-01-12 12:19:35.060656: I tensorflow/core/common_runtime/bfc_allocator.cc:979] Bin (65536): 	Total Chunks: 9, Chunks in use: 9. 891.0KiB allocated for chunks. 891.0KiB in use in bin. 864.0KiB client-requested in use in bin.
2021-01-12 12:19:35.060834: I tensorflow/core/common_runtime/bfc_allocator.cc:979] Bin (131072): 	Total Chunks: 2, Chunks in use: 2. 351.0KiB allocated for chunks. 351.0KiB in use in bin. 297.0KiB client-requested in use in bin.
2021-01-12 12:19:35.061014: I tensorflow/core/common_runtime/bfc_allocator.cc:979] Bin (262144): 	Total Chunks: 11, Chunks in use: 11. 4.18MiB allocated for chunks. 4.18MiB in use in bin. 3.90MiB client-requested in use in bin.
2021-01-12 12:19:35.061199: I tensorflow/core/common_runtime/bfc_allocator.cc:979] Bin (524288): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-01-12 12:19:35.061365: I tensorflow/core/common_runtime/bfc_allocator.cc:979] Bin (1048576): 	Total Chunks: 1, Chunks in use: 0. 1.28MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-01-12 12:19:35.061536: I tensorflow/core/common_runtime/bfc_allocator.cc:979] Bin (2097152): 	Total Chunks: 6, Chunks in use: 6. 21.14MiB allocated for chunks. 21.14MiB in use in bin. 21.14MiB client-requested in use in bin.
2021-01-12 12:19:35.061718: I tensorflow/core/common_runtime/bfc_allocator.cc:979] Bin (4194304): 	Total Chunks: 1, Chunks in use: 1. 4.62MiB allocated for chunks. 4.62MiB in use in bin. 3.38MiB client-requested in use in bin.
2021-01-12 12:19:35.061897: I tensorflow/core/common_runtime/bfc_allocator.cc:979] Bin (8388608): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-01-12 12:19:35.062071: I tensorflow/core/common_runtime/bfc_allocator.cc:979] Bin (16777216): 	Total Chunks: 11, Chunks in use: 11. 265.23MiB allocated for chunks. 265.23MiB in use in bin. 255.00MiB client-requested in use in bin.
2021-01-12 12:19:35.071648: I tensorflow/core/common_runtime/bfc_allocator.cc:979] Bin (33554432): 	Total Chunks: 4, Chunks in use: 4. 149.00MiB allocated for chunks. 149.00MiB in use in bin. 126.00MiB client-requested in use in bin.
2021-01-12 12:19:35.071909: I tensorflow/core/common_runtime/bfc_allocator.cc:979] Bin (67108864): 	Total Chunks: 2, Chunks in use: 2. 192.00MiB allocated for chunks. 192.00MiB in use in bin. 192.00MiB client-requested in use in bin.
2021-01-12 12:19:35.072307: I tensorflow/core/common_runtime/bfc_allocator.cc:979] Bin (134217728): 	Total Chunks: 3, Chunks in use: 3. 488.00MiB allocated for chunks. 488.00MiB in use in bin. 408.00MiB client-requested in use in bin.
2021-01-12 12:19:35.072559: I tensorflow/core/common_runtime/bfc_allocator.cc:979] Bin (268435456): 	Total Chunks: 6, Chunks in use: 4. 3.16GiB allocated for chunks. 1.78GiB in use in bin. 1.78GiB client-requested in use in bin.
2021-01-12 12:19:35.072812: I tensorflow/core/common_runtime/bfc_allocator.cc:995] Bin for 1.12GiB was 256.00MiB, Chunk State: 
2021-01-12 12:19:35.072955: I tensorflow/core/common_runtime/bfc_allocator.cc:1001]   Size: 424.00MiB | Requested Size: 6.00MiB | in_use: 0 | bin_num: 20, prev:   Size: 216.00MiB | Requested Size: 216.00MiB | in_use: 1 | bin_num: -1
2021-01-12 12:19:35.073199: I tensorflow/core/common_runtime/bfc_allocator.cc:1001]   Size: 992.00MiB | Requested Size: 0B | in_use: 0 | bin_num: 20, prev:   Size: 768.00MiB | Requested Size: 768.00MiB | in_use: 1 | bin_num: -1
2021-01-12 12:19:35.073426: I tensorflow/core/common_runtime/bfc_allocator.cc:1008] Next region of size 1048576
2021-01-12 12:19:35.073548: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c200000 of size 1280 next 1
2021-01-12 12:19:35.073673: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c200500 of size 256 next 7
2021-01-12 12:19:35.073795: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c200600 of size 256 next 8
2021-01-12 12:19:35.073916: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c200700 of size 256 next 10
2021-01-12 12:19:35.074041: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c200800 of size 256 next 9
2021-01-12 12:19:35.074163: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c200900 of size 256 next 15
2021-01-12 12:19:35.074288: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c200a00 of size 256 next 17
2021-01-12 12:19:35.074414: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c200b00 of size 256 next 2
2021-01-12 12:19:35.074536: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c200c00 of size 256 next 3
2021-01-12 12:19:35.074659: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c200d00 of size 256 next 20
2021-01-12 12:19:35.074784: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c200e00 of size 256 next 23
2021-01-12 12:19:35.074914: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c200f00 of size 256 next 22
2021-01-12 12:19:35.075043: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c201000 of size 256 next 31
2021-01-12 12:19:35.075173: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c201100 of size 256 next 36
2021-01-12 12:19:35.075300: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c201200 of size 256 next 39
2021-01-12 12:19:35.075426: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c201300 of size 256 next 4
2021-01-12 12:19:35.075550: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c201400 of size 256 next 5
2021-01-12 12:19:35.075697: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c201500 of size 1792 next 6
2021-01-12 12:19:35.075832: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c201c00 of size 2048 next 29
2021-01-12 12:19:35.075974: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c202400 of size 2048 next 35
2021-01-12 12:19:35.076112: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c202c00 of size 256 next 41
2021-01-12 12:19:35.076250: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c202d00 of size 256 next 43
2021-01-12 12:19:35.076391: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c202e00 of size 256 next 44
2021-01-12 12:19:35.087555: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c202f00 of size 256 next 46
2021-01-12 12:19:35.087739: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c203000 of size 256 next 48
2021-01-12 12:19:35.087914: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c203100 of size 49920 next 11
2021-01-12 12:19:35.088092: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c20f400 of size 27648 next 12
2021-01-12 12:19:35.088497: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c216000 of size 256 next 50
2021-01-12 12:19:35.088675: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c216100 of size 256 next 51
2021-01-12 12:19:35.088848: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c216200 of size 256 next 52
2021-01-12 12:19:35.089022: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c216300 of size 256 next 53
2021-01-12 12:19:35.089198: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c216400 of size 256 next 54
2021-01-12 12:19:35.089374: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c216500 of size 256 next 55
2021-01-12 12:19:35.089548: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c216600 of size 256 next 56
2021-01-12 12:19:35.089723: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c216700 of size 256 next 57
2021-01-12 12:19:35.089897: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c216800 of size 256 next 58
2021-01-12 12:19:35.090070: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c216900 of size 256 next 59
2021-01-12 12:19:35.090243: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c216a00 of size 256 next 60
2021-01-12 12:19:35.090414: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c216b00 of size 256 next 61
2021-01-12 12:19:35.090585: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c216c00 of size 256 next 62
2021-01-12 12:19:35.090757: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c216d00 of size 256 next 63
2021-01-12 12:19:35.090928: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c216e00 of size 256 next 64
2021-01-12 12:19:35.091098: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c216f00 of size 256 next 65
2021-01-12 12:19:35.091302: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c217000 of size 1792 next 66
2021-01-12 12:19:35.091498: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c217700 of size 256 next 67
2021-01-12 12:19:35.091693: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c217800 of size 256 next 68
2021-01-12 12:19:35.091886: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c217900 of size 256 next 69
2021-01-12 12:19:35.092081: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c217a00 of size 256 next 70
2021-01-12 12:19:35.092276: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c217b00 of size 256 next 71
2021-01-12 12:19:35.092471: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c217c00 of size 256 next 72
2021-01-12 12:19:35.092671: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c217d00 of size 2048 next 73
2021-01-12 12:19:35.092872: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c218500 of size 2048 next 74
2021-01-12 12:19:35.093071: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c218d00 of size 256 next 75
2021-01-12 12:19:35.093266: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c218e00 of size 256 next 77
2021-01-12 12:19:35.093461: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c218f00 of size 256 next 79
2021-01-12 12:19:35.093658: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c219000 of size 256 next 81
2021-01-12 12:19:35.093855: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c219100 of size 256 next 83
2021-01-12 12:19:35.094049: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c219200 of size 256 next 85
2021-01-12 12:19:35.094232: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c219300 of size 256 next 86
2021-01-12 12:19:35.094349: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c219400 of size 256 next 87
2021-01-12 12:19:35.094466: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c219500 of size 1792 next 88
2021-01-12 12:19:35.102780: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c219c00 of size 256 next 89
2021-01-12 12:19:35.102932: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c219d00 of size 256 next 91
2021-01-12 12:19:35.103081: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c219e00 of size 256 next 93
2021-01-12 12:19:35.103229: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c219f00 of size 256 next 95
2021-01-12 12:19:35.103549: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c21a000 of size 256 next 97
2021-01-12 12:19:35.103842: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c21a100 of size 256 next 99
2021-01-12 12:19:35.104071: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c21a200 of size 2048 next 101
2021-01-12 12:19:35.104290: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c21aa00 of size 2048 next 102
2021-01-12 12:19:35.104441: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c21b200 of size 256 next 104
2021-01-12 12:19:35.104791: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c21b300 of size 256 next 106
2021-01-12 12:19:35.105005: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c21b400 of size 256 next 108
2021-01-12 12:19:35.105260: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c21b500 of size 256 next 110
2021-01-12 12:19:35.105468: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c21b600 of size 256 next 112
2021-01-12 12:19:35.105685: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c21b700 of size 256 next 114
2021-01-12 12:19:35.105886: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c21b800 of size 256 next 115
2021-01-12 12:19:35.106095: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c21b900 of size 256 next 116
2021-01-12 12:19:35.106332: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c21ba00 of size 256 next 117
2021-01-12 12:19:35.106538: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c21bb00 of size 256 next 118
2021-01-12 12:19:35.106740: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c21bc00 of size 256 next 119
2021-01-12 12:19:35.106945: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c21bd00 of size 256 next 120
2021-01-12 12:19:35.107156: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c21be00 of size 256 next 121
2021-01-12 12:19:35.107351: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c21bf00 of size 256 next 122
2021-01-12 12:19:35.107548: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c21c000 of size 256 next 123
2021-01-12 12:19:35.107739: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c21c100 of size 256 next 124
2021-01-12 12:19:35.107932: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c21c200 of size 256 next 125
2021-01-12 12:19:35.108126: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c21c300 of size 256 next 126
2021-01-12 12:19:35.108325: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c21c400 of size 256 next 127
2021-01-12 12:19:35.108517: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c21c500 of size 256 next 128
2021-01-12 12:19:35.108681: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c21c600 of size 256 next 129
2021-01-12 12:19:35.109024: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] Free  at 70c21c700 of size 256 next 133
2021-01-12 12:19:35.109250: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c21c800 of size 256 next 136
2021-01-12 12:19:35.109484: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c21c900 of size 256 next 137
2021-01-12 12:19:35.109727: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c21ca00 of size 256 next 142
2021-01-12 12:19:35.109952: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c21cb00 of size 256 next 49
2021-01-12 12:19:35.110105: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c21cc00 of size 27648 next 13
2021-01-12 12:19:35.110413: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c223800 of size 55296 next 14
2021-01-12 12:19:35.110647: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c231000 of size 82944 next 47
2021-01-12 12:19:35.110884: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c245400 of size 138240 next 18
2021-01-12 12:19:35.111086: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c267000 of size 110592 next 19
2021-01-12 12:19:35.111294: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c282000 of size 110592 next 45
2021-01-12 12:19:35.111486: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c29d000 of size 110592 next 16
2021-01-12 12:19:35.111682: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c2b8000 of size 294912 next 18446744073709551615
2021-01-12 12:19:35.111935: I tensorflow/core/common_runtime/bfc_allocator.cc:1008] Next region of size 2097152
2021-01-12 12:19:35.112038: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c400000 of size 442368 next 40
2021-01-12 12:19:35.112146: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c46c000 of size 442368 next 24
2021-01-12 12:19:35.112254: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c4d8000 of size 442368 next 25
2021-01-12 12:19:35.112362: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c544000 of size 331776 next 42
2021-01-12 12:19:35.112469: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c595000 of size 438272 next 18446744073709551615
2021-01-12 12:19:35.112590: I tensorflow/core/common_runtime/bfc_allocator.cc:1008] Next region of size 4194304
2021-01-12 12:19:35.112685: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c600000 of size 442368 next 76
2021-01-12 12:19:35.112793: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c66c000 of size 331776 next 78
2021-01-12 12:19:35.112900: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c6bd000 of size 110592 next 80
2021-01-12 12:19:35.113007: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c6d8000 of size 82944 next 82
2021-01-12 12:19:35.113114: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c6ec400 of size 27648 next 84
2021-01-12 12:19:35.113222: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c6f3000 of size 27648 next 90
2021-01-12 12:19:35.113330: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c6f9c00 of size 55296 next 92
2021-01-12 12:19:35.113437: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c707400 of size 110592 next 94
2021-01-12 12:19:35.113544: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c722400 of size 221184 next 96
2021-01-12 12:19:35.113650: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c758400 of size 442368 next 98
2021-01-12 12:19:35.113757: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c7c4400 of size 442368 next 105
2021-01-12 12:19:35.113864: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c830400 of size 331776 next 107
2021-01-12 12:19:35.113974: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c881400 of size 110592 next 109
2021-01-12 12:19:35.114084: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c89c400 of size 82944 next 111
2021-01-12 12:19:35.114195: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70c8b0800 of size 27648 next 113
2021-01-12 12:19:35.114302: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] Free  at 70c8b7400 of size 1346560 next 18446744073709551615
2021-01-12 12:19:35.114424: I tensorflow/core/common_runtime/bfc_allocator.cc:1008] Next region of size 8388608
2021-01-12 12:19:35.114520: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70ca00000 of size 3538944 next 28
2021-01-12 12:19:35.114629: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70cd60000 of size 4849664 next 18446744073709551615
2021-01-12 12:19:35.114756: I tensorflow/core/common_runtime/bfc_allocator.cc:1008] Next region of size 33554432
2021-01-12 12:19:35.114856: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70d200000 of size 3981312 next 37
2021-01-12 12:19:35.114967: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70d5cc000 of size 3981312 next 38
2021-01-12 12:19:35.115076: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70d998000 of size 3538944 next 100
2021-01-12 12:19:35.115187: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70dcf8000 of size 3981312 next 103
2021-01-12 12:19:35.115299: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70e0c4000 of size 18071552 next 18446744073709551615
2021-01-12 12:19:35.118785: I tensorflow/core/common_runtime/bfc_allocator.cc:1008] Next region of size 33554432
2021-01-12 12:19:35.118912: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 70f200000 of size 33554432 next 18446744073709551615
2021-01-12 12:19:35.119067: I tensorflow/core/common_runtime/bfc_allocator.cc:1008] Next region of size 67108864
2021-01-12 12:19:35.119374: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 711200000 of size 28311552 next 34
2021-01-12 12:19:35.119667: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 712d00000 of size 38797312 next 18446744073709551615
2021-01-12 12:19:35.119915: I tensorflow/core/common_runtime/bfc_allocator.cc:1008] Next region of size 134217728
2021-01-12 12:19:35.120140: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 715200000 of size 25165824 next 131
2021-01-12 12:19:35.120478: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 716a00000 of size 25165824 next 132
2021-01-12 12:19:35.120662: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 718200000 of size 25165824 next 138
2021-01-12 12:19:35.120851: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 719a00000 of size 25165824 next 145
2021-01-12 12:19:35.121043: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 71b200000 of size 33554432 next 18446744073709551615
2021-01-12 12:19:35.121260: I tensorflow/core/common_runtime/bfc_allocator.cc:1008] Next region of size 536870912
2021-01-12 12:19:35.121415: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 71d200000 of size 402653184 next 135
2021-01-12 12:19:35.121588: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 735200000 of size 50331648 next 141
2021-01-12 12:19:35.121761: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 738200000 of size 25165824 next 146
2021-01-12 12:19:35.121929: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 739a00000 of size 3145728 next 148
2021-01-12 12:19:35.122094: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 739d00000 of size 25165824 next 149
2021-01-12 12:19:35.122263: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 73b500000 of size 30408704 next 18446744073709551615
2021-01-12 12:19:35.122456: I tensorflow/core/common_runtime/bfc_allocator.cc:1008] Next region of size 536870912
2021-01-12 12:19:35.122603: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 73d200000 of size 402653184 next 139
2021-01-12 12:19:35.122775: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 755200000 of size 134217728 next 18446744073709551615
2021-01-12 12:19:35.122967: I tensorflow/core/common_runtime/bfc_allocator.cc:1008] Next region of size 1073741824
2021-01-12 12:19:35.123118: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 75d200000 of size 100663296 next 144
2021-01-12 12:19:35.123343: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 763200000 of size 25165824 next 150
2021-01-12 12:19:35.123682: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 764a00000 of size 25165824 next 162
2021-01-12 12:19:35.123900: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 766200000 of size 100663296 next 183
2021-01-12 12:19:35.124130: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 76c200000 of size 150994944 next 167
2021-01-12 12:19:35.124303: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 775200000 of size 226492416 next 163
2021-01-12 12:19:35.124473: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] Free  at 782a00000 of size 444596224 next 18446744073709551615
2021-01-12 12:19:35.124671: I tensorflow/core/common_runtime/bfc_allocator.cc:1008] Next region of size 2147483648
2021-01-12 12:19:35.124822: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 7b9600000 of size 301989888 next 182
2021-01-12 12:19:35.124994: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] InUse at 7cb600000 of size 805306368 next 221
2021-01-12 12:19:35.125158: I tensorflow/core/common_runtime/bfc_allocator.cc:1028] Free  at 7fb600000 of size 1040187392 next 18446744073709551615
2021-01-12 12:19:35.125354: I tensorflow/core/common_runtime/bfc_allocator.cc:1033]      Summary of in-use Chunks by size: 
2021-01-12 12:19:35.125522: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 82 Chunks of size 256 totalling 20.5KiB
2021-01-12 12:19:35.125682: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 1 Chunks of size 1280 totalling 1.2KiB
2021-01-12 12:19:35.125843: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 3 Chunks of size 1792 totalling 5.2KiB
2021-01-12 12:19:35.126002: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 6 Chunks of size 2048 totalling 12.0KiB
2021-01-12 12:19:35.126162: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 5 Chunks of size 27648 totalling 135.0KiB
2021-01-12 12:19:35.126335: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 1 Chunks of size 49920 totalling 48.8KiB
2021-01-12 12:19:35.126493: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 2 Chunks of size 55296 totalling 108.0KiB
2021-01-12 12:19:35.126660: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 3 Chunks of size 82944 totalling 243.0KiB
2021-01-12 12:19:35.126829: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 6 Chunks of size 110592 totalling 648.0KiB
2021-01-12 12:19:35.127000: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 1 Chunks of size 138240 totalling 135.0KiB
2021-01-12 12:19:35.127171: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 1 Chunks of size 221184 totalling 216.0KiB
2021-01-12 12:19:35.127340: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 1 Chunks of size 294912 totalling 288.0KiB
2021-01-12 12:19:35.127506: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 3 Chunks of size 331776 totalling 972.0KiB
2021-01-12 12:19:35.127678: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 1 Chunks of size 438272 totalling 428.0KiB
2021-01-12 12:19:35.127894: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 6 Chunks of size 442368 totalling 2.53MiB
2021-01-12 12:19:35.128056: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 1 Chunks of size 3145728 totalling 3.00MiB
2021-01-12 12:19:35.128226: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 2 Chunks of size 3538944 totalling 6.75MiB
2021-01-12 12:19:35.128396: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 3 Chunks of size 3981312 totalling 11.39MiB
2021-01-12 12:19:35.128571: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 1 Chunks of size 4849664 totalling 4.62MiB
2021-01-12 12:19:35.128742: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 1 Chunks of size 18071552 totalling 17.23MiB
2021-01-12 12:19:35.128918: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 8 Chunks of size 25165824 totalling 192.00MiB
2021-01-12 12:19:35.129089: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 1 Chunks of size 28311552 totalling 27.00MiB
2021-01-12 12:19:35.129260: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 1 Chunks of size 30408704 totalling 29.00MiB
2021-01-12 12:19:35.129430: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 2 Chunks of size 33554432 totalling 64.00MiB
2021-01-12 12:19:35.129599: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 1 Chunks of size 38797312 totalling 37.00MiB
2021-01-12 12:19:35.129761: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 1 Chunks of size 50331648 totalling 48.00MiB
2021-01-12 12:19:35.129928: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 2 Chunks of size 100663296 totalling 192.00MiB
2021-01-12 12:19:35.130103: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 1 Chunks of size 134217728 totalling 128.00MiB
2021-01-12 12:19:35.130274: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 1 Chunks of size 150994944 totalling 144.00MiB
2021-01-12 12:19:35.130447: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 1 Chunks of size 226492416 totalling 216.00MiB
2021-01-12 12:19:35.130618: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 1 Chunks of size 301989888 totalling 288.00MiB
2021-01-12 12:19:35.130789: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 2 Chunks of size 402653184 totalling 768.00MiB
2021-01-12 12:19:35.130961: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] 1 Chunks of size 805306368 totalling 768.00MiB
2021-01-12 12:19:35.131130: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Sum Total of in-use chunks: 2.88GiB
2021-01-12 12:19:35.131293: I tensorflow/core/common_runtime/bfc_allocator.cc:1042] total_region_allocated_bytes_: 4579131392 memory_limit_: 4949437312 available bytes: 370305920 curr_region_allocation_bytes_: 4294967296
2021-01-12 12:19:35.131716: I tensorflow/core/common_runtime/bfc_allocator.cc:1048] Stats: 
Limit:                      4949437312
InUse:                      3093000960
MaxInUse:                   3559928576
NumAllocs:                         645
MaxAllocSize:               1503133696
Reserved:                            0
PeakReserved:                        0
LargestFreeBlock:                    0

2021-01-12 12:19:35.132202: W tensorflow/core/common_runtime/bfc_allocator.cc:441] ********************************************_________*************************______________________
2021-01-12 12:19:35.132389: W tensorflow/core/framework/op_kernel.cc:1763] OP_REQUIRES failed at concat_op.cc:158 : Resource exhausted: OOM when allocating tensor with shape[3,128,128,128,48] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
  File "C:/Users/Windground/PycharmProjects/3D-Seismic-Image-Fault-Segmentation/fault_segmentation.py", line 68, in <module>
    main()
  File "C:/Users/Windground/PycharmProjects/3D-Seismic-Image-Fault-Segmentation/fault_segmentation.py", line 30, in main
    model, history_dict = train_model(Dataset, CompileModel, TrainModel,
  File "C:\Users\Windground\PycharmProjects\3D-Seismic-Image-Fault-Segmentation\training.py", line 94, in train_model
    history = model.fit_generator(generator=tr_gen, validation_data=val_gen,
  File "C:\Users\Windground\PycharmProjects\3D-Seismic-Image-Fault-Segmentation\venv\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1847, in fit_generator
    return self.fit(
  File "C:\Users\Windground\PycharmProjects\3D-Seismic-Image-Fault-Segmentation\venv\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1100, in fit
    tmp_logs = self.train_function(iterator)
  File "C:\Users\Windground\PycharmProjects\3D-Seismic-Image-Fault-Segmentation\venv\lib\site-packages\tensorflow\python\eager\def_function.py", line 828, in __call__
    result = self._call(*args, **kwds)
  File "C:\Users\Windground\PycharmProjects\3D-Seismic-Image-Fault-Segmentation\venv\lib\site-packages\tensorflow\python\eager\def_function.py", line 888, in _call
    return self._stateless_fn(*args, **kwds)
  File "C:\Users\Windground\PycharmProjects\3D-Seismic-Image-Fault-Segmentation\venv\lib\site-packages\tensorflow\python\eager\function.py", line 2942, in __call__
    return graph_function._call_flat(
  File "C:\Users\Windground\PycharmProjects\3D-Seismic-Image-Fault-Segmentation\venv\lib\site-packages\tensorflow\python\eager\function.py", line 1918, in _call_flat
    return self._build_call_outputs(self._inference_function.call(
  File "C:\Users\Windground\PycharmProjects\3D-Seismic-Image-Fault-Segmentation\venv\lib\site-packages\tensorflow\python\eager\function.py", line 555, in call
    outputs = execute.execute(
  File "C:\Users\Windground\PycharmProjects\3D-Seismic-Image-Fault-Segmentation\venv\lib\site-packages\tensorflow\python\eager\execute.py", line 59, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.ResourceExhaustedError:  OOM when allocating tensor with shape[3,128,128,128,48] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
	 [[node model/concatenate_2/concat (defined at \Users\Windground\PycharmProjects\3D-Seismic-Image-Fault-Segmentation\training.py:94) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
 [Op:__inference_train_function_6617]

Function call stack:
train_function

2021-01-12 12:19:35.359483: W tensorflow/core/kernels/data/generator_dataset_op.cc:107] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated.
	 [[{{node PyFunc}}]]

Process finished with exit code 1