Dear,
I tried to test the Point-GNN on my own Lidar data, sometimes, there are errors, as is shown below.
Could anyone tell me the reason for those errors? I am really appreciative for any advices!
`(py37) objloc@deeplearning:~/OBJLOC/Point-GNN$ python3 run.py checkpoints/car_auto_T3_trainval/ --test --dataset_root_dir pointGNN_dataset_small --output_dir OUTPUT --level 2
/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:529: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:530: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:535: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
@ level 0 Graph, Add layer: layer1, type: scatter_max_point_set_pooling
WARNING:tensorflow:From /home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
Feature Dim:300
@ level 1 Graph, Add layer: layer2, type: scatter_max_graph_auto_center_net
Feature Dim:300
@ level 1 Graph, Add layer: layer3, type: scatter_max_graph_auto_center_net
Feature Dim:300
@ level 1 Graph, Add layer: layer4, type: scatter_max_graph_auto_center_net
Feature Dim:300
Final Feature Dim:300
Prediction 4 classes
2020-03-31 17:26:28.163067: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-03-31 17:26:28.185768: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3397755000 Hz
2020-03-31 17:26:28.186787: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x55d32ffca7f0 executing computations on platform Host. Devices:
2020-03-31 17:26:28.186831: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): ,
2020-03-31 17:26:28.324051: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: TITAN X (Pascal) major: 6 minor: 1 memoryClockRate(GHz): 1.531
pciBusID: 0000:01:00.0
totalMemory: 11.91GiB freeMemory: 11.07GiB
2020-03-31 17:26:28.324083: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2020-03-31 17:26:28.324847: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-03-31 17:26:28.324860: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2020-03-31 17:26:28.324867: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2020-03-31 17:26:28.324936: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10772 MB memory) -> physical GPU (device: 0, name: TITAN X (Pascal), pci bus id: 0000:01:00.0, compute capability: 6.1)
2020-03-31 17:26:28.326513: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x55d3306e2920 executing computations on platform CUDA. Devices:
2020-03-31 17:26:28.326531: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): TITAN X (Pascal), Compute Capability 6.1
Restore from checkpoint checkpoints/car_auto_T3_trainval/model-1400000
WARNING:tensorflow:From /home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
0%| | 0/21 [00:00<?, ?it/s]2020-03-31 17:26:29.195655: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
Configure the viewpoint as you want and press [q]
5%|████▌ | 1/21 [00:04<01:24, 4.22s/it]Configure the viewpoint as you want and press [q]
10%|█████████▏ | 2/21 [00:05<01:03, 3.37s/it]Configure the viewpoint as you want and press [q]
14%|█████████████▋ | 3/21 [00:06<00:49, 2.74s/it]Configure the viewpoint as you want and press [q]
19%|██████████████████▎ | 4/21 [00:08<00:38, 2.26s/it]Configure the viewpoint as you want and press [q]
24%|██████████████████████▊ | 5/21 [00:09<00:31, 1.94s/it]Configure the viewpoint as you want and press [q]
29%|███████████████████████████▍ | 6/21 [00:10<00:25, 1.73s/it]Configure the viewpoint as you want and press [q]
33%|████████████████████████████████ | 7/21 [00:11<00:21, 1.57s/it]Configure the viewpoint as you want and press [q]
38%|████████████████████████████████████▌ | 8/21 [00:12<00:18, 1.44s/it]Configure the viewpoint as you want and press [q]
43%|█████████████████████████████████████████▏ | 9/21 [00:14<00:17, 1.49s/it]Configure the viewpoint as you want and press [q]
48%|█████████████████████████████████████████████▏ | 10/21 [00:15<00:15, 1.42s/it]Configure the viewpoint as you want and press [q]
52%|█████████████████████████████████████████████████▊ | 11/21 [00:16<00:13, 1.34s/it]Configure the viewpoint as you want and press [q]
57%|██████████████████████████████████████████████████████▎ | 12/21 [00:29<00:43, 4.79s/it]Qt: Cannot set locale modifiers:
Configure the viewpoint as you want and press [q]
62%|██████████████████████████████████████████████████████████▊ | 13/21 [00:33<00:36, 4.54s/it]Configure the viewpoint as you want and press [q]
67%|███████████████████████████████████████████████████████████████▎ | 14/21 [00:59<01:15, 10.82s/it]Configure the viewpoint as you want and press [q]
71%|███████████████████████████████████████████████████████████████████▊ | 15/21 [01:08<01:02, 10.35s/it]2020-03-31 17:27:47.639016: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.30GiB. Current allocation summary follows.
2020-03-31 17:27:47.639090: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256): Total Chunks: 93, Chunks in use: 93. 23.2KiB allocated for chunks. 23.2KiB in use in bin. 7.3KiB client-requested in use in bin.
2020-03-31 17:27:47.639113: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512): Total Chunks: 6, Chunks in use: 6. 3.8KiB allocated for chunks. 3.8KiB in use in bin. 3.8KiB client-requested in use in bin.
2020-03-31 17:27:47.639134: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024): Total Chunks: 36, Chunks in use: 36. 46.8KiB allocated for chunks. 46.8KiB in use in bin. 44.2KiB client-requested in use in bin.
2020-03-31 17:27:47.639153: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-03-31 17:27:47.639170: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-03-31 17:27:47.639188: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192): Total Chunks: 1, Chunks in use: 1. 8.0KiB allocated for chunks. 8.0KiB in use in bin. 8.0KiB client-requested in use in bin.
2020-03-31 17:27:47.639208: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384): Total Chunks: 4, Chunks in use: 4. 64.0KiB allocated for chunks. 64.0KiB in use in bin. 64.0KiB client-requested in use in bin.
2020-03-31 17:27:47.639228: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768): Total Chunks: 1, Chunks in use: 1. 32.0KiB allocated for chunks. 32.0KiB in use in bin. 32.0KiB client-requested in use in bin.
2020-03-31 17:27:47.639249: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536): Total Chunks: 9, Chunks in use: 9. 721.5KiB allocated for chunks. 721.5KiB in use in bin. 721.4KiB client-requested in use in bin.
2020-03-31 17:27:47.639269: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072): Total Chunks: 1, Chunks in use: 1. 150.0KiB allocated for chunks. 150.0KiB in use in bin. 150.0KiB client-requested in use in bin.
2020-03-31 17:27:47.639289: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144): Total Chunks: 15, Chunks in use: 13. 5.43MiB allocated for chunks. 4.58MiB in use in bin. 4.47MiB client-requested in use in bin.
2020-03-31 17:27:47.639309: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288): Total Chunks: 1, Chunks in use: 1. 555.0KiB allocated for chunks. 555.0KiB in use in bin. 355.1KiB client-requested in use in bin.
2020-03-31 17:27:47.639327: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576): Total Chunks: 1, Chunks in use: 0. 1.36MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-03-31 17:27:47.639345: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-03-31 17:27:47.639361: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-03-31 17:27:47.639381: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608): Total Chunks: 4, Chunks in use: 3. 42.66MiB allocated for chunks. 34.49MiB in use in bin. 27.42MiB client-requested in use in bin.
2020-03-31 17:27:47.639402: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216): Total Chunks: 3, Chunks in use: 2. 75.98MiB allocated for chunks. 47.18MiB in use in bin. 46.70MiB client-requested in use in bin.
2020-03-31 17:27:47.639419: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-03-31 17:27:47.639436: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-03-31 17:27:47.639453: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728): Total Chunks: 1, Chunks in use: 0. 128.00MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-03-31 17:27:47.639472: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456): Total Chunks: 6, Chunks in use: 1. 8.25GiB allocated for chunks. 2.28GiB in use in bin. 2.28GiB client-requested in use in bin.
2020-03-31 17:27:47.639491: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 2.30GiB was 256.00MiB, Chunk State:
2020-03-31 17:27:47.639511: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 256.00MiB | Requested Size: 174.22MiB | in_use: 0
2020-03-31 17:27:47.639530: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 1.00GiB | Requested Size: 408.34MiB | in_use: 0
2020-03-31 17:27:47.639547: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 1.00GiB | Requested Size: 827.68MiB | in_use: 0
2020-03-31 17:27:47.639566: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 1.72GiB | Requested Size: 1.36MiB | in_use: 0, prev: Size: 2.28GiB | Requested Size: 2.28GiB | in_use: 1
2020-03-31 17:27:47.639583: I tensorflow/core/common_runtime/bfc_allocator.cc:619] Size: 2.00GiB | Requested Size: 1.01GiB | in_use: 0
2020-03-31 17:27:47.639601: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faada000000 of size 2448264192
2020-03-31 17:27:47.639615: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fab6bed8c00 of size 1846703104
2020-03-31 17:27:47.639628: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fabe0000000 of size 2147483648
2020-03-31 17:27:47.639641: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7facc8000000 of size 1073741824
2020-03-31 17:27:47.639654: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fad08000000 of size 1073741824
2020-03-31 17:27:47.639667: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fad48000000 of size 268435456
2020-03-31 17:27:47.639680: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fad58000000 of size 134217728
2020-03-31 17:27:47.639693: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fad60000000 of size 12432128
2020-03-31 17:27:47.639708: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fad60bdb300 of size 24482816
2020-03-31 17:27:47.639721: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fad62334700 of size 30193920
2020-03-31 17:27:47.639734: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fadd8000000 of size 1427456
2020-03-31 17:27:47.639748: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadd815c800 of size 15349760
2020-03-31 17:27:47.639761: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7fadd9000000 of size 8564224
2020-03-31 17:27:47.639774: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadd982ae00 of size 24990208
2020-03-31 17:27:47.639788: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde800000 of size 256
2020-03-31 17:27:47.639801: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde800100 of size 256
2020-03-31 17:27:47.639814: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde800200 of size 256
2020-03-31 17:27:47.639827: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde800300 of size 256
2020-03-31 17:27:47.639839: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde800400 of size 256
2020-03-31 17:27:47.639852: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde800500 of size 256
2020-03-31 17:27:47.639865: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde800600 of size 256
2020-03-31 17:27:47.639878: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde800700 of size 256
2020-03-31 17:27:47.639891: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde800800 of size 256
2020-03-31 17:27:47.639903: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde800900 of size 256
2020-03-31 17:27:47.639916: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde800a00 of size 256
2020-03-31 17:27:47.639929: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde800b00 of size 256
2020-03-31 17:27:47.639941: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde800c00 of size 256
2020-03-31 17:27:47.639954: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde800d00 of size 256
2020-03-31 17:27:47.639967: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde800e00 of size 256
2020-03-31 17:27:47.639980: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde800f00 of size 256
2020-03-31 17:27:47.639992: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde801000 of size 256
2020-03-31 17:27:47.640005: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde801100 of size 256
2020-03-31 17:27:47.640018: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde801200 of size 256
2020-03-31 17:27:47.640031: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde801300 of size 256
2020-03-31 17:27:47.640044: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde801400 of size 256
2020-03-31 17:27:47.640056: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde801500 of size 256
2020-03-31 17:27:47.640069: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde801600 of size 256
2020-03-31 17:27:47.640082: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde801700 of size 256
2020-03-31 17:27:47.640094: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde801800 of size 256
2020-03-31 17:27:47.640107: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde801900 of size 256
2020-03-31 17:27:47.640120: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde801a00 of size 256
2020-03-31 17:27:47.640132: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde801b00 of size 256
2020-03-31 17:27:47.640145: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde801c00 of size 256
2020-03-31 17:27:47.640158: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde801d00 of size 256
2020-03-31 17:27:47.640170: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde801e00 of size 256
2020-03-31 17:27:47.640183: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde801f00 of size 256
2020-03-31 17:27:47.640196: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde802000 of size 256
2020-03-31 17:27:47.640209: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde802100 of size 256
2020-03-31 17:27:47.640222: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde802200 of size 256
2020-03-31 17:27:47.640235: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde802300 of size 256
2020-03-31 17:27:47.640248: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde802400 of size 256
2020-03-31 17:27:47.640260: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde802500 of size 256
2020-03-31 17:27:47.640273: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde802600 of size 256
2020-03-31 17:27:47.640287: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde802700 of size 1280
2020-03-31 17:27:47.640300: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde802c00 of size 256
2020-03-31 17:27:47.640312: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde802d00 of size 256
2020-03-31 17:27:47.640325: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde802e00 of size 1280
2020-03-31 17:27:47.640338: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde803300 of size 256
2020-03-31 17:27:47.640351: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde803400 of size 1280
2020-03-31 17:27:47.640364: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde803900 of size 256
2020-03-31 17:27:47.640377: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde803a00 of size 1280
2020-03-31 17:27:47.640390: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde803f00 of size 256
2020-03-31 17:27:47.640403: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde804000 of size 256
2020-03-31 17:27:47.640416: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde804100 of size 256
2020-03-31 17:27:47.640429: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde804200 of size 256
2020-03-31 17:27:47.640442: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde804300 of size 256
2020-03-31 17:27:47.640455: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde804400 of size 1280
2020-03-31 17:27:47.640467: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde804900 of size 256
2020-03-31 17:27:47.640480: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde804a00 of size 1280
2020-03-31 17:27:47.640493: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde804f00 of size 256
2020-03-31 17:27:47.640506: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde805000 of size 1280
2020-03-31 17:27:47.640519: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde805500 of size 256
2020-03-31 17:27:47.640532: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde805600 of size 1280
2020-03-31 17:27:47.640545: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde805b00 of size 256
2020-03-31 17:27:47.640558: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde805c00 of size 256
2020-03-31 17:27:47.640570: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde805d00 of size 256
2020-03-31 17:27:47.640584: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde805e00 of size 256
2020-03-31 17:27:47.640597: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde805f00 of size 256
2020-03-31 17:27:47.640610: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde806000 of size 1280
2020-03-31 17:27:47.640622: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde806500 of size 256
2020-03-31 17:27:47.640635: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde806600 of size 1280
2020-03-31 17:27:47.640649: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde806b00 of size 256
2020-03-31 17:27:47.640665: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde806c00 of size 1280
2020-03-31 17:27:47.640676: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde807100 of size 256
2020-03-31 17:27:47.640687: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde807200 of size 1280
2020-03-31 17:27:47.640700: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde807700 of size 256
2020-03-31 17:27:47.640724: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde807800 of size 256
2020-03-31 17:27:47.640737: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde807900 of size 1280
2020-03-31 17:27:47.640749: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde807e00 of size 256
2020-03-31 17:27:47.640762: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde807f00 of size 256
2020-03-31 17:27:47.640775: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde808000 of size 512
2020-03-31 17:27:47.640788: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde808200 of size 256
2020-03-31 17:27:47.640801: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde808300 of size 256
2020-03-31 17:27:47.640814: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde808400 of size 256
2020-03-31 17:27:47.640827: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde808500 of size 256
2020-03-31 17:27:47.640839: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde808600 of size 256
2020-03-31 17:27:47.640852: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde808700 of size 256
2020-03-31 17:27:47.640865: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde808800 of size 256
2020-03-31 17:27:47.640877: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde808900 of size 1280
2020-03-31 17:27:47.640890: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde808e00 of size 256
2020-03-31 17:27:47.640903: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde808f00 of size 1280
2020-03-31 17:27:47.640915: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde809400 of size 1280
2020-03-31 17:27:47.640929: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde809900 of size 1792
2020-03-31 17:27:47.640941: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde80a000 of size 256
2020-03-31 17:27:47.640955: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde80a100 of size 16384
2020-03-31 17:27:47.640968: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde80e100 of size 256
2020-03-31 17:27:47.640981: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde80e200 of size 76800
2020-03-31 17:27:47.640994: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde820e00 of size 256
2020-03-31 17:27:47.641006: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde820f00 of size 1792
2020-03-31 17:27:47.641019: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde821600 of size 256
2020-03-31 17:27:47.641032: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde821700 of size 16384
2020-03-31 17:27:47.641045: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde825700 of size 256
2020-03-31 17:27:47.641057: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde825800 of size 76800
2020-03-31 17:27:47.641071: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde838400 of size 256
2020-03-31 17:27:47.641084: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde838500 of size 1792
2020-03-31 17:27:47.641097: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde838c00 of size 256
2020-03-31 17:27:47.641110: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde838d00 of size 16384
2020-03-31 17:27:47.641123: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde83cd00 of size 256
2020-03-31 17:27:47.641136: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde83ce00 of size 76800
2020-03-31 17:27:47.641148: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde84fa00 of size 256
2020-03-31 17:27:47.641161: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde84fb00 of size 1792
2020-03-31 17:27:47.641174: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde850200 of size 256
2020-03-31 17:27:47.641186: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde850300 of size 16384
2020-03-31 17:27:47.641199: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde854300 of size 256
2020-03-31 17:27:47.641211: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde854400 of size 76800
2020-03-31 17:27:47.641224: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde867000 of size 256
2020-03-31 17:27:47.641237: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde867100 of size 1024
2020-03-31 17:27:47.641250: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde867500 of size 256
2020-03-31 17:27:47.641263: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde867600 of size 76800
2020-03-31 17:27:47.641275: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde87a200 of size 256
2020-03-31 17:27:47.641289: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde87a300 of size 768
2020-03-31 17:27:47.641301: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde87a600 of size 256
2020-03-31 17:27:47.641314: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde87a700 of size 76800
2020-03-31 17:27:47.641327: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde88d300 of size 256
2020-03-31 17:27:47.641340: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadde88d400 of size 470016
2020-03-31 17:27:47.641353: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddec00000 of size 1280
2020-03-31 17:27:47.641366: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddec00500 of size 363776
2020-03-31 17:27:47.641379: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddec59200 of size 1280
2020-03-31 17:27:47.641392: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddec59700 of size 360192
2020-03-31 17:27:47.641405: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddecb1600 of size 1280
2020-03-31 17:27:47.641418: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddecb1b00 of size 360192
2020-03-31 17:27:47.641431: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadded09a00 of size 1280
2020-03-31 17:27:47.641444: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadded09f00 of size 768
2020-03-31 17:27:47.641456: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadded0a200 of size 256
2020-03-31 17:27:47.641469: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadded0a300 of size 76800
2020-03-31 17:27:47.641482: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadded1cf00 of size 256
2020-03-31 17:27:47.641495: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadded1d000 of size 360192
2020-03-31 17:27:47.641507: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadded74f00 of size 1280
2020-03-31 17:27:47.641520: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7fadded75400 of size 568320
2020-03-31 17:27:47.641534: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddee00000 of size 1280
2020-03-31 17:27:47.641546: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddee00500 of size 360192
2020-03-31 17:27:47.641559: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddee58400 of size 1280
2020-03-31 17:27:47.641572: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddee58900 of size 360192
2020-03-31 17:27:47.641584: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddeeb0800 of size 1280
2020-03-31 17:27:47.641597: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddeeb0d00 of size 768
2020-03-31 17:27:47.641610: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddeeb1000 of size 256
2020-03-31 17:27:47.641622: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddeeb1100 of size 76800
2020-03-31 17:27:47.641662: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddeec3d00 of size 256
2020-03-31 17:27:47.641678: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddeec3e00 of size 360192
2020-03-31 17:27:47.641690: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddef1bd00 of size 1280
2020-03-31 17:27:47.641703: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddef1c200 of size 363776
2020-03-31 17:27:47.641716: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddef74f00 of size 1280
2020-03-31 17:27:47.641728: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddef75400 of size 360192
2020-03-31 17:27:47.641741: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddefcd300 of size 1280
2020-03-31 17:27:47.641754: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddefcd800 of size 360192
2020-03-31 17:27:47.641766: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddf025700 of size 1280
2020-03-31 17:27:47.641779: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddf025c00 of size 153600
2020-03-31 17:27:47.641792: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddf04b400 of size 1280
2020-03-31 17:27:47.641805: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddf04b900 of size 32768
2020-03-31 17:27:47.641818: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddf053900 of size 512
2020-03-31 17:27:47.641831: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddf053b00 of size 8192
2020-03-31 17:27:47.641843: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddf055b00 of size 256
2020-03-31 17:27:47.641856: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddf055c00 of size 512
2020-03-31 17:27:47.641869: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddf055e00 of size 256
2020-03-31 17:27:47.641882: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddf055f00 of size 360192
2020-03-31 17:27:47.641894: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddf0ade00 of size 1280
2020-03-31 17:27:47.641907: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddf0ae300 of size 360192
2020-03-31 17:27:47.641920: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddf106200 of size 1280
2020-03-31 17:27:47.641932: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7faddf106700 of size 429568
2020-03-31 17:27:47.641946: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddf16f500 of size 124416
2020-03-31 17:27:47.641958: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Free at 0x7faddf18db00 of size 468224
2020-03-31 17:27:47.641972: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7faddf800000 of size 8388608
2020-03-31 17:27:47.641984: I tensorflow/core/common_runtime/bfc_allocator.cc:638] Summary of in-use Chunks by size:
2020-03-31 17:27:47.642001: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 93 Chunks of size 256 totalling 23.2KiB
2020-03-31 17:27:47.642017: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 512 totalling 1.5KiB
2020-03-31 17:27:47.642032: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 768 totalling 2.2KiB
2020-03-31 17:27:47.642047: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1024 totalling 1.0KiB
2020-03-31 17:27:47.642063: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 31 Chunks of size 1280 totalling 38.8KiB
2020-03-31 17:27:47.642078: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 1792 totalling 7.0KiB
2020-03-31 17:27:47.642093: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 8192 totalling 8.0KiB
2020-03-31 17:27:47.642109: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 16384 totalling 64.0KiB
2020-03-31 17:27:47.642124: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 32768 totalling 32.0KiB
2020-03-31 17:27:47.642140: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 8 Chunks of size 76800 totalling 600.0KiB
2020-03-31 17:27:47.642156: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 124416 totalling 121.5KiB
2020-03-31 17:27:47.642172: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 153600 totalling 150.0KiB
2020-03-31 17:27:47.642187: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 10 Chunks of size 360192 totalling 3.43MiB
2020-03-31 17:27:47.642203: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 363776 totalling 710.5KiB
2020-03-31 17:27:47.642219: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 470016 totalling 459.0KiB
2020-03-31 17:27:47.642234: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 568320 totalling 555.0KiB
2020-03-31 17:27:47.642249: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 8388608 totalling 8.00MiB
2020-03-31 17:27:47.642266: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 12432128 totalling 11.86MiB
2020-03-31 17:27:47.642282: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 15349760 totalling 14.64MiB
2020-03-31 17:27:47.642297: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 24482816 totalling 23.35MiB
2020-03-31 17:27:47.642313: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 24990208 totalling 23.83MiB
2020-03-31 17:27:47.642328: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 2448264192 totalling 2.28GiB
2020-03-31 17:27:47.642343: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 2.37GiB
2020-03-31 17:27:47.642361: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit: 11295313101
InUse: 2540349952
MaxInUse: 2540474368
NumAllocs: 1316
MaxAllocSize: 2448264192
2020-03-31 17:27:47.642408: W tensorflow/core/common_runtime/bfc_allocator.cc:271] *************************_______________________________________________________________________
2020-03-31 17:27:47.642457: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at concat_op.cc:153 : Resource exhausted: OOM when allocating tensor with shape[2040220,303] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
File "/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
return fn(*args)
File "/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[2040220,303] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node layer2/concat}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "run.py", line 260, in
results = sess.run(fetches, feed_dict=feed_dict)
File "/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[2040220,303] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node layer2/concat (defined at /home/objloc/OBJLOC/Point-GNN/models/gnn.py:352) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[node ArgMax (defined at run.py:142) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Caused by op 'layer2/concat', defined at:
File "run.py", line 140, in
t_is_training)
File "/home/objloc/OBJLOC/Point-GNN/models/models.py", line 146, in predict
**layer_kwargs)
File "/home/objloc/OBJLOC/Point-GNN/models/gnn.py", line 352, in apply_regular
axis=-1)
File "/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py", line 180, in wrapper
return target(*args, **kwargs)
File "/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py", line 1256, in concat
return gen_array_ops.concat_v2(values=values, axis=axis, name=name)
File "/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 1149, in concat_v2
"ConcatV2", values=values, axis=axis, name=name)
File "/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op
op_def=op_def)
File "/home/objloc/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 1801, in init
self._traceback = tf_stack.extract_stack()
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[2040220,303] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node layer2/concat (defined at /home/objloc/OBJLOC/Point-GNN/models/gnn.py:352) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[node ArgMax (defined at run.py:142) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
71%|███████████████████████████████████████████████████████████████████▊ | 15/21 [01:18<00:31, 5.25s/it]
(py37) objloc@deeplearning:~/OBJLOC/Point-GNN$
`