fmow / baseline Goto Github PK

Hi guys,

when i run the code, i get the OOM error at :

model.fit_generator(train_datagen,
steps_per_epoch=(len(trainData) / self.params.batch_size_cnn + 1),
epochs=self.params.cnn_epochs, callbacks=callbacks_list)

It is running on Windows with Tensorflow backend. Even when i changed the batch_size to very small, it still has this problem.
Thanks.
the whole information like this:
OOM when allocating tensor with shape[32,14,14,1680]

 [[Node: concatenate_45/concat = ConcatV2[N=2, T=DT_FLOAT, Tidx=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"](concatenate_44/concat, conv2d_93/convolution, concatenate_45/concat/axis)]]

 [[Node: metrics/acc/Mean/_10395 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_93784_metrics/acc/Mean", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

Caused by op 'concatenate_45/concat', defined at:

File "C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\Extensions\Microsoft\Python Tools for Visual Studio\2.2\visualstudio_py_launcher.py", line 78, in

vspd.debug(filename, port_num, debug_id, debug_options, run_as)

File "C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\Extensions\Microsoft\Python Tools for Visual Studio\2.2\visualstudio_py_debugger.py", line 2483, in debug

exec_file(file, globals_obj)

File "C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\Extensions\Microsoft\Python Tools for Visual Studio\2.2\visualstudio_py_util.py", line 111, in exec_file

exec_code(code, file, global_variables)

File "C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\Extensions\Microsoft\Python Tools for Visual Studio\2.2\visualstudio_py_util.py", line 87, in exec_code

exec(code_obj, global_variables)

File "G:\programs\baseline-master\code\runBaseline.py", line 40, in

main("-prepare ")

File "G:\programs\baseline-master\code\runBaseline.py", line 29, in main

baseline.train_cnn()

File "G:\programs\baseline-master\code\fmowBaseline.py", line 93, in train_cnn

model = get_cnn_model(self.params)

File "G:\programs\baseline-master\code\data_ml_functions\mlFunctions.py", line 46, in get_cnn_model

baseModel = densenet.DenseNetImageNet161(input_shape=(params.target_img_size[0], params.target_img_size[1], params.num_channels), include_top=False, input_tensor=input_tensor)

File "G:\programs\baseline-master\code\data_ml_functions\DenseNet\densenet.py", line 446, in DenseNetImageNet161

classes=classes, activation=activation)

File "G:\programs\baseline-master\code\data_ml_functions\DenseNet\densenet.py", line 164, in DenseNet

dropout_rate, weight_decay, subsample_initial_block, activation)

File "G:\programs\baseline-master\code\data_ml_functions\DenseNet\densenet.py", line 639, in __create_dense_net

dropout_rate=dropout_rate, weight_decay=weight_decay)

File "G:\programs\baseline-master\code\data_ml_functions\DenseNet\densenet.py", line 502, in __dense_block

x = concatenate([x, cb], axis=concat_axis)

File "C:\Users\VS 02\Anaconda3\lib\site-packages\keras\layers\merge.py", line 627, in concatenate

return Concatenate(axis=axis, **kwargs)(inputs)

File "C:\Users\VS 02\Anaconda3\lib\site-packages\keras\engine\topology.py", line 603, in call

output = self.call(inputs, **kwargs)

File "C:\Users\VS 02\Anaconda3\lib\site-packages\keras\layers\merge.py", line 347, in call

return K.concatenate(inputs, axis=self.axis)

File "C:\Users\VS 02\Anaconda3\lib\site-packages\keras\backend\tensorflow_backend.py", line 1778, in concatenate

return tf.concat([to_dense(x) for x in tensors], axis)

File "C:\Users\VS 02\Anaconda3\lib\site-packages\tensorflow\python\ops\array_ops.py", line 1066, in concat

name=name)

File "C:\Users\VS 02\Anaconda3\lib\site-packages\tensorflow\python\ops\gen_array_ops.py", line 493, in _concat_v2

name=name)

File "C:\Users\VS 02\Anaconda3\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 767, in apply_op

op_def=op_def)

File "C:\Users\VS 02\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 2630, in create_op

original_op=self._default_original_op, op_def=op_def)

File "C:\Users\VS 02\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 1204, in init

self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[32,14,14,1680]

 [[Node: concatenate_45/concat = ConcatV2[N=2, T=DT_FLOAT, Tidx=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:0"](concatenate_44/concat, conv2d_93/convolution, concatenate_45/concat/axis)]]

 [[Node: metrics/acc/Mean/_10395 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_93784_metrics/acc/Mean", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

Hi Guys,

Can you please point me in the right direction about this error I m getting while running cnn only mode.