Anyone training the code successful? about fcis HOT 14 CLOSED

Please do not to forget update the mxnet by
git checkout 62ecb60

from fcis.

My suggestion for this github :
Because mxnet and cuda is developing very rapidly. You need to update your cuda version to have this code run well. In my testing, only cuda 8 (0.64) is compatible with mxnet right now..

from fcis.

Called with argument: Namespace(cfg='experiments/fcis/cfgs/resnet_v1_101_coco_fcis_end2end_ohem.yaml', frequent=100)
{'BINARY_THRESH': 0.4,
'CLASS_AGNOSTIC': True,
'MASK_SIZE': 21,
'MXNET_VERSION': 'mxnet',
'SCALES': [(600, 1000)],
'TEST': {'BATCH_IMAGES': 1,
'CXX_PROPOSAL': False,
'HAS_RPN': True,
'ITER': 2,
'MASK_MERGE_THRESH': 0.5,
'MIN_DROP_SIZE': 2,
'NMS': 0.3,
'PROPOSAL_MIN_SIZE': 2,
'PROPOSAL_NMS_THRESH': 0.7,
'PROPOSAL_POST_NMS_TOP_N': 2000,
'PROPOSAL_PRE_NMS_TOP_N': 20000,
'RPN_MIN_SIZE': 2,
'RPN_NMS_THRESH': 0.7,
'RPN_POST_NMS_TOP_N': 300,
'RPN_PRE_NMS_TOP_N': 6000,
'USE_GPU_MASK_MERGE': True,
'USE_MASK_MERGE': True,
'test_epoch': 8},
'TRAIN': {'ASPECT_GROUPING': True,
'BATCH_IMAGES': 1,
'BATCH_ROIS': -1,
'BATCH_ROIS_OHEM': 128,
'BBOX_MEANS': [0.0, 0.0, 0.0, 0.0],
'BBOX_NORMALIZATION_PRECOMPUTED': True,
'BBOX_REGRESSION_THRESH': 0.5,
'BBOX_STDS': [0.2, 0.2, 0.5, 0.5],
'BBOX_WEIGHTS': array([ 1., 1., 1., 1.]),
'BG_THRESH_HI': 0.5,
'BG_THRESH_LO': 0,
'BINARY_THRESH': 0.4,
'CONVNEW3': True,
'CXX_PROPOSAL': False,
'ENABLE_OHEM': True,
'END2END': True,
'FG_FRACTION': 0.25,
'FG_THRESH': 0.5,
'FLIP': True,
'GAP_SELECT_FROM_ALL': False,
'IGNORE_GAP': False,
'LOSS_WEIGHT': [1.0, 10.0, 1.0],
'RESUME': False,
'RPN_ALLOWED_BORDER': 0,
'RPN_BATCH_SIZE': 256,
'RPN_BBOX_WEIGHTS': [1.0, 1.0, 1.0, 1.0],
'RPN_CLOBBER_POSITIVES': False,
'RPN_FG_FRACTION': 0.5,
'RPN_MIN_SIZE': 2,
'RPN_NEGATIVE_OVERLAP': 0.3,
'RPN_NMS_THRESH': 0.7,
'RPN_POSITIVE_OVERLAP': 0.7,
'RPN_POSITIVE_WEIGHT': -1.0,
'RPN_POST_NMS_TOP_N': 300,
'RPN_PRE_NMS_TOP_N': 6000,
'SHUFFLE': True,
'begin_epoch': 0,
'end_epoch': 8,
'lr': 0.0005,
'lr_step': '5.33',
'model_prefix': 'e2e',
'momentum': 0.9,
'warmup': True,
'warmup_lr': 5e-05,
'warmup_step': 250,
'wd': 0.0005},
'dataset': {'NUM_CLASSES': 81,
'dataset': 'coco',
'dataset_path': './data/coco',
'image_set': 'train2014+valminusminival2014',
'proposal': 'rpn',
'root_path': './data',
'test_image_set': 'test-dev2015'},
'default': {'frequent': 100, 'kvstore': 'device'},
'gpus': '0,1,2,3',
'network': {'ANCHOR_RATIOS': [0.5, 1, 2],
'ANCHOR_SCALES': [4, 8, 16, 32],
'FIXED_PARAMS': ['conv1',
'bn_conv1',
'res2',
'bn2',
'gamma',
'beta'],
'FIXED_PARAMS_SHARED': ['conv1',
'bn_conv1',
'res2',
'bn2',
'res3',
'bn3',
'res4',
'bn4',
'gamma',
'beta'],
'IMAGE_STRIDE': 0,
'NUM_ANCHORS': 12,
'PIXEL_MEANS': array([ 103.06, 115.9 , 123.15]),
'RCNN_FEAT_STRIDE': 16,
'RPN_FEAT_STRIDE': 16,
'pretrained': './model/pretrained_model/resnet_v1_101',
'pretrained_epoch': 0},
'output_path': './output/fcis/coco',
'symbol': 'resnet_v1_101_fcis'}
loading annotations into memory...
Done (t=14.96s)
creating index...
index created!
num_images 82783
prepare gt_sdsdb using 92.4870698452 seconds
generate cache_seg_inst using 424.751945019 seconds
append flipped images to roidb
loading annotations into memory...
Done (t=6.46s)
creating index...
index created!
num_images 35504
prepare gt_sdsdb using 37.5835440159 seconds
generate cache_seg_inst using 29.3039870262 seconds
append flipped images to roidb
filtered 2042 roidb entries: 236574 -> 234532
providing maximum shape [('data', (1, 3, 600, 1000)), ('gt_boxes', (1, 100, 5)), ('gt_masks', (1, 100, 600, 1000))] [('proposal_label', (1, 28728)), ('proposal_bbox_target', (1, 48, 38, 63)), ('proposal_bbox_weight', (1, 48, 38, 63))]
data shape:
{'data': (1L, 3L, 600L, 800L),
'gt_boxes': (1L, 1L, 5L),
'gt_masks': (1L, 600L, 800L),
'im_info': (1L, 3L),
'proposal_bbox_target': (1L, 48L, 38L, 50L),
'proposal_bbox_weight': (1L, 48L, 38L, 50L),
'proposal_label': (1L, 22800L)}
lr 0.0005 lr_epoch_diff [5.33] lr_iters [312513]
Segmentation fault

from fcis.

[08:28:57] /opt/data/penggao/mxnet/mxnet/dmlc-core/include/dmlc/logging.h:304: [08:28:57] /opt/data/penggao/mxnet/mxnet/mshadow/mshadow/./././dot_engine-inl.h:532: Check failed: err == CUBLAS_STATUS_SUCCESS (13 vs. 0) Cublas: Sgemm fail

Stack trace returned 9 entries:
[bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet-0.9.5-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7f7a2bc8ba5c]
[bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet-0.9.5-py2.7.egg/mxnet/libmxnet.so(_ZN7mshadow4expr10BLASEngineINS_3gpuEfE4gemmEPNS_6StreamIS2_EEbbiiifPKfiS8_ifPfi+0x15b) [0x7f7a2d6ca95b]
[bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet-0.9.5-py2.7.egg/mxnet/libmxnet.so(ZN5mxnet2op13ConvolutionOpIN7mshadow3gpuEfE7ForwardERKNS_9OpContextERKSt6vectorINS_5TBlobESaIS9_EERKS8_INS_9OpReqTypeESaISE_EESD_SD+0xbb9) [0x7f7a2d6daf89]
[bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet-0.9.5-py2.7.egg/mxnet/libmxnet.so(+0x10b5608) [0x7f7a2c6f1608]
[bt] (4) /usr/local/lib/python2.7/dist-packages/mxnet-0.9.5-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x8c) [0x7f7a2c6dc3ec]
[bt] (5) /usr/local/lib/python2.7/dist-packages/mxnet-0.9.5-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvvEZZN5mxnet6engine23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlvE_E9_M_invokeERKSt9_Any_data+0x58) [0x7f7a2c6df168]
[bt] (6) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb1a60) [0x7f7a471f9a60]
[bt] (7) /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184) [0x7f7a4994d184]
[bt] (8) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f7a4967abed]

[08:28:57] /opt/data/penggao/mxnet/mxnet/dmlc-core/include/dmlc/logging.h:304: [08:28:57] src/engine/./threaded_engine.h:329: [08:28:57] /opt/data/penggao/mxnet/mxnet/mshadow/mshadow/./././dot_engine-inl.h:532: Check failed: err == CUBLAS_STATUS_SUCCESS (13 vs. 0) Cublas: Sgemm fail

Stack trace returned 9 entries:
[bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet-0.9.5-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7f7a2bc8ba5c]
[bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet-0.9.5-py2.7.egg/mxnet/libmxnet.so(_ZN7mshadow4expr10BLASEngineINS_3gpuEfE4gemmEPNS_6StreamIS2_EEbbiiifPKfiS8_ifPfi+0x15b) [0x7f7a2d6ca95b]
[bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet-0.9.5-py2.7.egg/mxnet/libmxnet.so(ZN5mxnet2op13ConvolutionOpIN7mshadow3gpuEfE7ForwardERKNS_9OpContextERKSt6vectorINS_5TBlobESaIS9_EERKS8_INS_9OpReqTypeESaISE_EESD_SD+0xbb9) [0x7f7a2d6daf89]
[bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet-0.9.5-py2.7.egg/mxnet/libmxnet.so(+0x10b5608) [0x7f7a2c6f1608]
[bt] (4) /usr/local/lib/python2.7/dist-packages/mxnet-0.9.5-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x8c) [0x7f7a2c6dc3ec]
[bt] (5) /usr/local/lib/python2.7/dist-packages/mxnet-0.9.5-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvvEZZN5mxnet6engine23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlvE_E9_M_invokeERKSt9_Any_data+0x58) [0x7f7a2c6df168]
[bt] (6) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb1a60) [0x7f7a471f9a60]
[bt] (7) /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184) [0x7f7a4994d184]
[bt] (8) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f7a4967abed]

An fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging.

Stack trace returned 6 entries:
[bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet-0.9.5-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7f7a2bc8ba5c]
[bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet-0.9.5-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x376) [0x7f7a2c6dc6d6]
[bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet-0.9.5-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvvEZZN5mxnet6engine23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlvE_E9_M_invokeERKSt9_Any_data+0x58) [0x7f7a2c6df168]
[bt] (3) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb1a60) [0x7f7a471f9a60]
[bt] (4) /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184) [0x7f7a4994d184]
[bt] (5) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f7a4967abed]

terminate called after throwing an instance of 'dmlc::Error'
what(): [08:28:57] src/engine/./threaded_engine.h:329: [08:28:57] /opt/data/penggao/mxnet/mxnet/mshadow/mshadow/./././dot_engine-inl.h:532: Check failed: err == CUBLAS_STATUS_SUCCESS (13 vs. 0) Cublas: Sgemm fail

Stack trace returned 9 entries:
[bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet-0.9.5-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7f7a2bc8ba5c]
[bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet-0.9.5-py2.7.egg/mxnet/libmxnet.so(_ZN7mshadow4expr10BLASEngineINS_3gpuEfE4gemmEPNS_6StreamIS2_EEbbiiifPKfiS8_ifPfi+0x15b) [0x7f7a2d6ca95b]
[bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet-0.9.5-py2.7.egg/mxnet/libmxnet.so(ZN5mxnet2op13ConvolutionOpIN7mshadow3gpuEfE7ForwardERKNS_9OpContextERKSt6vectorINS_5TBlobESaIS9_EERKS8_INS_9OpReqTypeESaISE_EESD_SD+0xbb9) [0x7f7a2d6daf89]
[bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet-0.9.5-py2.7.egg/mxnet/libmxnet.so(+0x10b5608) [0x7f7a2c6f1608]
[bt] (4) /usr/local/lib/python2.7/dist-packages/mxnet-0.9.5-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x8c) [0x7f7a2c6dc3ec]
[bt] (5) /usr/local/lib/python2.7/dist-packages/mxnet-0.9.5-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvvEZZN5mxnet6engine23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlvE_E9_M_invokeERKSt9_Any_data+0x58) [0x7f7a2c6df168]
[bt] (6) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb1a60) [0x7f7a471f9a60]
[bt] (7) /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184) [0x7f7a4994d184]
[bt] (8) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f7a4967abed]

An fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging.

Stack trace returned 6 entries:
[bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet-0.9.5-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7f7a2bc8ba5c]
[bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet-0.9.5-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x376) [0x7f7a2c6dc6d6]
[bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet-0.9.5-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvvEZZN5mxnet6engine23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlvE_E9_M_invokeERKSt9_Any_data+0x58) [0x7f7a2c6df168]
[bt] (3) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb1a60) [0x7f7a471f9a60]
[bt] (4) /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184) [0x7f7a4994d184]
[bt] (5) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f7a4967abed]

from fcis.

Do you copy all files under operator_cxx to contrib?

from fcis.

lr 0.0005 lr_epoch_diff [5.33] lr_iters [312513]

Everything goes on well right now

Epoch[0] Batch [100] Speed: 4.21 samples/sec Train-RPNAcc=0.818688, RPNLogLoss=0.420959, RPNL1Loss=0.177084, FCISAcc=0.538366, FCISAccFG=0.002557, FCISLogLoss=4.356684, FCISL1Loss=0.085457, FCISMaskLoss=0.696656,
Epoch[0] Batch [200] Speed: 5.85 samples/sec Train-RPNAcc=0.837677, RPNLogLoss=0.385342, RPNL1Loss=0.183377, FCISAcc=0.684439, FCISAccFG=0.001307, FCISLogLoss=3.720182, FCISL1Loss=0.083474, FCISMaskLoss=0.720103,

from fcis.

@gaopeng-eugene did you ever encounter an error with h5py? if so, how does one go about solving it? thanks!
'''
Epoch[0] Batch [27600] Speed: 1.34 samples/sec Train-RPNAcc=0.928724, RPNLogLoss=0.175137, RPNL1Loss=0.112139, FCISAcc=0.719911, FCISAccFG=0.203269, FCISLogLoss=1.146860, FCISL1Loss=0.108758, FCISMaskLoss=0.417763,
Epoch[0] Batch [27700] Speed: 1.33 samples/sec Train-RPNAcc=0.928798, RPNLogLoss=0.174975, RPNL1Loss=0.112081, FCISAcc=0.719898, FCISAccFG=0.203784, FCISLogLoss=1.146106, FCISL1Loss=0.108755, FCISMaskLoss=0.417524,
Exception in thread Thread-8:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "experiments/fcis/../../fcis/../lib/utils/PrefetchingIter.py", line 60, in prefetch_func
self.next_batch[i] = self.iters[i].next()
File "experiments/fcis/../../fcis/core/loader.py", line 99, in next
self.get_batch_parallel()
File "experiments/fcis/../../fcis/core/loader.py", line 161, in get_batch_parallel
rst = self.parfetch(roidb)
File "experiments/fcis/../../fcis/core/loader.py", line 183, in parfetch
gt_masks = get_gt_masks(roidb[0]['cache_seg_inst'], data['im_info'][0,:2].astype('int'))
File "experiments/fcis/../../fcis/../lib/mask/mask_transform.py", line 25, in get_gt_masks
gt_masks = hkl.load(gt_mask_file)
File "/usr/local/lib/python2.7/dist-packages/hickle.py", line 616, in load
h5f = file_opener(fileobj)
File "/usr/local/lib/python2.7/dist-packages/hickle.py", line 154, in file_opener
h5f = h5.File(filename, mode)
File "/usr/local/lib/python2.7/dist-packages/h5py/_hl/files.py", line 271, in init
fid = make_fid(name, mode, userblock_size, fapl, swmr=swmr)
File "/usr/local/lib/python2.7/dist-packages/h5py/_hl/files.py", line 101, in make_fid
fid = h5f.open(name, flags, fapl=fapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/tmp/pip-nCYoKW-build/h5py/_objects.c:2840)
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/tmp/pip-nCYoKW-build/h5py/_objects.c:2798)
File "h5py/h5f.pyx", line 78, in h5py.h5f.open (/tmp/pip-nCYoKW-build/h5py/h5f.c:2117)
IOError: Unable to open file (File signature not found)
'''

from fcis.

@encodingwaddles I met the same error, did u solve it?

from fcis.

@scholltan no not yet ....

from fcis.

@oh233 could you solve this problem? thanks!

from fcis.

It seems to be an IO error. hickle cannot load the mask and give this error

from fcis.

@oh233 this error occurred in both ubuntu 14.04 and 16.06 after thousands batches training, could u give some advise on how to fix it? thank you!

from fcis.

Add print gt_mask_file before this line https://github.com/msracver/FCIS/blob/master/lib/mask/mask_transform.py#L25. When the error occurs, check whether the file exists.

from fcis.

thanks for the tip @oh233 ! the coco dataset did have some empty images (file size 0), once those were removed. training works :D

from fcis.