unsky / deformable-convnets-caffe Goto Github PK

I added cpp cu hpp and proto definition to my caffe, compile is ok. But when I ran the MNIST example, I got this error.

F0113 09:54:20.415395  2813 layer_factory.hpp:81] Check failed: registry.count(type) == 1 (0 vs. 1) Unknown layer type: DeformableConvolution (known types: AbsVal, Accuracy, ArgMax, BNLL, BatchNorm, BatchReindex, Bias, Concat, ContrastiveLoss, Convolution, Crop, Data, Deconvolution, Dropout, DummyData, ELU, Eltwise, Embed, EuclideanLoss, Exp, Filter, Flatten, FocalLoss, HDF5Data, HDF5Output, HingeLoss, Im2col, ImageData, InfogainLoss, InnerProduct, Input, LRN, LSTM, LSTMUnit, Log, MVN, MemoryData, MultinomialLogisticLoss, PReLU, Parameter, Pooling, Power, Python, RNN, ROIPooling, ReLU, Reduction, Reshape, SPP, Scale, Sigmoid, SigmoidCrossEntropyLoss, Silence, Slice, SmoothL1Loss, Softmax, SoftmaxWithLoss, Split, TanH, Threshold, Tile, TripletLoss, WindowData)
*** Check failure stack trace: ***
    @     0x7fcc531445cd  google::LogMessage::Fail()
    @     0x7fcc53146433  google::LogMessage::SendToLog()
    @     0x7fcc5314415b  google::LogMessage::Flush()
    @     0x7fcc53146e1e  google::LogMessageFatal::~LogMessageFatal()
    @     0x7fcc53947e6c  caffe::Net<>::Init()
    @     0x7fcc539495ae  caffe::Net<>::Net()
    @     0x7fcc538d6775  caffe::Solver<>::InitTrainNet()
    @     0x7fcc538d7b65  caffe::Solver<>::Init()
    @     0x7fcc538d7e7f  caffe::Solver<>::Solver()
    @     0x7fcc538f03f1  caffe::Creator_SGDSolver<>()
    @           0x40ada8  train()
    @           0x4075a8  main
    @     0x7fcc51ad7830  __libc_start_main
    @           0x407e79  _start
    @              (nil)  (unknown)
Aborted (core dumped)

thx for your great work, in readme:
bottom[1] (offset): (batch_size, deformable_group * kernel[0] * kernel[1]*2, height, width)

according to your code (im2col) and the pytorch version, i think the last 2 dims of bottom[1],should be [height_output,width_output].

Hi，thank you for your contributions.I want to use your faster rcnn，can you please tell me where are the caffemodels for train and test?

I have used the deformable conv in training my own model, but the offsets predicted in several iters reach over 1000. Solution provided by MSRA of paddding has been tried , it doesn't work. There maybe something wrong in the code, does anyone solve the problem?

Hi, when I compile the matcaffe, I face an issue:

/home/lxk/code/tracking/Deformable-ConvNets/matlab/+caffe/private/caffe_.cpp:45:40: error: template argument 1 is invalid
/home/lxk/code/tracking/Deformable-ConvNets/matlab/+caffe/private/caffe_.cpp:45:40: error: template argument 2 is invalid
/home/lxk/code/tracking/Deformable-ConvNets/matlab/+caffe/private/caffe_.cpp:45:50: error: invalid type in declaration before ‘;’ token
/home/lxk/code/tracking/Deformable-ConvNets/matlab/+caffe/private/caffe_.cpp:46:37: error: template argument 1 is invalid
/home/lxk/code/tracking/Deformable-ConvNets/matlab/+caffe/private/caffe_.cpp:46:37: error: template argument 2 is invalid
/home/lxk/code/tracking/Deformable-ConvNets/matlab/+caffe/private/caffe_.cpp:46:44: error: invalid type in declaration before ‘;’ token
/home/lxk/code/tracking/Deformable-ConvNets/matlab/+caffe/private/caffe_.cpp:174:66: error: template argument 1 is invalid
/home/lxk/code/tracking/Deformable-ConvNets/matlab/+caffe/private/caffe_.cpp:174:66: error: template argument 2 is invalid
/home/lxk/code/tracking/Deformable-ConvNets/matlab/+caffe/private/caffe_.cpp: In function ‘mxArray* ptr_vec_to_handle_vec(const int&)’:
/home/lxk/code/tracking/Deformable-ConvNets/matlab/+caffe/private/caffe_.cpp:175:57: error: request for member ‘size’ in ‘ptr_vec’, which is of non-class type ‘const int’
/home/lxk/code/tracking/Deformable-ConvNets/matlab/+caffe/private/caffe_.cpp:176:31: error: request for member ‘size’ in ‘ptr_vec’, which is of non-class type ‘const int’
/home/lxk/code/tracking/Deformable-ConvNets/matlab/+caffe/private/caffe_.cpp:177:27: error: invalid types ‘const int[int]’ for array subscript
/home/lxk/code/tracking/Deformable-ConvNets/matlab/+caffe/private/caffe_.cpp: In function ‘void get_solver(int, mxArray**, int, const mxArray**)’:
/home/lxk/code/tracking/Deformable-ConvNets/matlab/+caffe/private/caffe_.cpp:193:3: error: reference to ‘shared_ptr’ is ambiguous

Could you tell me the solution? Thank you.

Hi,
we just need add little deformable convolution at the end of the net?

After I train the network you provide and obtain the Resnet_dcn.caffemodel, I compiled the matlab interface and try to load the caffemodel. However, an error appeared (caffe.layerparameter has no field named deformable-cnvolution parameter). I do not know whether it is caused by the incorrect compilation of matcaffe or the loss of the newly added deformable convolutional layer in caffe_.mexa64. Could you help me?

You mentioned that
use faster rcnn
Train &test:
./experiments/scripts/faster_rcnn_end2end.sh 0 ResNet50 pascal_voc
However, an error appeared, File not found: /home/kyl/Deformable-ConvNets-caffe/deformable_conv_faster-rcnn/output/faster_rcnn_end2end/voc_2007_trainval/ResNet50_dcn_iter_30000.caffemodel.
Could you tell me how to solve this problem or where to download this pretrained caffemodel.

How many iterations does the model need to reproduce the results?
I use the same hyper-parameters as for the baseline faster rcnn w/o deform conv., the final loss is much higher than the baseline, and the mAP is not satisfactory.

My setting is max_iter = 30k (with 4 gpus, iter_size 2) . base_lr = 0.001, gamma = 0.1, stepsize = 20k
The baseline mAP is 77.9 while the deform conv. one is 73.x. The backbone model is ResNet-101

check failed: error==cudaSuccess <9 vs 0> invalid configuration argument

the log said this error happened because of the deformable_im2col.cu ,so can you help me?

Thanks for the DCN impl. in caffe. I am curious about the performance of the baseline model w/o deform_conv. Besides, have you tried more complex model such as ResNet-101?

1.既然offset的输出是72那后面的dec卷积是不是应该带group=4呢？为什么要用group卷积呢？
2.为什么输入dec的feature map是conv1的不是pool1的呢？做了pooling之后尺寸不是对不上了吗？

Could you please share me the trained caffemodel by email?

Hi,
Thanks for your sharing. I try to use the codes in my caffe, adding all the words you mentioned in Readme, unfortunately, when i run the LeNet network, it failed and encounter the error as the title, I have tried many times, still failed... when i make your caffe, and run the LeNet network, it worked. Now, i am very confused, could you give me some advise? I will be very appreciate it.

I want to know how to initialize the weight param of “offset” conv layer , 0 or xavier?
What initial lr we should use, small or big?
Thanks.

In mnist network given,why used dilation convolution in offset layer? For the offset have a lager search area in feature map or not?

Hi, my friend , according to your work ,I modify the yolo to deform-yolo, but the loss always are 3~4 whatever what I adjust lr and batchsize and so on ,can you give me some advice anout how to train effectively, Thank you very much

Hi,
As you posted in ReadMe: when using Deformable_CNN, it get 78.1 on VOC 2007 test set, I am wondering the results without Deformabel_CNN, whether you could post publicly for clear comparison
Best

I tried to train a classification net using dcn (ResNet-50 with dcn in stage 5).
I view feature maps of the deformable conv layer and find that it cannot learn any useful things. Actually the content on each feature map is either 0 or a fixed constant.
I am wondering if there is a problem with the code or dcn is hard to train on the classification task.
@unsky Do you have time to have a look at my classification network and maybe discuss this phenomenon with me？Thank you very much！
Here is my qq：819375556

Dear @unsky ,

May I know the reason why you introduce deformable_group parameter? How does it affect the performance? Should deformable_group be equal to group?

Thanks!
Kun

How can I use it ?
where is the model ?

thank you very much

感谢您的分享，有一些存疑的地方。
按照给出的脚本训练测试，准确率在10%，10000次的迭代，训练显示loss一直维持在87左右并不能收敛。
其次给出的offset的channel与defconv的group是如何对应的，这部分有点迷惑。

@unsky HI
按照READEME成功将deformable_conv嫁接到了多个自己的caffe中，如：py-R-FCN．

但在进行检验，即执行./experiments/scripts/faster_rcnn_end2end.sh 0 ResNet50 pascal_voc的时候，总会报如下错误：

F0626 16:18:14.308534 28297 net.cpp:784] Cannot copy param 0 weights from layer 'res5a_branch2b'; shape mismatch.  Source param shape is 512 512 3 3 (2359296); target param shape is 512 128 3 3 (589824). To learn this layer's parameters from scratch rather than copying from a saved net, rename the layer.
*** Check failure stack trace: ***

而在Deformable-ConvNets-caffe/deformable_conv_faster-rcnn下执行，就不会出现该错误．

同样的train.prototxt文件（用的是Deformable-ConvNets-caffe/deformable_conv_faster-rcnn下的文件），同样的预训练模型，为什么会出现不匹配的情况呢？？

Thank you for caffe version of DCN.
I have some questions

1. why offset & dec dimmension is 72 dim, 512 dim?
   I think offset is eaqual to bottom dimemsion....
2. why dec layer is needed?
3. In Resnet 50 proto, you apply RCNN layer. however, DCN is able to apply all the convolution layer (e.g., VGG16, conv1 ~ 5)
4. what is the role the layer of offset & dec

thank you

Creating symlink /home/songzhao/code/Deformable-ConvNets-caffe/caffe/python/caffe/_caffe.so -> /home/songzhao/code/Deformable-ConvNets-caffe/caffe/build/lib/_caffe.so
../lib/libcaffe.so.1.0.0-rc3: undefined reference to cv::String::allocate(unsigned long)' ../lib/libcaffe.so.1.0.0-rc3: undefined reference to cv::imread(cv::String const&, int)'
../lib/libcaffe.so.1.0.0-rc3: undefined reference to cv::String::deallocate()' ../lib/libcaffe.so.1.0.0-rc3: undefined reference to cv::imencode(cv::String const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator >&, std::vector<int, std::allocator > const&)'
collect2: error: ld returned 1 exit status
tools/CMakeFiles/upgrade_net_proto_binary.dir/build.make:134: recipe for target 'tools/upgrade_net_proto_binary' failed
make[2]: *** [tools/upgrade_net_proto_binary] Error 1
CMakeFiles/Makefile2:663: recipe for target 'tools/CMakeFiles/upgrade_net_proto_binary.dir/all' failed
make[1]: *** [tools/CMakeFiles/upgrade_net_proto_binary.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 93%] Built target pycaffe

Hi, thank your work , but I met an error when I compiled , the route of opencv is correct and there are not error if compiling commonly , Did you meet it once?

Hi @unsky

when i training the Deformable caffe on voc data, the error is below:

/home/chengshuai/test/Deformable-ConvNets-caffe-master/deformable_conv_faster-rcnn/tools/../lib/rpn/proposal_layer.py:180: RuntimeWarning: invalid value encountered in greater_equal
keep = np.where((ws >= min_size) & (hs >= min_size))[0]
./experiments/scripts/faster_rcnn_end2end.sh: line 57: 20194 ## Floating point exception(core dumped) ./tools/train_net.py --gpu ${GPU_ID} --solver models/${PT_DIR}/${NET}/faster_rcnn_end2end/solver.prototxt --weights data/imagenet_models/ResNet50.v2.caffemodel --imdb ${TRAIN_IMDB} --iters ${ITERS} --cfg experiments/cfgs/faster_rcnn_end2end.yml ${EXTRA_ARGS}

deformable_col2im_gpu函数里：
this->col_buffer_.shape(0)应该改为this->col_buffer_.count()
bottom_diff+n*this->input_offset_dim_应该改为‘bottom_diff+n*this->bottom_dim_’

hello, could I use Deformable-ConvNets-caffe by pycaffe to write the python conv code, and then generate the *.prototxt?

Actually, I am confused about the conv op for deformable ROI pooling. Can you provide one train.prototxt with deformable ROI pooling? Thanks.

I got message saying that
src/caffe/layers/deformable_conv_layer.cpp:213:16: error: ‘const class caffe::DeformableConvolutionParameter’ has no member named ‘group’

I am using cuda 8.0 & cudnn 5.1

Thanks for your incredible work.
I have a question about the shared networks . Why the RPN stage does not use deformable ConvNets in your prototxt?

Hi,have you test your results on faster res50 def?

And it will throw error when compiling.

At end of source: error: expected a "}"