Should the pad be zero when training? about caffe-vdsr HOT 10 CLOSED

Hi, there is no problem in the pad setting.
For a 41x41 patch, you should pad it to 42x42 or it will be 40x40 after convolution (kernel 3x3, stride 1).

You can generate training sample pairs like this:
61x61 low resolution image --> 41x41 high resolution image
Then you can set pad=0 for all 20 conv layers.
For test, just pad your input image like this in matlab

im = padarray(im,[border_size,border_size],'symmetric'); % 'symmetric' or other pad type

from caffe-vdsr.

Hi, zehao:
Thanks very much for the comment. I still have some confusions.

When pad =0 and kernel size 3x3, image size will reduce to (w-2, h-2) after every conv layer.
Thus, for a 41x41 patch, should I pad it to 43x43 rather than 42x42?

if not 'post' or 'pre' options are given, the im = padarray(im,[border_size,border_size],'symmetric');
will result in im_new.size = im_old.size + (2border_size, 2border_size).
So for a 41x41 high resolution image, I need 81x81 low resolution image for 20 conv layers?

The author of the VDSR paper also advised to pad zero rows/columns (pad=1).
However, I am very curious to what extent it infer errors when training for the pad=1 vs pad=0.

from caffe-vdsr.

Sorry, you are right. I make a low-level mistake.
Pad 1 will generate 43x43 result.
Actually, you can use pad 1, it will not influence results.

from caffe-vdsr.

However, I am very curious to what extent it infer errors when training for the pad=1 vs pad=0.

Hi, I don't understand what you mean.
The reason we set pad=1 in every kernel 3x3 stride 1 convolution layer is to ensure the size of output is the same as input.

layer {
  name: "conv1"
  type: "Convolution"
  bottom: "data"
  top: "conv1"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 0.1
  }
  convolution_param {
    num_output: 64
    kernel_size: 3
    stride: 1
    pad: 1
    weight_filler {
      type: "msra"
    }
    bias_filler {
      type: "constant"
    }
  }
}

So, assume you have a 256x256 image, after 20 convolution layers, the output is still 256x256 if you set pad=1 in every kernel 3x3 stride 1 convolution layers.

I think my code has no problem. Did you encounter any problem during training?

from caffe-vdsr.

Thank for the reply. The code is correct.

My problem is about the (data, label) pairs generation during the training.
41x41 patches are extracted from the origin images and down-sample and then up-sample images, acting as the label and data, respectively.
However, I think for each 41x41 label patch, it is related to surrounding 43x43 pixels in the origin images as the kernel size is 3x3. So, during train phase, if we take 41x41 low resolution patch as data and corresponding 41x41 high resolution patch as label, it will infer some errors (zeros are padded, which actual should be real pixels from origin images ). Shortly speaking, in my opinion, we should not pad zeros but taking pixels from the images.

Thus, it is advised to modify the pad to 0 (no pad) in the first conv layer (we still keep pad=1 for the rest layers) and the data patch size to be 43x43 (we keep the label patch size to 41x41 as other layers do not change the image size).

Hope I describe my meaning clearly.

Do you think it would benefit if employing such modification?

from caffe-vdsr.

Hi, I understand your meaning.
You can have a try. But I think the improvement maybe minimal.

from caffe-vdsr.

As you said, the improvement was indeed little:
I test the 4 512x512 bmp files in the Set14:

pad=1(origin)：
bi_psnr vs sr_psnr: 22.226748 22.131368
bi_psnr vs sr_psnr: 28.006033 28.586006
bi_psnr vs sr_psnr: 23.603370 23.923569
bi_psnr vs sr_psnr: 28.800017 29.824224

pad=0 (after revise)：
bi_psnr vs sr_psnr: 22.226748 22.266395
bi_psnr vs sr_psnr: 28.006033 28.650680
bi_psnr vs sr_psnr: 23.603370 24.025409
bi_psnr vs sr_psnr: 28.800017 29.648510

Th trial gained about 0.1db for the first three files, however became 0.2db worse for the last one.

from caffe-vdsr.

Another issue arise, I do not find too much quality difference for a 0.5db or even 1db psnr change when I displayed the result image on my computer.
I searched the Internet, and found the the PSNR did not quite reflect the subjective feeling. Other objective metrics such the PSNR-HVS-M was better. However I was surprised that most of the SR paper were likey to empoy PNSR or SSIM to assess the image quality.

Back to the caffe model, would it be hard to change the loss function from MSE to PSNR-HVS-M (more related to the MOS score)?

from caffe-vdsr.

Hi, it's true that PSNR can not reflect the subjective performances of images.
You can check my another repo about super-resolution. In section Perceptual Loss and GAN, there are several works which focu on imporve the subjective performance of super-resolution methods.

Sorry, I am not very familiar with PSNR-HVS-M. If it is derivable, I think it can be a loss function.

from caffe-vdsr.

hi, zehao:

Thanks very much for the comment~
I will check the repo

from caffe-vdsr.