Normalization in RGBDiff model about tsn-pytorch HOT 10 CLOSED

No modification. For Caffe style models, we do not perform the division either.

from tsn-pytorch.

and I found that:

self.input_mean = [0.485, 0.456, 0.406] + [0] * 3 * self.new_length
self.input_std = self.input_std + [np.mean(self.input_std) * 2] * 3 * self.new_length

It seems that for each stack, the way you normalize the first frame is different from the other frames. Is there any reason not to normalize in the same way?

Thank you.

from tsn-pytorch.

1. RGBdiff is calculated by subtracting consecutive frames. There is no need for subtracting means on frames beforehand. We mostly experimented with Caffe style models so the pixels are used as is. If you need Torch-style input you may add the scaling back.

2. These lines are not used as described in 1. But I think it is a good idea do a clean-up of the code to remove the confusing parts.

from tsn-pytorch.

That makes sense. Thank you for clarifying.
So what you're saying is that for the RGBDiff model, we only need to divide the frame values by std (no need for subtracting mean). Is that correct?

from tsn-pytorch.

Yes. Subtracted means will be removed in computing differences. So there is no need to do it beforehand.

from tsn-pytorch.

One more question about the std. Did you just divide by the original input_std for the RGBDiff model? Or you did some modification on it?

Thanks.

from tsn-pytorch.

I saw you have an argument keep_rgb for the RGBDiff model and set as False. What's the performance if you set keep_rgb as True? And how did you normalize that since you have both RGB and RGBDiff for inputs?
Thanks.

from tsn-pytorch.

It is just a simple trial without too much investigation, where we found marginal improvement. Perhaps its due to we only keep the first frame's RGB. We subtract means on the kept RGB. You are free to experiment with your own normalization.

from tsn-pytorch.

@yjxiong hi,
For generating the RGB diff images as the input to train the flow branch, you said we just directly subtract two consecutive frames. Does that mean, for example, I want to generate all the input images (frames, optical flow, rgb diff) from dense_flow code, I added the following in the dense_flow_gpu.cpp:

image_diff = capture_image - prev_image;
imencode(".jpg", image_diff, str_img);

Is this the correct way to generate the diff image? Do we need to set any bound (like optical flow one) to normalize the image_diff (-255255) to (0255) in dense_flow_gpu.cpp code?
After storing all the diff images, we run the RGB diff script to train the RGB diff branch, right?

Many thanks.

from tsn-pytorch.

@gwh0112
No. RGBDiff is generated on the fly during training. We can find the code for this cause in the repo.

from tsn-pytorch.