Crop Area differences between frames in sequence, may be related to aberrations in mask. about background-matting HOT 3 CLOSED

@Teravus Thanks a lot for trying out the code on your data and providing me with your valuable feedback.

Masks: Currently the network crops out a 512x512 region from the input frame based on the masksDL. If two frames have slight differences in masksDL prediction the aspect ratio of the crop will be different. However, the current testing code takes the output of the network, uncrop it and puts it back to the original frame resolution. Follow the image example and the same can be applied to videos to make sure the final output resolution/aspect ratio is the same as the input. You can write your own cropping function. You just need to make sure the input to the network is 512x512 centered around the human. While training I did data augmentation to allow the network some flexibility.

Reflection: Unfortunately strong reflection very close to the foreground is very hard to remove for the network. It learns to use color difference, segmentation mask, and human shape priors to predict the matte. If one of the response is very strong, the network latches on to that.

Future directions: All the constraints mentioned in the 'Capture Conditions' are certainly a drawback of the network. The main future directions are: allowing any camera movement (then it is truly green-screen), robust to auto-focus, auto-exposure, reflections, shadows.

Let me know if you have any additional questions

from background-matting.

Seems like the crop/resize happens even when the input/output sizes are the same.
I exported a 512x512 video to PNG (img and back). I also cropped the green background to 512x512 and ran it through.

The next picture is an overlay of the matting output (large version) and the deeplab mask applied to the original _img.png for the frame(the smaller version)

from background-matting.

I was able to disable the crop by commenting out lines 80-85 and adding;
back_img1 = back_img10
back_img2 = back_img20

This fixes the resize issue for me. Obviously, this would only work if your frame size matches the expected input to the network.

Thanks for your patience.

from background-matting.