Extrinsic/Intrisic Camera Input about srt HOT 4 CLOSED

Hi! As defined in the paper in Eqs. 4 and 5, pose information is added to the input by concatenating encodings of camera position and viewing direction to every pixel of the image. This information is given by camera_pos and rays.

The camera matrices are used internally by the data loaders to compute this information. You can see how this happens for the NMR dataset here. For Multishapenet, this job is done by the Sunds library.

from srt.

Hello, just to add another question to this, how does the model handle inputs without pose information? It seems that the encoder components must take camera_pos and ray information to create a ray encoding for the transformer.

from srt.

Yes, I didn't experiment with unposed imagery. From what I understand, in their experiments with UpSRT, the authors simply didn't concatenate the pose information to the encoder input. So you would just skip this step.

from srt.

I see, thank you for the answer !

from srt.