Longer frames issues. about video-chatgpt HOT 4 CLOSED

Hi @wang9danzuishuai,

Thank you for your interest in our work. Currently the codebase is configured to use 100 frames only from a video. Given 100 video frames, features are extracted using pretrained CLIP model and the -2nd layer outputs are considered. This gives us a tensor of shape t, s, c where t=100, s=256 is the number of spatial tokens and c=1024 is the feature dimensions.

We construct spatiotemporal features as by taking average across t and s dimensions.

temporal features -> average(t, s, c, dim=s) -> (t, c) -> (100, 1025)
spatial features -> average(t, s, c, dim=t) -> (s, c) -> (256 x 1024)

all features = 100 + 256 -> (356 x 1024)

As can be seen from the above explanation, the total number of tokens for 200 frames would be 200 + 256 = 456 and so on.

And in order to implement this change in the code base, we have set the number of total features to 456 at a few places. For example changing 356 to 456 here and changing the padding size accordingly here. Similarly, during training the changes should be made at here as well.

Please let me know if it solves the issue. Thanks

from video-chatgpt.

Thanks so much. It works right now. It's very kind of you to help us with this.
Wish you happiness every day!

from video-chatgpt.

Thanks so much. It works right now. It's very kind of you to help us with this. Wish you happiness every day!

Hi @wang9danzuishuai , I'm interesting about your attempt, how does the frame number adjustment work in your tests? Did it improve on understanding longer videos?

from video-chatgpt.

@Kratos-Wen Hi, I just followed the instructions given by @mmaaz60, and changed num_frm from 100 to 200.
But this change seems had no improvement on understanding longer videos.
If a video is about 30 minutes, 100 frames and 200 frames could have no difference.
I guess changing the frame extracting method may work well such as extracting 1 frame per second in one video.

from video-chatgpt.