Hello, and thanks for such a great contribution to the field of interleaved LMMs! This

Thank you <a class="user-mention notranslate" data-hovercard-type="user" data-hovercar

Multi-Image or Multi-Video Inference Example about vila HOT 2 OPEN

chancharikmitra commented on August 12, 2024

Multi-Image or Multi-Video Inference Example

from vila.

Comments (2)

DtYXs commented on August 12, 2024

Hello, I think in the VILA code, images are embed specifically in the locations of the <image> tokens.

VILA/llava/model/llava_arch.py

Lines 371 to 391 in 0085724

 for i in range(num_images + 1): 

 cur_new_input_embeds.append(cur_input_embeds_no_im[i]) 

 cur_new_labels.append(cur_labels_noim[i]) 

 if i < num_images: 

 cur_image_features = image_features[cur_image_idx] 

 cur_image_idx += 1 

 cur_new_input_embeds.append(cur_image_features) 

 cur_new_labels.append( 

 torch.full( 

 (cur_image_features.shape[0],), 

 IGNORE_INDEX, 

 device=cur_labels.device, 

 dtype=cur_labels.dtype, 

 ) 

 ) 

 cur_new_input_embeds = torch.cat(cur_new_input_embeds) 

 cur_new_labels = torch.cat(cur_new_labels) 

 new_input_embeds.append(cur_new_input_embeds) 

 new_labels.append(cur_new_labels)

from vila.

chancharikmitra commented on August 12, 2024

Thank you @DtYXs for the clarification about the <image> token and its placement! Given that, do you have any insights on why zero-shot performance on VILA-1.5-8b might be lower than what is being reported? Few-shot improvements are fantastic as advertised. Perhaps it is related to my concerns regarding masking and the conv_mode formatting. However, looking deeper at the eval scripts, I see that the conv_mode was passed directly - so 'llama_3' indeed would have been used.

from vila.

Recommend Projects

Multi-Image or Multi-Video Inference Example about vila HOT 2 OPEN

Comments (2)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	for i in range(num_images + 1):
	cur_new_input_embeds.append(cur_input_embeds_no_im[i])
	cur_new_labels.append(cur_labels_noim[i])
	if i < num_images:
	cur_image_features = image_features[cur_image_idx]
	cur_image_idx += 1
	cur_new_input_embeds.append(cur_image_features)
	cur_new_labels.append(
	torch.full(
	(cur_image_features.shape[0],),
	IGNORE_INDEX,
	device=cur_labels.device,
	dtype=cur_labels.dtype,
	)
	)

	cur_new_input_embeds = torch.cat(cur_new_input_embeds)
	cur_new_labels = torch.cat(cur_new_labels)

	new_input_embeds.append(cur_new_input_embeds)
	new_labels.append(cur_new_labels)