Hello! Thanks for sharing your nice project. I have some uncertainties about the d

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

<a class="user-mention notranslate" data-hovercard-type="user" data-hover

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

Question about the dataloader about fpsg HOT 6 OPEN

lailvlong commented on July 29, 2024

Question about the dataloader

from fpsg.

Comments (6)

voidstrike commented on July 29, 2024

@lailvlong Hey, thanks for your interest in our project!

The short answer to your question is Yes, but it doesn't violate the evaluation protocol.
If you look at

FPSG/src/datasets/modelnet.py

Lines 110 to 112 in 238b1de

 def __getitem__(self, index): 

 img_path, pc_path = self.data_corpus[index].split('\t') 

 data_instance_class = img_path.split('/')[-4]

You will see that we access the reference tensors (containing both TRAIN & TEST split) based on"data_corpus", which is generated in the constructor according to the input "config_path". Look back to the main training script
https://github.com/voidstrike/FPSG/blob/main/src/trainNetwork.py#L85-L90
you will see the "config_path" are indeed different.
Moreover, noted that the base classes and novel classes are mutually exclusive. We use all data from base classes to train the model and test the model on novel classes, it's not a standard 80/20 split.

from fpsg.

lailvlong commented on July 29, 2024

@lailvlong Hey, thanks for your interest in our project!

The short answer to your question is Yes, but it doesn't violate the evaluation protocol. If you look at

FPSG/src/datasets/modelnet.py

Lines 110 to 112 in 238b1de

def __getitem__(self, index):

img_path, pc_path = self.data_corpus[index].split('\t')

data_instance_class = img_path.split('/')[-4]

You will see that we access the reference tensors (containing both TRAIN & TEST split) based on"data_corpus", which is generated in the constructor according to the input "config_path". Look back to the main training script
https://github.com/voidstrike/FPSG/blob/main/src/trainNetwork.py#L85-L90
you will see the "config_path" are indeed different.
Moreover, noted that the base classes and novel classes are mutually exclusive. We use all data from base classes to train the model and test the model on novel classes, it's not a standard 80/20 split.

Thanks for your reply.
Yes, the "config_path" for the train and test dataloaders are different, but the “reference_path” of them are the same.

FPSG/src/trainNetwork.py

Lines 85 to 87 in 238b1de

 if opt.dataset == 'modelnet': 

 ds = FewShotModelNet(config_path, reference_path, n_classes=n_way, n_support=n_shot, n_query=n_query, transform=_modelnet_tfs) 

 ds_test = FewShotModelNet(test_path, reference_path, n_classes=n_way, n_support=n_shot, n_query=n_query, transform=_modelnet_tfs)

And the “reference_path” contains the files of both the base and novel categories. Thus, for the train dataloader, data of novel categories would also be loaded into "self.img_corpus" and "self.pc_corpus".

FPSG/src/datasets/modelnet.py

Lines 130 to 153 in 238b1de

 def _build_reference(self, ): 

 assert self.auxiliary_dir is not None, 'Auxiliary folder is not generated yet!!!' 

 tmp_img_list, tmp_pc_list = list(), list() 

 for eachFile in os.listdir(self.auxiliary_dir): 

 if not eachFile.endswith('.txt'): 

 continue 

 class_name = eachFile.split('.')[0].split('+')[1] 

 # self.reference[class_name] = dict() 

 print(f'Building Reference dataset for {class_name} ...') 

 class_ds = FewShotSubModelNet(os.path.join(self.auxiliary_dir, eachFile), transform=self.tfs, tgt_transform=self.tgt_tfs) 

 loader = DataLoader(class_ds, batch_size=len(class_ds), shuffle=False) 

 for stacked_img, stacked_pc in loader: 

 self.reference[class_name]['imgs'] = stacked_img 

 self.reference[class_name]['pcs'] = stacked_pc 

 tmp_img_list.append(stacked_img) 

 tmp_pc_list.append(stacked_pc) 

 break # Follow the protonet, only need one sample because batch_size equal to the dataset length 

 self.img_corpus = torch.cat(tmp_img_list, dim=0) 

 self.pc_corpus = torch.cat(tmp_pc_list, dim=0)

During iterating the train dataloader, data in the "self.img_corpus" and "self.pc_corpus" would be fetched as "ans[xad]" and "ans[pcad]", including those of novel categories.

FPSG/src/datasets/modelnet.py

Lines 110 to 126 in 238b1de

 def __getitem__(self, index): 

 img_path, pc_path = self.data_corpus[index].split('\t') 

 data_instance_class = img_path.split('/')[-4] 

 query_matrix = { 

 'class': data_instance_class, 

 'img_data': self.reference[data_instance_class]['imgs'], 

 'pc_data': self.reference[data_instance_class]['pcs'], 

 } 

 ans = extract_episode(self.n_support, self.n_query, query_matrix) # Original episode 

 # tgt_path = self.reference[data_instance_class]['imgs'][ans['tmp']] 

 # print(f'{index}: {tgt_path}') 

 example_idx = torch.randperm(self.item_len)[:self.n_support] # Adding additional img-pc pairs to avoid model collapse 

 ans['xad'] = self.img_corpus[example_idx] 

 ans['pcad'] = self.pc_corpus[example_idx]

When computing the intra_support loss during training, there are two options, namely "option 2" and "option 1".

FPSG/src/models/few_shot.py

Lines 63 to 73 in 238b1de

 def loss(self, sample): 

 # Gather input 

 xs, xq, pcs, pcq = Variable(sample['xs']), Variable(sample['xq']), Variable(sample['pcs']), Variable(sample['pcq']) 

 # NOTE: following 2 options are interchangeable 

 # xad, pcad = xs, pcs # Option 1 

 xad, pcad = Variable(sample['xad']), Variable(sample['pcad']) # Option 2 

 return self._loss_single_class(xs, xq, xad, pcs, pcq, pcad)

The codes select "option 2", which uses the "ans[xad]" and "ans[pcad]" and may involves the data of the novel categories. This is the point that i feel puzzled.
In contrast, the "option 1" that uses the "ans[xs]" and "ans[pcs]" should have been adopted. Moreover, i have tried both "option 1" and "option 2". With my observation, in ModelNet dataset, the performance of "option 1" (avg_cd=7.+) are much worser than that of "option 2" (avg_cd=2.+). It is very weried that the two options have such different performances. Or maybe that "option 2" has used the evaluation data for training.

from fpsg.

voidstrike commented on July 29, 2024

@lailvlong Oh my god, I see, that's definitely an information leak.

The reason we adopt "option 2" is we found that "option 1" will give us almost the same point cloud during the test phase. We thought it was a model collapse because the model observes one and only one class per episode, therefore, we tried to make some variants by sampling random (IMG, PC) pairs from the dataset (so each episode contains more than one class). We just use "randperm" and forgot "self.img_corpus" and "self.pc_corpus" contain evaluation points.

Thanks for your question and it shows that the problem is solved by information leak rather than bringing more classes in. Really sorry for the mistake.

Last but not least, could you please somehow try to modify the code such that keeps random sampling from "self.img_corpus" and "self.pc_corpus" but avoid information leak? This one is absolutely based on your interest and feel free to ignore it. Sorry again for the bug.

from fpsg.

lailvlong commented on July 29, 2024

@voidstrike Yes, i also find that, during the test phase, the generated point clouds of different query images are very similar in an episode. In my opinion, it is not caused by model collapse but by that the support prototypes (class-specific prior) dominates the generation. Since reconstructing the shape of query image is much harder than restoring the shape of the support prototype (mean shape of the support point clouds). Meanwhile, restoring the shape of the support prototype can also offer a low loss. In this way, the model prefers to ignore the query image and output the same results.
To enable the model to consider more categories in one gradient descent step, maybe we can construct a mini-batch with several episodes. In your project, we can set "n_way>1 " and make other corresponding modifications.

from fpsg.

voidstrike commented on July 29, 2024

@lailvlong A good perspective! Thanks for your investigation & suggestions, considering more categories in one gradient step definitely sounds promising and makes sense to me. Unfortunately, I already left the university and I'm no longer working in this area.
Please feel free to modify the code if you want and looking forward to your paper. :)

from fpsg.

BTAROCKET commented on July 29, 2024

@lailvlong @voidstrike >
Hello! I get the same question. Could you please share the solution about it. I am also confused about how to test on base class. I have tried but got bad result which is about 10 times as long as the traing phase CD. Looking forward to your reply. Thank you!

from fpsg.

Question about the dataloader about fpsg HOT 6 OPEN

Comments (6)

Related Issues (2)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	def __getitem__(self, index):
	img_path, pc_path = self.data_corpus[index].split('\t')
	data_instance_class = img_path.split('/')[-4]

	if opt.dataset == 'modelnet':
	ds = FewShotModelNet(config_path, reference_path, n_classes=n_way, n_support=n_shot, n_query=n_query, transform=_modelnet_tfs)
	ds_test = FewShotModelNet(test_path, reference_path, n_classes=n_way, n_support=n_shot, n_query=n_query, transform=_modelnet_tfs)

	def _build_reference(self, ):
	assert self.auxiliary_dir is not None, 'Auxiliary folder is not generated yet!!!'
	tmp_img_list, tmp_pc_list = list(), list()

	for eachFile in os.listdir(self.auxiliary_dir):
	if not eachFile.endswith('.txt'):
	continue

	class_name = eachFile.split('.')[0].split('+')[1]
	# self.reference[class_name] = dict()

	print(f'Building Reference dataset for {class_name} ...')
	class_ds = FewShotSubModelNet(os.path.join(self.auxiliary_dir, eachFile), transform=self.tfs, tgt_transform=self.tgt_tfs)
	loader = DataLoader(class_ds, batch_size=len(class_ds), shuffle=False)

	for stacked_img, stacked_pc in loader:
	self.reference[class_name]['imgs'] = stacked_img
	self.reference[class_name]['pcs'] = stacked_pc
	tmp_img_list.append(stacked_img)
	tmp_pc_list.append(stacked_pc)
	break # Follow the protonet, only need one sample because batch_size equal to the dataset length

	self.img_corpus = torch.cat(tmp_img_list, dim=0)
	self.pc_corpus = torch.cat(tmp_pc_list, dim=0)

	def loss(self, sample):
	# Gather input
	xs, xq, pcs, pcq = Variable(sample['xs']), Variable(sample['xq']), Variable(sample['pcs']), Variable(sample['pcq'])

	# NOTE: following 2 options are interchangeable

	# xad, pcad = xs, pcs # Option 1
	xad, pcad = Variable(sample['xad']), Variable(sample['pcad']) # Option 2


	return self._loss_single_class(xs, xq, xad, pcs, pcq, pcad)