The pcb from megvii-research

pcb's People

Contributors

Stargazers

Watchers

pcb's Issues

run on single GPU

Hello, thank you very much for your work. When I reproduce your code, I find that the code cannot run on a single GPU, and the following error occurs. I would like to ask how to run code on a single GPU.

RuntimeError: MaskRCNN: IterativeRoIHead: Shared2FCBBoxHead: CrossEntropyPCBLoss: Default process group has not been initialized, please make sure to call init_process_group.

A potential bug in gathering data from multiple processes

Hi, I encountered a stuck issue when training in DDP, and here is the reason I figured out:

PCB/mmdet/models/losses/cross_entropy_pcb_loss.py

Lines 281 to 291 in 066cd92

 pos_inds = label < self.num_classes 

 neg_inds = label == self.num_classes 

 if pos_inds.sum() > 0: 

 pred_fg_distri = F.softmax(cls_score[pos_inds, :self.num_classes], dim=1) 

 fg_confusion_matrix_tmp = torch.zeros_like(self.fg_confusion_matrix).scatter_add_(0, label[pos_inds].view(-1,1).repeat(1,self.num_classes), pred_fg_distri) 

 fg_confusion_matrix_tmp_pool = [torch.zeros_like(fg_confusion_matrix_tmp) for i in range(torch.distributed.get_world_size())] 

 torch.distributed.all_gather(fg_confusion_matrix_tmp_pool, fg_confusion_matrix_tmp) 

 fg_confusion_matrix_tmp = sum(fg_confusion_matrix_tmp_pool)

the if branch starting from line 284 may be entered for some processes, and may not for others (e.g. some training images contain no labelled instance), so the gather function in line 289 will permanently wait for those processes that never execute line 284~288.

One possible solution is to check if all processes satisfy pos_inds.sum() > 0. If not, do not update the confusion matrix.

Recommend Projects

megvii-research / pcb Goto Github PK

pcb's People

Contributors

Stargazers

Watchers

Forkers

pcb's Issues

run on single GPU

A potential bug in gathering data from multiple processes

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	pos_inds = label < self.num_classes
	neg_inds = label == self.num_classes
	if pos_inds.sum() > 0:
	pred_fg_distri = F.softmax(cls_score[pos_inds, :self.num_classes], dim=1)

	fg_confusion_matrix_tmp = torch.zeros_like(self.fg_confusion_matrix).scatter_add_(0, label[pos_inds].view(-1,1).repeat(1,self.num_classes), pred_fg_distri)

	fg_confusion_matrix_tmp_pool = [torch.zeros_like(fg_confusion_matrix_tmp) for i in range(torch.distributed.get_world_size())]
	torch.distributed.all_gather(fg_confusion_matrix_tmp_pool, fg_confusion_matrix_tmp)
	fg_confusion_matrix_tmp = sum(fg_confusion_matrix_tmp_pool)