The utilization rate of GPU is low about mdvc HOT 4 CLOSED

I also remember that during training gpu load was ~50-70% while during the 1-by-1 prediction it ramped up to 100%. So i would assume it is related to torchtext parts.

from mdvc.

Yes, with your help ,I know how to solve this problem. Thanks a lot !

from mdvc.

Yes. I am glad you asked.

I am not sure if both are related because I remember my CPU was using 16 threads at 100% load when training the model. It is even with the num_workers = 0 (I mean the default option).

About the data loader. Yes, you are right it is a bit strange. I came up with this workaround because previously (maybe even still) torchtext and PyTorch were using different approaches for data loading which made it very painful to combine the good stuff about text from torchtext with vision-style pipelines.

An explanation of how it works is a bit tricky:

It is a tricky part.

The DataLoader which is going to wrap this dataset

Besides the iterator (datasetloader variable) that automatically pads sentences to be of the same length, this caption_iterator() outputs the
train_vocab which can be used as token-to-word mapping which is also cumbersome to implement on your own because you need to build a vocabulary (ew!).

Anyway, the data items which we get from self.caption_loader defined here:

So, to get the video features, I need to iterate through the self.caption_loader which I got from the caption_iterator()
to get the indices to the dataset rows (the paths to precalculated features). However, I can't index self.caption_loader with an index somehow to retrieve these captions and indices – it fails. So, I came up with just
doing it in the opposite way: adding the dataset row index as a field in self.caption_loader See here:

As a result, the batch size is not 1 as it is defined in the PyTorch DataLoader here:

So, on a high level: I am wrapping a torchtext BucketIterator inside of the PyTorch DataLoader with batch size = 1. This means that num_workers won't do much here because it loads a batch of one item.

This was the price of having text-padding out-of-the-box back then along with PyTorch dataset class.

Q: Have you ensured that the indices that caption_iterator returned with captions
(caption_data) are unique when you do caption_data = next(self.caption_loader_iter)?
A: Yes, and they are shuffled after every epoch in update_iterator(self): (defined here:

from mdvc.

Oh, thanks a lot. I see what you're trying to do. Really nice work! I only have 2 threads assigned to this task when training the model so it took too much time in fetching dataset. I think I should use more threads if I want to improve the utilization of GPU. When I have 8 threads assigned to the task, it looks good.

from mdvc.