Comments (16)
I use TITAN Xp with torch1.9.1 to train this model,and i installed the packing and testing it works,the dataset is vcoco and download with the script.Thank you for a lot.
from upt.
Hi @OBVIOUSDAWN,
Thanks for taking an interest in our work.
The Nan loss problem was quite a pain. I ran into the issue a long time ago and managed to resolve it by using larger batch sizes. The problem was that the spatial encodings have bad scales, which made the training very unstable. I see that you are using only one GPU to train. So the batch size is most likely insufficient.
Here are a few things you can try
- For the log terms in the pairwise positional encodings, use log(1+x) instead of log(x+epsilon).
- Add batch norm in the spatial head that computes the pairwise positional encodings.
- Increase batch size (probably the easiest option).
Hope that resolves the issue.
Cheers,
Fred.
from upt.
Dir sir,
I tried this model on new server with 3090*4 ,bacisize =4,which shows the same error on rank3.In your second suggestion,do you mean the "Pairwise Box Positional Encodings" in the paper?I find a "PositionEmbeddingSine" in /detr/model/position_encoding.py to change its eps show the same error ,and i also tired change "binary_focal_loss_with_logits" /"compute_spatial_encodings"'s eps in /ops.py.I print out the whole network but i dont know which part belongs to Pairwise Box Positional Encodings model.I look forward to receiving your reply.Thank you for a lot.
from upt.
...do you mean the "Pairwise Box Positional Encodings" in the paper
Yes, it is implemented in ops.py
. If you are running on 4 GPUs with batch size as 4, you should have an effective batch size of 16. I think that's sufficiently large. Are you still getting the error?
Fred.
from upt.
yes,the effective bachsize is 16,it shows the same error.And i tride to change
features.append( torch.cat([f, torch.log(f + eps)], 1) )
by using log(1+x) instead of log(x+epsilon) in "compute_spatial_encodings",i got the same error.I look forward to receiving your reply.Thank you for a lot.
from upt.
That's odd. If the batch size is 16, it should work now. Can you try some different seeds?
Fred.
from upt.
Hi, @fredzzhang .
Thank you for your contribution. I am very interested in your work. Therefore, I want to deepen my understanding of your paper with running code. But I can't run the code.
I encountered the same error using the same command on 3090.
python main.py --world-size 1 --dataset vcoco --data-root vcoco/ --partitions trainval test --pretrained checkpoints/detr-r50-vcoco.pth --output-dir checkpoints/upt-r50-vcoco2
I haven't changed any code, just download code and checkpoint model according to Readme.
Then I want to run the training command, but failed with this error.
Could you give me some help to solve it?
from upt.
Hi @leijue222,
That should be an issue related to the batch size. I trained the model on 8 GPUs with a batch size of 2 per GPU—effectively a batch size of 16. So since you are training with one GPU, you need to set the batch size to 16.
Let me know if that works.
Fred.
from upt.
Wow, Thanks Fred! It worked!
It is indeed a problem of batch size.
At present, the video memory changes from 12G to 23G. It is unknown whether the single card 3090 with bs=16 will explode the video memory later.
By the way, how much time did you spend training vcoco.
from upt.
Towards end of the Model Zoo section, I added some stats for 8 TITAN X GPUs, which in the case of VCOCO, would be 40 minutes. I don't know how long it will take one 3090 to train it. It shouldn't be too long.
Fred.
from upt.
Thanks again, I love this job.
from upt.
I meet the same error using the command on 3090.
python main.py --world-size 1 --batch-size 16 --dataset vcoco --data-root vcoco/ --partitions trainval test --pretrained checkpoints/detr-r50-vcoco.pth --output-dir checkpoints/upt-r50-vcoco
Could you help me to solve the problem? Thanks.
from upt.
Hi @yuchen2199,
Sometimes the training could be unstable even with batch size of 16. If possible, further increasing the batch size should make it happen less often.
Fred.
from upt.
Thanks for your early reply. I solved the problem after increasing batch-size. This is really an interesting work.
from upt.
Hi,
I am getting the HOI loss is NaN issue when training on a different dataset. The code used to work fine earlier. But, when I tried to train on images where there is only one human bbox and one object bbox, I started facing this issue.
I have tried:
- Setting batch size to 16 and 32
- features.append(
torch.cat([f, torch.log(f + 1)], 1)
) - adding Batch Norm
self.spatial_head = nn.Sequential(
nn.Linear(36, 128),
nn.BatchNorm1d(128), # Batch normalization after the first linear layer
nn.ReLU(),
nn.Linear(128, 256),
nn.BatchNorm1d(256), # Batch normalization after the second linear layer
nn.ReLU(),
nn.Linear(256, representation_size),
nn.BatchNorm1d(representation_size), # Batch normalization after the third linear layer
nn.ReLU(),
)
But, I am still getting this issue.
Do you have any suggestions on how I can solve this issue?
from upt.
To show an example,
I modified the output of compute_spatial_encodings() function like this (adding 5000 so that it's easy to visualize. ):
So, the input to spatial head is
What is a meaningful fix for this?
Scaling, replacing problematic values in the input etc?
from upt.
Related Issues (20)
- list index out of range HOT 2
- error when test vcoco HOT 14
- confused about the vcoco dataset HOT 3
- HICO-DET training accuracy problem HOT 2
- A question about v-coco dataset HOT 2
- Detection results, mAP calculation HOT 2
- torch.multiprocessing.spawn.ProcessRaisedException HOT 2
- Multiple loss training code HOT 2
- train HOT 2
- Detected human–object interactions on video HOT 1
- VCOCO Scenario 1 and Scenario 2 HOT 1
- AttributeError: 'Namespace' object has no attribute 'lr_backbone' HOT 7
- list out of range and checkpoint's state_dict mismatch HOT 2
- UPT predicts scenario 1 of V-COCO HOT 2
- Validation set division on HICO-DET HOT 4
- Real-time measurement HOT 1
- GFLOPs and params HOT 1
- Code to visualize attention maps HOT 1
- suppress overconfident objects HOT 1
- VCOCO evaluation HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from upt.