yikang-li / factorizablenet Goto Github PK

/home/prudhvik/FactorNet/FactorizableNet/data/VRD/images/sg_test_images/161638840_4de67b36c9_o.jpg
use_gt_boxes=args.use_gt_boxes)
File "/home/prudhvik/FactorNet/FactorizableNet/models/HDN_v2/engines_v1.py", line 123, in test
for i, sample in enumerate(loader): # (im_data, im_info, gt_objects, gt_relationships)
File "/home/prudhvik/anaconda/envs/py2/lib/python2.7/site-packages/torch/utils/data/dataloader.py", line 637, in next
return self._process_next_batch(batch)
File "/home/prudhvik/anaconda/envs/py2/lib/python2.7/site-packages/torch/utils/data/dataloader.py", line 658, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
AttributeError: Traceback (most recent call last):
File "/home/prudhvik/anaconda/envs/py2/lib/python2.7/site-packages/torch/utils/data/dataloader.py", line 138, in _worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/prudhvik/FactorNet/FactorizableNet/lib/datasets/VRD_loader.py", line 66, in getitem
img_original_shape = img.shape
AttributeError: 'NoneType' object has no attribute 'shape'

Hi, I see that some of the images are not being read by imread() function and causing this issue. I am using VRD dataset for evaluating the pretrained model.

files for import lib.roi_data_layer.roidb as rdl_roidb missing. thanks for updating.

After running the evaluation script, I have the result testing_result.pkl, but I'm unsure how to interpret the annotations. For example, printing result[0]['relationships'] shows:

[(25, 88, 4, 0.052851997),
 (29, 25, 10, 0.052471865),
 (29, 51, 10, 0.040551774),
 (7, 25, 10, 0.038298395),
 (3, 25, 10, 0.036385886),
 (29, 74, 8, 0.035008814),
...]

I assumed that the fourth value in each tuple is the confidence of that detected relationship. But then I found that there are no relationships above threshold=0.5. How can I decode these values such that I can deduce tuples of the form (subject_class, object_class, predicate_class) for every image?

P.S. I tried looking at visualize_graph.py to figure this out but found it unnecessarily complicated for what I'm trying to do. I don't need to visualize a scene graph; I just want to decode the relationships in each image.

I ran your code with your trained-model for VG-MSDN dataset, but I faced huge gap between your result and mine.

In the ReadMe document, the performance of provided (pre-trained for VG-MSDN) model is mentioned Recall@50 12.984% and Recall@100 16.506%.

However, I ran your code with provided model, I got below result.

I download visual genome dataset from official website, there are two zip files images.zip and images2.zip.
After download them, I merged two directory (images, images2) into single directory, then create symbolic link as mentioned your ReadMe File.

Could you give me some hint for what I missed?

In figure3， it shows the avg-pooled features(m × 515×1×1),but it didn't dot product 1*512. Is the number 515 wrong? 512??

File "train_FN.py", line 400, in
main()
File "train_FN.py", line 279, in main
use_gt_boxes=args.use_gt_boxes)
File "/home/swf/F-Net/models/HDN_v2/engines_v1.py", line 133, in test
use_gt_boxes=use_gt_boxes)
File "/home/swf/F-Net/models/HDN_v2/factorizable_network_v4.py", line 280, in evaluate
object_result, predicate_result = self.forward_eval(im_data, im_info,)
File "/home/swf/F-Net/models/HDN_v2/factorizable_network_v4.py", line 232, in forward_eval
features, object_rois, _ = self.rpn(im_data, im_info)
File "/home/swf/anaconda3/envs/f-net/lib/python2.7/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/swf/F-Net/models/RPN/RPN.py", line 115, in forward
mappings=self.opts['mappings'])
File "/home/swf/F-Net/models/RPN/RPN.py", line 141, in proposal_layer
_feat_stride, opts, anchor_scales, anchor_ratios, mappings)
File "/home/swf/F-Net/lib/rpn_msr/proposal_layer.py", line 134, in proposal_layer
keep = nms(np.hstack((proposals, scores)).astype(np.float32), nms_thres)
File "/home/swf/F-Net/lib/fast_rcnn/nms_wrapper.py", line 26, in nms
return nms_gpu(dets[:, :4], dets[:, 4], thresh).cpu().numpy()
RuntimeError: Not compiled with GPU support (nms at /home/dell/F-Net/lib/layer_utils/csrc/nms.h:22)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x45 (0x7fc66e615cf5 in /home/swf/anaconda3/envs/f-net/lib/python2.7/site-packages/torch/lib/libc10.so)

epoch0
Top-50 Recall: [Pred] 11.864% [Phr] 9.651% [Rel] 4.358%
Top-100 Recall: [Pred] 15.013% [Phr] 12.640% [Rel] 5.590%

epoch1
Top-50 Recall: [Pred] 11.905% [Phr] 11.497% [Rel] 5.128%
Top-100 Recall: [Pred] 15.273% [Phr] 14.760% [Rel] 6.507%

Train with VG-MSDN dataset from scratch.
I am wondering that to achieve the best model, how many epochs should I train?

Best regards

Hi, thanks a lot for sharing this repository, it looks really awesome!

What is the degree of improvement in scores before re-training end-to-end of the whole model (i.e. after training on the rpn and features? in stage 2?)
(I'm considering using your model with fixed extracted features from another detector so was wondering about that).

In particular I'm interested most in results on visual genome graph detection/generation.

Thanks a lot!

Hi, great work on the project! Just wanted to ask if it is possible to run the network on unannotated raw images without loading the annotated.json file,

It seems that under your factorizable_net_v4.py there is a function forward_evaluate() to generate object names, ROIs, predicates... etc. So I'm assuming that I can use this function to evaluate unannotated raw images?

Provided that the loader file is changed to not load the annotated.json, only load the image as numpy arrays and the --evaluate uses forward_evaluate() instead of evaluate() to output the results.

Hence, if my goal is to just output some results that I do not need to compute the recall on, can I remove the loading of the annotated.json (categories, inverse weights, objects etc.) entirely and do as stated above? Thanks a lot!

Hi,

Thank you for the code.

Can you please provide the scripts you used for preparing train.json/test.json so people can use your models on custom data?

Regards,

Thank you for releasing this repo. Would it be possible to use this model to do a demo on my own video (not just an image)? If yes, what would the steps be?

Thank you,

Hello author:
When I trying to run your code, it rports:

THCudaCheck FAIL file=/pytorch/torch/lib/THC/generic/THCTensorMath.cu line=35 error=8 : invalid device function Traceback (most recent call last): File "train_FN.py", line 390, in <module> main() File "train_FN.py", line 277, in main use_gt_boxes=args.use_gt_boxes) File "/home/linxin/FactorizableNet/models/HDN_v2/engines_v1.py", line 123, in test use_gt_boxes=use_gt_boxes) File "/home/linxin/FactorizableNet/models/HDN_v2/factorizable_network_v4.py", line 271, in evaluate object_result, predicate_result = self.forward_eval(im_data, im_info,) File "/home/linxin/FactorizableNet/models/HDN_v2/factorizable_network_v4.py", line 232, in forward_eval pooled_object_features = self.roi_pool_object(features, object_rois).view(len(object_rois), -1) File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 357, in __call__ result = self.forward(*input, **kwargs) File "/home/linxin/FactorizableNet/lib/roi_align/modules/roi_align.py", line 16, in forward self.spatial_scale)(features, rois) File "/home/linxin/FactorizableNet/lib/roi_align/functions/roi_align.py", line 22, in forward output = features.new(num_rois, num_channels, self.aligned_height, self.aligned_width).zero_() RuntimeError: cuda runtime error (8) : invalid device function at /pytorch/torch/lib/THC/generic/THCTensorMath.cu:35

I have changed the lib/make.sh file since my CUDA_ARCH do not support 6.0. The make.sh seems to work for me, only having a few warning. After reading [I](jwyang/faster-rcnn.pytorch#110

) have re-built the make.sh for a few times, the cuda error does not overcomed.
/home/linxin/FactorizableNet/lib/roi_crop/src/roi_crop.c: In function ‘BilinearSamplerBHWD_updateGradInput’: /home/linxin/FactorizableNet/lib/roi_crop/src/roi_crop.c:190:14: warning: unused variable ‘inBottomRight’ [-Wunused-variable] real inBottomRight=0; ^ /home/linxin/FactorizableNet/lib/roi_crop/src/roi_crop.c:189:14: warning: unused variable ‘inBottomLeft’ [-Wunused-variable] real inBottomLeft=0; ^ /home/linxin/FactorizableNet/lib/roi_crop/src/roi_crop.c:188:14: warning: unused variable ‘inTopRight’ [-Wunused-variable] real inTopRight=0; ^ /home/linxin/FactorizableNet/lib/roi_crop/src/roi_crop.c:187:14: warning: unused variable ‘inTopLeft’ [-Wunused-variable] real inTopLeft=0; ^ /home/linxin/FactorizableNet/lib/roi_crop/src/roi_crop.c:186:14: warning: unused variable ‘v’ [-Wunused-variable] real v=0; ^ /home/linxin/FactorizableNet/lib/roi_crop/src/roi_crop.c: In function ‘BilinearSamplerBCHW_updateGradInput’: /home/linxin/FactorizableNet/lib/roi_crop/src/roi_crop.c:441:14: warning: unused variable ‘inBottomRight’ [-Wunused-variable] real inBottomRight=0; ^ /home/linxin/FactorizableNet/lib/roi_crop/src/roi_crop.c:440:14: warning: unused variable ‘inBottomLeft’ [-Wunused-variable] real inBottomLeft=0; ^ /home/linxin/FactorizableNet/lib/roi_crop/src/roi_crop.c:439:14: warning: unused variable ‘inTopRight’ [-Wunused-variable] real inTopRight=0; ^ /home/linxin/FactorizableNet/lib/roi_crop/src/roi_crop.c:438:14: warning: unused variable ‘inTopLeft’ [-Wunused-variable] real inTopLeft=0; ^ /home/linxin/FactorizableNet/lib/roi_crop/src/roi_crop.c:437:14: warning: unused variable ‘v’ [-Wunused-variable] real v=0; ^

My environment is CUDA8.0 pytorch0.3.1 python2.7

Hope to recieve your reply！

###THX

Than you for your excellent code. I found today and when I try to download have a problem:

Can you review the link to the pretrained RPN for Visual Genome? Now I get the same file VRD.tgz for the VRD dataset.

Best regards.

Would you mind sharing the conda environment that the current code from the master branch works with?
That is, could you provide a link to a env.yml file?

Hi,

Few questions regarding the multi-gpu settings:

1. What should be the command to run the model with multi-gpu?
2. Have you tested the model gets the same performance in the multi-gpu settings? (Asking in particular because the readme stresses that this is beta version so not sure, also I believe that recent commit also was about updating pytorch version and saw couple of other updates)
3. In the standard single-gpu setting, how much time did it take you to train 1 epoch? I noticed that the gpu utilization is very low and was wondering maybe it's because of the recent multi-gpu commit?

In particular, I get a couple of warnings when trying to run it with the command on the readme page:

/juice/u/nlp/packages/anaconda_slurm/envs/dorarad-py2/lib/python2.7/site-packages/torch/nn/_reduction.py:47: UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead.
  warnings.warn(warning.format(ret))
/juice/u/nlp/packages/anaconda_slurm/envs/dorarad-py2/lib/python2.7/site-packages/torch/nn/parallel/_functions.py:61: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
  warnings.warn('Was asked to gather along dimension 0, but all '

Thanks a lot!!

Could anyone tell what is HDN model being referred very much in the code. It is nowhere mentioned in the paper. I am not getting the expanded form of it.

Thanks,
Sidharth

what is your pytorch version? thank you.

Dear Author,
Due to the pandemic, I cannot go to my lab to reinstall a linux for my pc. I've tried many ways to build this project on win10, but never got a luck. For unknown reason, that pc cannot update to the new 2004 version of win10 which allows wsl 2, therefore, I am not able to use linux subsystem either. If possible, please show me the way to build this project on win10. Thank you very much!

In evaluate stage,predict boxes and ground truth boxes are used to compute a iou score to retain reasonable boxes.But what to do without ground truth?

Hi

I want to confirm if the Visual Genome dataset used for getting the accuracy mentioned in the paper is train_small/train/train_fat ?

Thanks.

Thanks for the generous code release!

I have a quick question, as mentioned in README.md, you converted original annotations into json files. For VRD, it should have 4000 training images and 1000 test images. However, in the provided json files, there are 3780 training images and 955 test images. May I know why there is such a discrepancy?

Thanks!

Hi,I want to know how to compute weight of predicate,which is in your inverse_weight of MSDN project.

Would it be possible to provide a example code to run FNet on my own image and show the result?

After setting up the environment, I tried evaluating the model but keep running into this error:

Traceback (most recent call last):
  File "train_FN.py", line 20, in <module>
    import lib.datasets as datasets
  File "/n/fs/cap/bfeng/FactorizableNet/lib/datasets/__init__.py", line 10, in <module>
    from .VRD_loader import VRD
  File "/n/fs/cap/bfeng/FactorizableNet/lib/datasets/VRD_loader.py", line 19, in <module>
    from lib.rpn_msr.anchor_target_layer import anchor_target_layer
  File "/n/fs/cap/bfeng/FactorizableNet/lib/rpn_msr/anchor_target_layer.py", line 20, in <module>
    from lib.fast_rcnn.bbox_transform import bbox_transform
  File "/n/fs/cap/bfeng/FactorizableNet/lib/fast_rcnn/__init__.py", line 9, in <module>
    from . import nms_wrapper
  File "/n/fs/cap/bfeng/FactorizableNet/lib/fast_rcnn/nms_wrapper.py", line 8, in <module>
    from lib.layer_utils.roi_layers import nms as nms_gpu
  File "/n/fs/cap/bfeng/FactorizableNet/lib/layer_utils/roi_layers/__init__.py", line 3, in <module>
    from .nms import nms
  File "/n/fs/cap/bfeng/FactorizableNet/lib/layer_utils/roi_layers/nms.py", line 3, in <module>
    from layer_utils import _C
ImportError: /n/fs/cap/bfeng/FactorizableNet/lib/layer_utils/_C.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe26detail36_typeMetaDataInstance_preallocated_4E

I ran make in the lib folder with gcc version 5.3.1, CUDA 9.0, PyTorch 0.4.1. Can someone please help with this error?

Hello!
I've been looking into how to train this model on an external dataset, and I was hoping to get some clarification on the JSON files you feed the data loaders. I imagine the kmeans_anchors file is generated from the RPN model that you use, but could I get some insight into what exactly is in the inverse_weights file? I can tell that you're weighting the object and predicate classes by some value, but how exactly were those values generated? Or would I need them at all if I make a data loader class for my dataset?

Hi,

Thanks for publicizing the code and it is really well-written. However, there seems to a minor prpblem related to evaluating object detection result. In train_FN.py, there is no reference for function "python_eval" as shown below:

if args.evaluate_object:
result = model.module.engines.test_object_detection(test_loader, model, nms=args.nms,use_gt_boxes=args.use_gt_boxes)
print('============ Done ============')
path_dets = save_detections(result, None, options['logs']['dir_logs'], is_testing=True)
print('Evaluating...')
python_eval(path_dets, osp.join(data_opts['dir'], 'object_xml'))
return

Would it possible to upload the code for this function? Thanks in advance

While running the evaluation code, which config file is being used ? Is it config.py under lib/fast_rcnn or config2.py under lib/fast_rcnn ?

First of all thanks for the repository.
I have some queries hope they get answered soon.

1. • How to run code to generate scene graph for a single test-image.
     Do we still have to download whole dataset to run or pnly downloading pretrained model is fine ?
2. • I get it that you are outputting scene graph in an image. But is there a possibility to store scene graph in XML/JSON format or any just to storage ?

Thanks in advance.

the order is
CUDA_VISIBLE_DEVICES=0 python train_FN.py --evaluate_object
--path_opt options/models/VRD.yaml
--pretrained_model output/best_model_4.h5
========== Testing =======
============ Done ============
Done dumping to: output/FN_VRD_2_iters_SGD/evaluate_object_detection
Evaluating...
Traceback (most recent call last):
File "train_FN.py", line 402, in
main()
File "train_FN.py", line 299, in main
python_eval(path_dets, osp.join(data_opts['dir'], 'object_xml'))
NameError: global name 'python_eval' is not defined.
How to deal with this problem?Thanks!

Hey @yikang-li ,

Can you provide the model which makes use of external knowledge such as region captions, which is described in your paper? The current model (factorizable_network_v4.py) cannot incorporate the region captions and in the RPN_FN.yaml file, the configurations for objects are given but the configurations for regions are not given. Without the use of these external captions, the evaluation of the pretrained MSDN model is not giving the reported results in the README file.

Hi, thank you for your great code.
The download link of VG-MSDN, VG-DR-Net are the same and point to the vg_msdn.tgz file, could you check it please?

Further more:
In part: Evaluation with our Pretrained Models
CUDA_VISIBLE_DEVICES=0 python train_FN.py --evaluate --dataset_option=normal
--path_opt options/models/VG-MSDN.yaml
--pretrained_model output/trained_models/Model-VG-MDSN.h5

The pretrained model name is Model-VG-MSDN.h5 but not Model-VG-MDSN.h5 ...
And the results of VG-DR-Net is the same as the VG-MSDN and is different from the paper.

The link for VRD dataset seems to be unavailable: http://imagenet.stanford.edu/internal/jcjohns/scene_graphs/sg_dataset.zip

I'm not sure whether the link is wrong or other reasons.

I train the RPN on custom data, and the value of rpn_loss_box is about 2.7(so high?). Is it right?
Thanks.

But in the figure2(3), the subgraph has contained three object. But the below only refers to two object. So could the subgraph merged from three objects be computed by three object proposals' score's product?

Hey @yikang-li,

The _C.so file in lib/layer_utils is giving a segmentation fault in all the files it is being called. For example, the first fault comes in lib/ayer_utils/roi_layers/nms.py (nms = _C.nms) file.

In conclusion, wherever the _C.so file is imported is giving me a segmentation fault. Could you perhaps help me out?

These are my settings:
CUDA = 10.1
python = 2.7.16
PyTorch = 1.0.1
gcc = 5.2.0

Thank you very much,
Sandeep.

Here is the fault:

`Fatal Python error: Segmentation fault

Thread 0x00007fe32b9ec700 (most recent call first):
File " /home/xyz/opt/python-2.7.16/lib/python2.7/threading.py", line 340 in wait
File " /home/xyz/opt/python-2.7.16/lib/python2.7/multiprocessing/queues.py", line 252 in _feed
File " /home/xyz/opt/python-2.7.16/lib/python2.7/threading.py", line 754 in run
File " /home/xyz/opt/python-2.7.16/lib/python2.7/threading.py", line 801 in __bootstrap_inner
File " /home/xyz/opt/python-2.7.16/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x00007fe32c550700 (most recent call first):
File " /home/xyz/opt/python-2.7.16/lib/python2.7/threading.py", line 340 in wait
File " /home/xyz/opt/python-2.7.16/lib/python2.7/multiprocessing/queues.py", line 252 in _feed
File " /home/xyz/opt/python-2.7.16/lib/python2.7/threading.py", line 754 in run
File " /home/xyz/opt/python-2.7.16/lib/python2.7/threading.py", line 801 in __bootstrap_inner
File " /home/xyz/opt/python-2.7.16/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x00007fe32cd51700 (most recent call first):
File " /home/xyz/opt/python-2.7.16/lib/python2.7/threading.py", line 340 in wait
File " /home/xyz/opt/python-2.7.16/lib/python2.7/multiprocessing/queues.py", line 252 in _feed
File " /home/xyz/opt/python-2.7.16/lib/python2.7/threading.py", line 754 in run
File " /home/xyz/opt/python-2.7.16/lib/python2.7/threading.py", line 801 in __bootstrap_inner
File " /home/xyz/opt/python-2.7.16/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x00007fe3e0932700 (most recent call first):
File " /home/xyz/opt/python-2.7.16/lib/python2.7/threading.py", line 340 in wait
File " /home/xyz/opt/python-2.7.16/lib/python2.7/multiprocessing/queues.py", line 252 in _feed
File " /home/xyz/opt/python-2.7.16/lib/python2.7/threading.py", line 754 in run
File " /home/xyz/opt/python-2.7.16/lib/python2.7/threading.py", line 801 in __bootstrap_inner
File " /home/xyz/opt/python-2.7.16/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x00007fe3e0131700 (most recent call first):
File " /home/xyz/opt/python-2.7.16/lib/python2.7/multiprocessing/queues.py", line 131 in get
File " /raid/xyz/python-environments/ipk3/lib/python2.7/site-packages/torch/utils/data/dataloader.py", line 158 in _pin_memory_loop
File " /home/xyz/opt/python-2.7.16/lib/python2.7/threading.py", line 754 in run
File " /home/xyz/opt/python-2.7.16/lib/python2.7/threading.py", line 801 in __bootstrap_inner
File " /home/xyz/opt/python-2.7.16/lib/python2.7/threading.py", line 774 in __bootstrap

Current thread 0x00007fe40d711740 (most recent call first):
File " /raid/xyz/FactorizableNet/lib/fast_rcnn/nms_wrapper.py", line 26 in nms
File " /raid/xyz/FactorizableNet/lib/rpn_msr/proposal_layer.py", line 134 in proposal_layer
File " /raid/xyz/FactorizableNet/models/RPN/RPN.py", line 141 in proposal_layer
File " /raid/xyz/FactorizableNet/models/RPN/RPN.py", line 115 in forward
File " /raid/xyz/python-environments/ipk3/lib/python2.7/site-packages/torch/nn/modules/module.py", line 489 in call
File " /raid/xyz/FactorizableNet/models/HDN_v2/factorizable_network_v4.py", line 143 in forward
File " /raid/xyz/python-environments/ipk3/lib/python2.7/site-packages/torch/nn/modules/module.py", line 489 in call
File " /raid/xyz/python-environments/ipk3/lib/python2.7/site-packages/torch/nn/parallel/parallel_apply.py", line 59 in _worker
File " /raid/xyz/python-environments/ipk3/lib/python2.7/site-packages/torch/nn/parallel/parallel_apply.py", line 77 in parallel_apply
File " /raid/xyz/python-environments/ipk3/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 153 in parallel_apply
File " /raid/xyz/FactorizableNet/models/modules/dataParallel.py", line 36 in forward
File " /raid/xyz/python-environments/ipk3/lib/python2.7/site-packages/torch/nn/modules/module.py", line 489 in call
File " /raid/xyz/FactorizableNet/models/HDN_v2/engines_v1.py", line 44 in train
File "train_FN.py", line 336 in main
File "train_FN.py", line 403 in
Segmentation fault`

Hello, I'm trying to test using the pretrained model, but the error appeared after testing 1000 images, and my GPU has 10986MiB memory:
RuntimeError: CUDA out of memory. Tried to allocate 957.12 MiB (GPU 0; 10.73 GiB total capacity; 8.16 GiB already allocated; 801.19 MiB free; 865.05 MiB cached)
How could I solve this problem? How much memory needed for testing?

Hi,

I am getting the following runtime warning while evaluating

Any thoughts on why this might be happening? Thanks!