ltkong218 / fastflownet Goto Github PK

Hi, @ltkong218,

Thanks for your exciting work, do you have any plan to release the evaluation code of MPI-Sitel training and KITTI?

Hi, i am running your code. I saw in your paper the ground-truth flow was divided by 20 before training, but seems didn't see this parameter in FastFlowNet.py, have you used this div_flow parameter in code?

Hi,
I wanted to install the correlation package using Cuda 9 and Pytorch 0.4 but I found this error:
"119 | #error -- unsupported GNU version! gcc versions later than 6 are not supported!"
So I tried installing older version of gcc but I couldent (The OS didnt allow installing old version)
My ubuntu is version 20 (Kubuntu).
Regards,
Ali

Hi，I have a problem like this, how should I solve it?

Traceback (most recent call last):
File "/data/FastFlowNet-main/demo.py", line 45, in
output = model(input_t).data
File "/root/anaconda3/envs/FastFlowNet/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/anaconda3/envs/FastFlowNet/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/data/FastFlowNet-main/models/FastFlowNet.py", line 134, in forward
cv6 = torch.index_select(self.corr(f16, f26), dim=1, index=self.index.to(f16).long())
File "/root/anaconda3/envs/FastFlowNet/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/anaconda3/envs/FastFlowNet/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
TypeError: forward() missing 1 required positional argument: 'input2'

我是Windows平台下cuda10.2 torch1.9.0，没办法编译你提供的代码种的corrlation模块，然后用你说的那个第三方的库https://github.com/ClementPinard/Pytorch-Correlation-extension 编译过了，但是怎么替换你网络中的self.corr；主要是这两个接口参数完全不一样，spatial_correlation_sample( kernel_size=3,patch_size=1, stride=2,padding=0,dilation=2,dilation_patch=1)；你的接口是 Correlation(pad_size=4, kernel_size=1, max_displacement=4, stride1=1, stride2=1, corr_multiply=1)；怎么一一对应起来

My env: Pytorch: 1.9.0+cu111
GPU: NVIDIA GeForce RTX 2080 Ti
Sys: Ubuntu 18.04.5 LTS

I ran python setup.py build
error: command 'gcc' failed with exit status 1

And no surprise, I encountered ModuleNotFoundError: No module named 'correlation_cuda' when I ran python demo.py

I tried:
sudo apt-get update
sudo apt-get install python-dev
sudo apt-get install python3-dev
sudo apt-get install libevent-dev
sudo apt-get install build-essential
sudo apt-get install libblas-dev libatlas-base-dev
sudo apt install gcc
But still not work.

Help me pls! Thx!

A few questions about exporting FastFlowNet to TensorRT for inference on the Xavier TX2.

• Which version of TensorRT was used to test FastFlowNet on the Xavier TX2?
• Which toolkit or opensource project did you use to export FastFlowNet from pytorch to TensorRT? Did you use torch2trt?

Have you tried deploying your model on android?

run on colab.research.google.com;
and output:
sh: 1: ping: not found
sh: 1: ping: not found
sh: 1: ping: not found
sh: 1: ping: not found
Traceback (most recent call last):
File "/content/drive/MyDrive/FastFlowNet-main/demo.py", line 45, in
output = model(input_t).data
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/content/drive/MyDrive/FastFlowNet-main/models/FastFlowNet.py", line 134, in forward
cv6 = torch.index_select(self.corr(f16, f26), dim=1, index=self.index.to(f16).long())
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/content/drive/MyDrive/FastFlowNet-main/models/correlation_package/correlation.py", line 59, in forward
result = CorrelationFunction(self.pad_size, self.kernel_size, self.max_displacement,self.stride1, self.stride2, self.corr_multiply)(input1, input2)
File "/usr/local/lib/python3.7/dist-packages/torch/autograd/function.py", line 262, in call
"Legacy autograd function with non-static forward method is deprecated. "
RuntimeError: Legacy autograd function with non-static forward method is deprecated. Please use new-style autograd function with static forward method. (Example: https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function)

(base) root@autodl-container-c6d211963c-81d44250:/autodl-fs/FastFlowNet-main# python benchmark.py
Traceback (most recent call last):
File "benchmark.py", line 5, in
from models.FastFlowNet import FastFlowNet
File "/root/autodl-fs/FastFlowNet-main/models/FastFlowNet.py", line 5, in
from .correlation_package.correlation import Correlation
File "/root/autodl-fs/FastFlowNet-main/models/correlation_package/correlation.py", line 4, in
import correlation_cuda
ModuleNotFoundError: No module named 'correlation_cuda'
(base) root@autodl-container-c6d211963c-81d44250:/autodl-fs/FastFlowNet-main#

Hi there,

I tested the accuracy of the model using pretrained weights in checkpoints/ on KITTI 2015 flow training dataset, and got lower accuracy than that reported in your paper.

Using ./checkpoints/fastflownet_ft_mix.pth gives accuracy of

NOC
0.203071 0.203071 0.284329 0.284329 0.216956 0.216956 1.000000 
OCC
0.301341 0.301341 0.306706 0.306706 0.302163 0.302163 1.000000 

./checkpoints/fastflownet_ft_kitti.pth gives accuracy of

NOC
0.242663 0.242663 0.172001 0.172001 0.230589 0.230589 1.000000 
OCC
0.310403 0.310403 0.192049 0.192049 0.292272 0.292272 1.000000 

The paper shows the model could give (8.21%) Fl-all on KITTI 2015 flow training dataset.

So I'm wondering if the weights that can reproduced the accuracies in paper is included in this repo ? Please clarify.

Thanks so much !

How to using this git repo on Cuda 10, because i need to synchronize with my other project ?

Platform: xavier. jetpack4.4.1 cuda:10.2 pytorch:1.6

the correlation_packege in this repo is for pytorch0.4 which is not suite for pytorch 1.6, so I used flownet2's (https://github.com/NVIDIA/flownet2-pytorch/tree/master/networks/correlation_package)

then everything is ok.

the output is :

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:3384: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
warnings.warn("Default grid_sample and affine_grid behavior has changed "

but the result is not same in the repository.

There are some magic factors when upsample flow to higher resolution：
https://github.com/ltkong218/FastFlowNet/blob/main/models/FastFlowNet.py#L140

https://github.com/ltkong218/FastFlowNet/blob/main/models/FastFlowNet.py#L147

https://github.com/ltkong218/FastFlowNet/blob/main/models/FastFlowNet.py#L154

https://github.com/ltkong218/FastFlowNet/blob/main/models/FastFlowNet.py#L161

What's the meaninig of 0.625, 1.25, 2.5, 5? Is there any geometry motivation?

I think the factors should be 2, because when you upsample a flow to a resolution with double height and width, the flow is double due to double pixels between origin points and corresponding points.

Hi,
have you tried training on real scene such as market or subway? I have finetuned the model according to IRR-PWC by using your './checkpoints/fastflownet_ft_mix.pth' in subway real scene, but the results are much worse than flownet2's.
And I met another weird problem: during predicting, whether I multiply the optical flow result by div_flow(20), there seems no difference on the flow-png(flow result transferred to png).

Hello, I use the model to train my dataset, but the tensor size after pooling and up sampling is different, so the torch.cat command cannot be executed(such as cat6 = torch.cat([cv6, r16, flow7_up], 1)). I used the following function to process the data, and then this error occurred. Do you know how to solve this problem? Or how to deal with the problem of inconsistent tensor size?

Hi, I have been trying to train fastflownet. After reference to the IRR-PWC, I found occlusion wasn't considered in fastflownet model, So I'd like to ask: if I want to use occlusion loss, how can occlusion be added to model part?

Hi,

Thanks for sharing. What a nice work!

I have not been able to figure out why does CDDC block yield 53 channels?

Thanks,

LA Tran

Hi Lingtong, the performance of FastFlowNet is surprising. I want to deploy it on TX2. But I find there's a custom layer (correlation layer), thus the model can not be convert to onnx directly. I also try torch2tensorRT, but fail on this layer. How do you deploy it on TX2? Can you share the code about model conversion and test code on TX2 using TensorRT？

你好，我在复现您的算法时，发现精度只能达到题述水平，我确认以下几点和您的实现一致：

i am new to this area, would you relsese the code with "spatial-correlation-sampler" function for me to learn, thank you !

能不能提供一下train文件，感谢

Hi, Thanks for your great work. I am confused why flow6_up is multiplied by 0.625 here. eg: f25_w = self.warp(f25, flow6_up*0.625)

Thanks for the repo!

In case other people are struggling to get a TensorRT environment working: https://hub.docker.com/r/pullmyleg/tensorrt8_cuda11.3_pytorch1.10.2_fastflownet