tub-rip / event_based_optical_flow Goto Github PK

Hi, thank you very much for your work.

I tried with a sequence of MVSEC and it works fine, and I want to try with raw events from a mono-type event camera (.aedat4), is there any way to read this type of extension?

I also want to try the approach to format source data in hdf5, can you guide me through the hierarchical structure? And what script should I use/modify to accept this type of data?

Thanks in advance for your time and help.

Event camera : DVXplorer Mini

Hello,

I wonder if this code supports training on custom dataset? I would like to train it on the Prophesee dataset. It seems that a lot of modifications are needed.

Thanks,

Hi, thanks for such great work!

I tried to load the model file specified in this comment using pickle, but I got this error:

_pickle.UnpicklingError: A load persistent id instruction was encountered,
but no persistent_load function was specified.

Code:

import pickle

f = open("model_flow32/archive/data.pkl", "rb")
model = pickle.load(f)

If that file is no longer available, could you share it again?
Also, it would be great if you could explain how to load a pre-trained model in "run_inference.py".

Thanks for your great work here! But we can not see the code？

Hi shiba,
Thanks for the amazing work. I have tried your code, and everything works well.
But for me, I am using dvxplore, I manage to modify the setting and data loader. But meets following error:

'''
Traceback (most recent call last):
File "main.py", line 211, in
best_motion: np.ndarray = solv.optimize(batch)
File "/home/eleboss/Documents/Vid2Curve-dev-main/ev_curve/src/solver/patch_contrast_pyramid.py", line 161, in optimize
best_motion_per_scale, opt_result = self.run_scipy_over_scale(events)
File "/home/eleboss/Documents/Vid2Curve-dev-main/ev_curve/src/utils/misc.py", line 127, in wrapper
retval = func(*args, **kwargs)
File "/home/eleboss/Documents/Vid2Curve-dev-main/ev_curve/src/solver/patch_contrast_pyramid.py", line 194, in run_scipy_over_scale
opt_result = self.run_scipy(events, best_motion_per_scale)
File "/home/eleboss/Documents/Vid2Curve-dev-main/ev_curve/src/solver/patch_contrast_pyramid.py", line 303, in run_scipy
result = scipy_autograd.minimize(
File "/home/eleboss/Documents/Vid2Curve-dev-main/ev_curve/src/solver/scipy_autograd/scipy_minimize.py", line 100, in minimize
optim_res = sopt.minimize(
File "/home/eleboss/anaconda3/lib/python3.8/site-packages/scipy/optimize/_minimize.py", line 689, in minimize
res = _minimize_newtoncg(fun, x0, args, jac, hess, hessp, callback,
File "/home/eleboss/anaconda3/lib/python3.8/site-packages/scipy/optimize/_optimize.py", line 1862, in _minimize_newtoncg
sf = _prepare_scalar_function(
File "/home/eleboss/anaconda3/lib/python3.8/site-packages/scipy/optimize/_optimize.py", line 263, in _prepare_scalar_function
sf = ScalarFunction(fun, x0, args, grad, hess,
File "/home/eleboss/anaconda3/lib/python3.8/site-packages/scipy/optimize/_differentiable_functions.py", line 158, in init
self._update_fun()
File "/home/eleboss/anaconda3/lib/python3.8/site-packages/scipy/optimize/_differentiable_functions.py", line 251, in _update_fun
self._update_fun_impl()
File "/home/eleboss/anaconda3/lib/python3.8/site-packages/scipy/optimize/_differentiable_functions.py", line 155, in update_fun
self.f = fun_wrapped(self.x)
File "/home/eleboss/anaconda3/lib/python3.8/site-packages/scipy/optimize/_differentiable_functions.py", line 137, in fun_wrapped
fx = fun(np.copy(x), *args)
File "/home/eleboss/anaconda3/lib/python3.8/site-packages/scipy/optimize/_optimize.py", line 76, in call
self._compute_if_needed(x, *args)
File "/home/eleboss/anaconda3/lib/python3.8/site-packages/scipy/optimize/_optimize.py", line 70, in _compute_if_needed
fg = self.fun(x, *args)
File "/home/eleboss/Documents/Vid2Curve-dev-main/ev_curve/src/solver/scipy_autograd/torch_wrapper.py", line 38, in get_value_and_grad
loss = self.eval_func(input_var)
File "/home/eleboss/Documents/Vid2Curve-dev-main/ev_curve/src/solver/scipy_autograd/base_wrapper.py", line 109, in _eval_func
loss = self.func(input_var)
File "/home/eleboss/Documents/Vid2Curve-dev-main/ev_curve/src/solver/patch_contrast_pyramid.py", line 304, in
lambda x: self.objective_scipy(x, events, coarser_motion),
File "/home/eleboss/Documents/Vid2Curve-dev-main/ev_curve/src/solver/patch_contrast_pyramid.py", line 453, in objective_scipy
loss = self.calculate_cost(
File "/home/eleboss/Documents/Vid2Curve-dev-main/ev_curve/src/solver/patch_contrast_base.py", line 281, in calculate_cost
arg_cost = self.get_arg_for_cost(events, warp, motion_model, coarse_flow)
File "/home/eleboss/Documents/Vid2Curve-dev-main/ev_curve/src/solver/patch_contrast_base.py", line 308, in get_arg_for_cost
backward_events, backward_feat = self.warper.warp_event(
File "/home/eleboss/Documents/Vid2Curve-dev-main/ev_curve/src/warp.py", line 188, in warp_event
return self.warp_event_from_optical_flow(events, motion, ref_time)
File "/home/eleboss/Documents/Vid2Curve-dev-main/ev_curve/src/warp.py", line 305, in warp_event_from_optical_flow
warped_torch[..., 0] = event[..., 0] - dt * torch.gather(flow_flat[:, 0], 1, _ind)
RuntimeError: index 409412 is out of bounds for dimension 1 with size 307200

'''

Could you also share the setting for high resolution like DSCE or do you have any hint about this error?

Thanks for your amazing work. I have two questions:

1. There are different “motion model” when warping events, such as “dense-flow”, “dense-flow-voxel”. What do they represent respectively? How should I choose?

2. When applying to unsupervised-learning methods, as you mentioned in the paper, the EV-FlowNet, I’d like to take the event volume as the input, as proposed in Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotion by Alex Zihao Zhu. So I need to center crop the event volume to 256x256, then the predicted flow is 256x256x2. But the resolution of event sensor is 346x256.What should I do in the warp function? Do I need to choose the events triggered at x∈[45,300] and y∈[2,257]? Because the predicted flow only covers those regions. Am I right?

Thanks for the great work!

I'm trying to run the method on some higher resolution data. But I'm getting terrible results. Do you have any suggestion as to what configuration to use?

Resolution: 720 x 1280
termination loss value: 8152.2227

Further, what are the requirement for "crop_height" and "crop_width" in the config? I'm getting errors here where i is outside motion0. It'll run if I use a multiple of the values in the given config.

Very nice work. I have a question about section 3.3 of the article, may I ask how equation (7) is obtained?

Hi there, I've runned

python3 main.py --config_file ./configs/mvsec_indoor_no_timeaware.yaml

without changing the original config you provide.

Seems like it is computing optical flow between ind1 and ind2 but is not iterating through the full dataset. Is there a parameter in yaml config to do it? Also, is there a config to output the actual values of optical flow in a file?

One last question, in the paper the multi-scale approach use the information from previous batch to initialize the values of the different scales in the next batch, is this implemented?

Thanks in advance

Hi, Shiba, Why do you use a fixed number of events in optimization? As showed in config files (.yml files), the 'n_events_per_batch' is fixed, and events closer to the start time are abandoned if events are more than 'n_events_per_batch' according to the codes. In fact we want to predict the flow with fixed time interval instead of fixed number of events. According to the loss function below, the loss is computed with two iwe, which are in the shape of h,w, which is fixed for any 'n_events_per_batch'.

I just want to discuss about the reason behind it. Danke!

Hello, thank you for your excellent work!

I am currently attempting to run this method on the DSEC dataset. I have tested a dataset containing a total of 957,721 events. The test results are satisfactory, but the model's runtime is approximately 11 minutes. I am unsure if this runtime is normal or if there is an issue with my operation at some step. I look forward to your response. Below is my configuration file:

solver:
  method: "pyramidal_patch_contrast_maximization"
  time_aware: False
  patch:
    initialize: "random"
    scale: 5
    crop_height: 480
    crop_width: 640
    filter_type: "bilinear"
  motion_model: "2d-translation"
  warp_direction: "first"
  parameters:
    - "trans_x"
    - "trans_y"
  cost: "hybrid"
  outer_padding: 0
  cost_with_weight:
    multi_focal_normalized_gradient_magnitude: 1.
    total_variation: 0.01
  iwe:
    method: "bilinear_vote"
    blur_sigma: 1

optimizer:
  n_iter: 40
  method: "Newton-CG"
  max_iter: 25
  parameters:
    trans_x:
      min: -150
      max: 150
    trans_y:
      min: -150
      max: 150

Does this network output data in.flo format？If it does, could you upload the Optical flows resualts (in.flo format) of DSEC dataset and MVSEC dataset?

Cause i want to use the the Optical flows resualts (in.flo format) directly for my research. Thanks!

Hi Shiba,

It would be nice if you could release codes for ''Other solver (especially DNN)''.

Greetings,

Hi,
Thanks for your amazing work. Can you provide some pretrained models and checkpoint files and the config files for DSEC dataset?

Hi, thanks for the nice work!!
How about the runtime of optimization proposed in this paper? e.g., process single sequence of MVSEC dataset.

Hi,

Firstly, thank you for your work.

Why is that the evaluation metrics are computed with the mask of the original events instead of the expanded events (due to the 30k requirement)? Do the results in your paper reflect this (both qualitative results and the illustrative figures)?

When trying to run burgers I get the following error:

(eventOF) lupus@DESKTOP-IFAQBRF:~/event_based_optical_flow$ python ./main.py --config_file ./configs/mvsec_indoor_burgers.yaml
Torch scatter needs to run some special interpolation.
2022-12-02 11:49:33,906 - src.data_loader.base - INFO - Loading directory in /home/lupus/event_based_optical_flow/datasets/MVSEC/hdf5/MVSEC
2022-12-02 11:49:33,906 - src.data_loader.base - INFO - No undistortion.
2022-12-02 11:49:33,906 - src.data_loader.mvsec - INFO - Use sequence indoor_flying1
2022-12-02 11:49:33,907 - src.data_loader.mvsec - INFO - Undistort events = False
2022-12-02 11:49:36,851 - src.data_loader.mvsec - INFO - Loading ground truth flow /home/lupus/event_based_optical_flow/datasets/MVSEC/gt_flow/indoor_flying1_gt_flow_dist.npz
2022-12-02 11:49:45,186 - src.data_loader.mvsec - INFO - Use only valid frames.
2022-12-02 11:49:45,770 - src.data_loader.mvsec - WARNING - directly load calib_param is not implemented!! please use rectify instead.
2022-12-02 11:49:45,778 - src.solver.patch_contrast_pyramid - INFO - Pyramidal patch.
2022-12-02 11:49:45,934 - src.solver.base - INFO - Load hybrid cost
2022-12-02 11:49:45,945 - src.costs.hybrid - INFO - Log functions are mix of {'multi_focal_normalized_gradient_magnitude': 1.0, 'total_variation': 0.01}
2022-12-02 11:49:46,129 - src.warp - INFO - Set camera matrix K.
2022-12-02 11:49:46,132 - src.feature_calculator - WARNING - Feature calculation is disabled in this source code.
2022-12-02 11:49:46,132 - src.feature_calculator - WARNING - Feature calculation is disabled in this source code.
2022-12-02 11:49:46,139 - src.solver.base - INFO - Setup time-aware parameters
2022-12-02 11:49:46,140 - src.solver.base - INFO - No scaling before upwind
2022-12-02 11:49:46,141 - src.solver.base - INFO - Configuration:
    self.normalize_t_in_batch = True
2022-12-02 11:49:46,142 - src.solver.base - INFO - Configuration:
    {'n_iter': 40, 'method': 'Newton-CG', 'max_iter': 25, 'parameters': {'trans_x': {'min': -150, 'max': 150}, 'trans_y': {'min': -150, 'max': 150}}}
    {'method': 'pyramidal_patch_contrast_maximization', 'time_aware': True, 'time_bin': 10, 'flow_interpolation': 'burgers', 't0_flow_location': 'middle', 'patch': {'initialize': 'random', 'scale': 5, 'crop_height': 256, 'crop_width': 336, 'filter_type': 'bilinear'}, 'motion_model': '2d-translation', 'warp_direction': 'first', 'parameters': ['trans_x', 'trans_y'], 'cost': 'hybrid', 'outer_padding': 0, 'cost_with_weight': {'multi_focal_normalized_gradient_magnitude': 1.0, 'total_variation': 0.01}, 'iwe': {'method': 'bilinear_vote', 'blur_sigma': 1}}
2022-12-02 11:49:46,162 - src.feature_calculator - WARNING - Feature calculation is disabled in this source code.
2022-12-02 11:49:46,162 - src.feature_calculator - WARNING - Feature calculation is disabled in this source code.
2022-12-02 11:49:46,165 - src.feature_calculator - WARNING - Feature calculation is disabled in this source code.
2022-12-02 11:49:46,165 - src.feature_calculator - WARNING - Feature calculation is disabled in this source code.
2022-12-02 11:49:46,175 - src.feature_calculator - WARNING - Feature calculation is disabled in this source code.
2022-12-02 11:49:46,175 - src.feature_calculator - WARNING - Feature calculation is disabled in this source code.
2022-12-02 11:49:46,176 - src.feature_calculator - WARNING - Feature calculation is disabled in this source code.
2022-12-02 11:49:46,177 - src.feature_calculator - WARNING - Feature calculation is disabled in this source code.
2022-12-02 11:49:46,183 - __main__ - INFO - Single-frame optimization
2022-12-02 11:49:46,426 - src.solver.patch_contrast_pyramid - INFO - Start optimization.
2022-12-02 11:49:46,426 - src.solver.patch_contrast_pyramid - INFO - DoF is 680
2022-12-02 11:49:46,438 - src.solver.patch_contrast_pyramid - INFO - Scale 1
2022-12-02 11:49:46,439 - src.solver.patch_contrast_base - INFO - random initialization
/home/lupus/event_based_optical_flow/src/solver/scipy_autograd/scipy_minimize.py:100: OptimizeWarning: Unknown solver options: gtol
  optim_res = sopt.minimize(
2022-12-02 11:49:47,653 - src.solver.patch_contrast_pyramid - INFO - loss = tensor(4.7517, dtype=torch.float64, grad_fn=<AddBackward0>)
Traceback (most recent call last):
  File "/home/lupus/event_based_optical_flow/./main.py", line 179, in <module>
    best_motion: np.ndarray = solv.optimize(batch)
  File "/home/lupus/event_based_optical_flow/src/solver/patch_contrast_pyramid.py", line 161, in optimize
    best_motion_per_scale, opt_result = self.run_scipy_over_scale(events)
  File "/home/lupus/event_based_optical_flow/src/utils/misc.py", line 127, in wrapper
    retval = func(*args, **kwargs)
  File "/home/lupus/event_based_optical_flow/src/solver/patch_contrast_pyramid.py", line 194, in run_scipy_over_scale
    opt_result = self.run_scipy(events, best_motion_per_scale)
  File "/home/lupus/event_based_optical_flow/src/solver/patch_contrast_pyramid.py", line 303, in run_scipy
    result = scipy_autograd.minimize(
  File "/home/lupus/event_based_optical_flow/src/solver/scipy_autograd/scipy_minimize.py", line 100, in minimize
    optim_res = sopt.minimize(
  File "/home/lupus/anaconda3/envs/eventOF/lib/python3.10/site-packages/scipy/optimize/_minimize.py", line 696, in minimize
    res = _minimize_newtoncg(fun, x0, args, jac, hess, hessp, callback,
  File "/home/lupus/anaconda3/envs/eventOF/lib/python3.10/site-packages/scipy/optimize/_optimize.py", line 1849, in _minimize_newtoncg
    sf = _prepare_scalar_function(
  File "/home/lupus/anaconda3/envs/eventOF/lib/python3.10/site-packages/scipy/optimize/_optimize.py", line 263, in _prepare_scalar_function
    sf = ScalarFunction(fun, x0, args, grad, hess,
  File "/home/lupus/anaconda3/envs/eventOF/lib/python3.10/site-packages/scipy/optimize/_differentiable_functions.py", line 158, in __init__
    self._update_fun()
  File "/home/lupus/anaconda3/envs/eventOF/lib/python3.10/site-packages/scipy/optimize/_differentiable_functions.py", line 251, in _update_fun
    self._update_fun_impl()
  File "/home/lupus/anaconda3/envs/eventOF/lib/python3.10/site-packages/scipy/optimize/_differentiable_functions.py", line 155, in update_fun
    self.f = fun_wrapped(self.x)
  File "/home/lupus/anaconda3/envs/eventOF/lib/python3.10/site-packages/scipy/optimize/_differentiable_functions.py", line 137, in fun_wrapped
    fx = fun(np.copy(x), *args)
  File "/home/lupus/anaconda3/envs/eventOF/lib/python3.10/site-packages/scipy/optimize/_optimize.py", line 76, in __call__
    self._compute_if_needed(x, *args)
  File "/home/lupus/anaconda3/envs/eventOF/lib/python3.10/site-packages/scipy/optimize/_optimize.py", line 70, in _compute_if_needed
    fg = self.fun(x, *args)
  File "/home/lupus/event_based_optical_flow/src/solver/scipy_autograd/torch_wrapper.py", line 40, in get_value_and_grad
    grads = torch.autograd.grad(loss, input_var_grad)
  File "/home/lupus/anaconda3/envs/eventOF/lib/python3.10/site-packages/torch/autograd/__init__.py", line 300, in grad
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: could not compute gradients for some functions
(eventOF) lupus@DESKTOP-IFAQBRF:~/event_based_optical_flow$ python ./main.py --config_file ./configs/mvsec_indoo