uzh-rpg / rpg_vid2e Goto Github PK

Open source implementation of CVPR 2020 "Video to Events: Recycling Video Dataset for Event Cameras"

License: GNU General Public License v3.0

CMake 0.75% C++ 13.58% Python 78.68% Cuda 7.00%

rpg_vid2e's Introduction

Video to Events: Recycling Video Datasets for Event Cameras

This repository contains code that implements video to events conversion as described in Gehrig et al. CVPR'20 and the used dataset. The paper can be found here

If you use this code in an academic context, please cite the following work:

Daniel Gehrig, Mathias Gehrig, Javier Hidalgo-Carrió, Davide Scaramuzza, "Video to Events: Recycling Video Datasets for Event Cameras", The Conference on Computer Vision and Pattern Recognition (CVPR), 2020

@InProceedings{Gehrig_2020_CVPR,
  author = {Daniel Gehrig and Mathias Gehrig and Javier Hidalgo-Carri\'o and Davide Scaramuzza},
  title = {Video to Events: Recycling Video Datasets for Event Cameras},
  booktitle = {{IEEE} Conf. Comput. Vis. Pattern Recog. (CVPR)},
  month = {June},
  year = {2020}
}

News

We now support frame interpolation done by FILM.
We release a web app and interactive demo which generates events and converts your webcam to events. Try it out here.
We now also release new python bindings for esim with GPU support. Details are here

Web App and Interactive Demo

Try out our the interactive demo and webcam support here.

Dataset

The synthetic N-Caltech101 dataset, as well as video sequences used for event conversion can be found here. For each sample of each class it contains events in the form class/image_%04d.npz and images in the form class/image_%05d/images/image_%05d.png, as well as the corresponding timestamps of the images in class/image_%04d/timestamps.txt.

Installation

Clone the repo recursively with submodules

git clone [email protected]:uzh-rpg/rpg_vid2e.git --recursive

Installation

First download the FILM checkpoint, and move it to the current root

    wget https://rpg.ifi.uzh.ch/data/VID2E/pretrained_models.zip -O /tmp/temp.zip
    unzip /tmp/temp.zip -d rpg_vid2e/
    rm -rf /tmp/temp.zip

make sure to install the following * Anaconda Python 3.9 * CUDA Toolkit 11.2.1 * cuDNN 8.1.0

conda create --name vid2e python=3.9
conda activate vid2e
pip install -r rpg_vid2e/requirements.txt
conda install -y -c conda-forge pybind11 matplotlib
conda install -y pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch

Build the python bindings for ESIM

pip install rpg_vid2e/esim_py/

Build the python bindings with GPU support with

pip install rpg_vid2e/esim_torch/

Adaptive Upsampling

This package provides code for adaptive upsampling with frame interpolation based on Super-SloMo

Consult the README for detailed instructions and examples.

esim_py

This package exposes python bindings for ESIM which can be used within a training loop.

For detailed instructions and example consult the README

esim_torch

This package exposes python bindings for ESIM with GPU support.

For detailed instructions and example consult the README

Example

To run an example, first upsample the example videos

device=cpu
# device=cuda:0
python upsampling/upsample.py --input_dir=example/original --output_dir=example/upsampled --device=$device

This will generate upsampling/upsampled with in the example/upsampled folder. To generate events, use

python esim_torch/generate_events.py --input_dir=example/upsampled \
                                     --output_dir=example/events \
                                     --contrast_threshold_neg=0.2 \
                                     --contrast_threshold_pos=0.2 \
                                     --refractory_period_ns=0

rpg_vid2e's People

Contributors

Stargazers

Watchers

rpg_vid2e's Issues

When install esim_py, I meet error, but it still hint me successfully installed. Do I successfully install esim_py？

Processing /root/rpg_vid2e/esim_py
Building wheels for collected packages: esim-py
  Building wheel for esim-py (setup.py): started

  running build
  running build_ext
  CMake Error: The current CMakeCache.txt directory /tmp/pip-req-build-350x_psj/build/temp.linux-x86_64-3.7/CMakeCache.txt is different than the directory /root/rpg_vid2e/esim_py/build/temp.linux-x86_64-3.7 where CMakeCache.txt was created. This may result in binaries being created in the wrong place. If you are not sure, reedit the CMakeCache.txt
  CMake Error: The source "/tmp/pip-req-build-350x_psj/CMakeLists.txt" does not match the source "/root/rpg_vid2e/esim_py/CMakeLists.txt" used to generate cache.  Re-run cmake with a different source directory.
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-req-build-350x_psj/setup.py", line 62, in <module>
      zip_safe=False,
    File "/opt/conda/envs/vid2e/lib/python3.7/site-packages/setuptools/__init__.py", line 153, in setup
      return distutils.core.setup(**attrs)
    File "/opt/conda/envs/vid2e/lib/python3.7/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/opt/conda/envs/vid2e/lib/python3.7/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/opt/conda/envs/vid2e/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/opt/conda/envs/vid2e/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 299, in run
      self.run_command('build')
    File "/opt/conda/envs/vid2e/lib/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/conda/envs/vid2e/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/opt/conda/envs/vid2e/lib/python3.7/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/opt/conda/envs/vid2e/lib/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/conda/envs/vid2e/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/tmp/pip-req-build-350x_psj/setup.py", line 31, in run
      self.build_extension(ext)
    File "/tmp/pip-req-build-350x_psj/setup.py", line 49, in build_extension
      subprocess.check_call(['cmake', ext.sourcedir] + cmake_args, cwd=self.build_temp, env=env)
    File "/opt/conda/envs/vid2e/lib/python3.7/subprocess.py", line 363, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['cmake', '/tmp/pip-req-build-350x_psj', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/tmp/pip-req-build-350x_psj/build/lib.linux-x86_64-3.7', '-DPYTHON_EXECUTABLE=/opt/conda/envs/vid2e/bin/python', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1.
  ----------------------------------------
  ERROR: Failed building wheel for esim-py
  Running setup.py clean for esim-py
Failed to build esim-py
Installing collected packages: esim-py
    Running setup.py install for esim-py: started
    Running setup.py install for esim-py: finished with status 'done'
  DEPRECATION: esim-py was installed using the legacy 'setup.py install' method, because a wheel could not be built for it. A possible replacement is to fix the wheel build issue reported above. You can find discussion regarding this at https://github.com/pypa/pip/issues/8368.
Successfully installed esim-py-0.0.1

Can you please provide or point to the script for applying saccadic motion to images ?

dataset problem

I have download dataset from weblink http://rpg.ifi.uzh.ch/data/VID2E/ncaltech_syn_images.zip,
but didn't find the video sequences used for event conversion.
After unziped ncaltech_syn_images.zip downloaded from the former weblink, I obtained folder sim-N-Caltech101, there is only /sim-N-Caltech101/class/image_%04d.npz files.
Could anyone help with how to find the images( in the form class/image_%05d/images/image_%05d.png), as well as the corresponding timestamps of the images( in class/image_%04d/timestamps.txt)?

simulation error in ngspice Fatal error: BSIM3v32: mosfet cmosn, model m3: Effective channel length <= 0

I was trying to simulate my circuit on ngspice which i have been made on kicad. Now is show me error like this:

Fatal error: BSIM3v32: mosfet cmosn, model m3: Effective channel length <= 0
doAnalyses: no such parameter on this device

run simulation(s) aborted

Question about generating events from stamped image sequence

Hi,
I have a question about the function "generateFromStampedImageSequence" in esim_py.

# generate events from list of images and timestamps 
events_list_of_images = esim.generateFromStampedImageSequence(
    list_of_image_files,   # list of absolute paths to images
    list_of_timestamps     # list of timestamps in ascending order
)

I successfully installed esim_py and could generate events with "generateFromStampedImageSequence".
But when I put image list and timestamp of length 2, it produces empty list of events.
For the detail explanation, if list_of_image_files = [img1, img2], and list_of_timestamps is the .txt file with two timestamps (e.g. 0.000000000 and 0.001865671), events_list_of_images produces [].

It works fine when the length of image sequence is higher than 2.
Is there any reason for that or am I the only one having this issue?

Thanks in advance!

Improve performance of upsampler.

Hi,

I am generating events from a very large Dataset, and the upsampler seems to be a bottle neck. Do you have some suggestions or advice on how can I improve the speed of upsampler?

Best regards,
Dave

ModuleNotFoundError: No module named 'esim_torch'

I successfully compiled and installed the esim_torch package with the commands:

cd rpg_vid2e/esim_torch
pip install .

However, I got the following errors while trying python test/test.py:

Traceback (most recent call last):
File "/xxx/rpg_vid2e/esim_torch/test/test.py", line 7, in
import esim_torch
ModuleNotFoundError: No module named 'esim_torch'

GOOGLE COLAB: connectd.x86_64-ubuntu16.04 download failed!

Hi everyone,

Thanks for making ESIM WEB APP available on Google Colab.

I tried to run the first code snippet as you suggested

!curl -LkO https://raw.githubusercontent.com/remoteit/installer/master/scripts/auto-install.sh
! chmod +x ./auto-install.sh
! sudo ./auto-install.sh

The following is the output:

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 12130  100 12130    0     0  68531      0 --:--:-- --:--:-- --:--:-- 68531
The connectd package has been deprecated.  We do not recommend this package for production.
Please find the new instructions for single-device installation at:
https://link.remote.it/support/rpi-linux-quick-install

To prepare an OS image for mass production that will register a pre-defined set of Remote.It
Services, please see: https://link.remote.it/docs/oem-overview .

Press any key to continue.  Press ^C to terminate this script.
a

 curl found 
********************************************************
remote.it platform and binary tester version 2.6.39 
Current directory /content
Thu 23 Feb 2023 10:57:34 AM UTC
********************************************************
Linux 3b76f5719e7a 5.10.147+ #1 SMP Sat Dec 10 16:00:40 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Detected architecture is x86_64
/usr/bin/dpkg
Debian OS detected.
amd64 architecture detected.

Downloading and testing connectd.x86_64-ubuntu16.04...
connectd.x86_64-ubuntu16.04 download failed!

Do you know a fix to this? I am afraid the problem comes from the fact that Ubuntu version of Google Colab is now 20.04 and not 16.04 anymore

Youssef

error building esim_torch

I met a problem when building cuda version. Could you please offer some help? The error message is as follows. Thanks for your time.

Processing ./esim_torch
Building wheels for collected packages: esim-cuda
Building wheel for esim-cuda (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: /home/cll/anaconda3/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-g6ejhw3n/setup.py'"'"'; file='"'"'/tmp/pip-req-build-g6ejhw3n/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-oqcutjzb
cwd: /tmp/pip-req-build-g6ejhw3n/
Complete output (14 lines):
running bdist_wheel
running build
running build_ext
building 'esim_cuda' extension
creating /tmp/pip-req-build-g6ejhw3n/build
creating /tmp/pip-req-build-g6ejhw3n/build/temp.linux-x86_64-3.8
Emitting ninja build file /tmp/pip-req-build-g6ejhw3n/build/temp.linux-x86_64-3.8/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
1.10.2
creating build/lib.linux-x86_64-3.8
g++ -pthread -shared -B /home/cll/anaconda3/compiler_compat -L/home/cll/anaconda3/lib -Wl,-rpath=/home/cll/anaconda3/lib -Wl,--no-as-needed -Wl,--sysroot=/ /tmp/pip-req-build-g6ejhw3n/build/temp.linux-x86_64-3.8/esim_cuda_kernel.o -L/home/cll/anaconda3/lib/python3.8/site-packages/torch/lib -L/usr/local/cuda/lib64 -lc10 -ltorch -ltorch_cpu -ltorch_python -lcudart -lc10_cuda -ltorch_cuda_cu -ltorch_cuda_cpp -o build/lib.linux-x86_64-3.8/esim_cuda.cpython-38-x86_64-linux-gnu.so
g++: error: /tmp/pip-req-build-g6ejhw3n/build/temp.linux-x86_64-3.8/esim_cuda_kernel.o: No such file or directory
error: command 'g++' failed with exit status 1

ERROR: Failed building wheel for esim-cuda
Running setup.py clean for esim-cuda
Failed to build esim-cuda
Installing collected packages: esim-cuda
Running setup.py install for esim-cuda ... error
ERROR: Command errored out with exit status 1:
command: /home/cll/anaconda3/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-g6ejhw3n/setup.py'"'"'; file='"'"'/tmp/pip-req-build-g6ejhw3n/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-gmmfujvl/install-record.txt --single-version-externally-managed --compile --install-headers /home/cll/anaconda3/include/python3.8/esim-cuda
cwd: /tmp/pip-req-build-g6ejhw3n/
Complete output (14 lines):
running install
running build
running build_ext
building 'esim_cuda' extension
creating /tmp/pip-req-build-g6ejhw3n/build
creating /tmp/pip-req-build-g6ejhw3n/build/temp.linux-x86_64-3.8
Emitting ninja build file /tmp/pip-req-build-g6ejhw3n/build/temp.linux-x86_64-3.8/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
1.10.2
creating build/lib.linux-x86_64-3.8
g++ -pthread -shared -B /home/cll/anaconda3/compiler_compat -L/home/cll/anaconda3/lib -Wl,-rpath=/home/cll/anaconda3/lib -Wl,--no-as-needed -Wl,--sysroot=/ /tmp/pip-req-build-g6ejhw3n/build/temp.linux-x86_64-3.8/esim_cuda_kernel.o -L/home/cll/anaconda3/lib/python3.8/site-packages/torch/lib -L/usr/local/cuda/lib64 -lc10 -ltorch -ltorch_cpu -ltorch_python -lcudart -lc10_cuda -ltorch_cuda_cu -ltorch_cuda_cpp -o build/lib.linux-x86_64-3.8/esim_cuda.cpython-38-x86_64-linux-gnu.so
g++: error: /tmp/pip-req-build-g6ejhw3n/build/temp.linux-x86_64-3.8/esim_cuda_kernel.o: No such file or directory
error: command 'g++' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /home/cll/anaconda3/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-g6ejhw3n/setup.py'"'"'; file='"'"'/tmp/pip-req-build-g6ejhw3n/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-gmmfujvl/install-record.txt --single-version-externally-managed --compile --install-headers /home/cll/anaconda3/include/python3.8/esim-cuda Check the logs for full command output.

Unsuccessful of command pip install rpg_vid2e/esim_py/

Hi,

When i tried to execute the command 'pip install rpg_vid2e/esim_py/'. It gave the following error.'
Is there any suggestion? (i am working in ubuntu)
rocessing ./rpg_vid2e/esim_py
Preparing metadata (setup.py) ... done
Building wheels for collected packages: esim-py
Building wheel for esim-py (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [83 lines of output]
running bdist_wheel
running build
running build_ext
-- The C compiler identification is GNU 9.4.0
-- The CXX compiler identification is GNU 9.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found PythonInterp: /home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/bin/python (found version "3.9.12")
-- Found PythonLibs: /home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/libpython3.9.so
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Found pybind11: /home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/include (found version "2.6.1" )
CMake Error at CMakeLists.txt:5 (find_package):
By not providing "FindOpenCV.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "OpenCV", but
CMake did not find one.

    Could not find a package configuration file provided by "OpenCV" with any
    of the following names:
  
      OpenCVConfig.cmake
      opencv-config.cmake
  
    Add the installation prefix of "OpenCV" to CMAKE_PREFIX_PATH or set
    "OpenCV_DIR" to a directory containing one of the above files.  If "OpenCV"
    provides a separate development package or SDK, be sure it has been
    installed.
  
  
  -- Configuring incomplete, errors occurred!
  See also "/home/dtu-neurorobotics-desk2/Research_Work/Video2Events_ESIM/rpg_vid2e/esim_py/build/temp.linux-x86_64-cpython-39/CMakeFiles/CMakeOutput.log".
  Traceback (most recent call last):
    File "<string>", line 2, in <module>
    File "<pip-setuptools-caller>", line 34, in <module>
    File "/home/dtu-neurorobotics-desk2/Research_Work/Video2Events_ESIM/rpg_vid2e/esim_py/setup.py", line 53, in <module>
      setup(
    File "/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/__init__.py", line 87, in setup
      return distutils.core.setup(**attrs)
    File "/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
      return run_commands(dist)
    File "/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
      dist.run_commands()
    File "/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 973, in run_commands
      self.run_command(cmd)
    File "/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command
      super().run_command(command)
    File "/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 992, in run_command
      cmd_obj.run()
    File "/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 299, in run
      self.run_command('build')
    File "/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
      self.distribution.run_command(command)
    File "/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command
      super().run_command(command)
    File "/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 992, in run_command
      cmd_obj.run()
    File "/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/command/build.py", line 24, in run
      super().run()
    File "/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/command/build.py", line 132, in run
      self.run_command(cmd_name)
    File "/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
      self.distribution.run_command(command)
    File "/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command
      super().run_command(command)
    File "/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 992, in run_command
      cmd_obj.run()
    File "/home/dtu-neurorobotics-desk2/Research_Work/Video2Events_ESIM/rpg_vid2e/esim_py/setup.py", line 31, in run
      self.build_extension(ext)
    File "/home/dtu-neurorobotics-desk2/Research_Work/Video2Events_ESIM/rpg_vid2e/esim_py/setup.py", line 49, in build_extension
      subprocess.check_call(['cmake', ext.sourcedir] + cmake_args, cwd=self.build_temp, env=env)
    File "/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/subprocess.py", line 373, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['cmake', '/home/dtu-neurorobotics-desk2/Research_Work/Video2Events_ESIM/rpg_vid2e/esim_py', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/home/dtu-neurorobotics-desk2/Research_Work/Video2Events_ESIM/rpg_vid2e/esim_py/build/lib.linux-x86_64-cpython-39', '-DPYTHON_EXECUTABLE=/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/bin/python', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1.
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for esim-py
Running setup.py clean for esim-py
Failed to build esim-py
Installing collected packages: esim-py
Running setup.py install for esim-py ... error
error: subprocess-exited-with-error

× Running setup.py install for esim-py did not run successfully.
│ exit code: 1
╰─> [87 lines of output]
running install
/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running build
running build_ext
-- The C compiler identification is GNU 9.4.0
-- The CXX compiler identification is GNU 9.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found PythonInterp: /home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/bin/python (found version "3.9.12")
-- Found PythonLibs: /home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/libpython3.9.so
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Found pybind11: /home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/include (found version "2.6.1" )
CMake Error at CMakeLists.txt:5 (find_package):
By not providing "FindOpenCV.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "OpenCV", but
CMake did not find one.

    Could not find a package configuration file provided by "OpenCV" with any
    of the following names:
  
      OpenCVConfig.cmake
      opencv-config.cmake
  
    Add the installation prefix of "OpenCV" to CMAKE_PREFIX_PATH or set
    "OpenCV_DIR" to a directory containing one of the above files.  If "OpenCV"
    provides a separate development package or SDK, be sure it has been
    installed.
  
  
  -- Configuring incomplete, errors occurred!
  See also "/home/dtu-neurorobotics-desk2/Research_Work/Video2Events_ESIM/rpg_vid2e/esim_py/build/temp.linux-x86_64-cpython-39/CMakeFiles/CMakeOutput.log".
  Traceback (most recent call last):
    File "<string>", line 2, in <module>
    File "<pip-setuptools-caller>", line 34, in <module>
    File "/home/dtu-neurorobotics-desk2/Research_Work/Video2Events_ESIM/rpg_vid2e/esim_py/setup.py", line 53, in <module>
      setup(
    File "/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/__init__.py", line 87, in setup
      return distutils.core.setup(**attrs)
    File "/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
      return run_commands(dist)
    File "/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
      dist.run_commands()
    File "/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 973, in run_commands
      self.run_command(cmd)
    File "/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command
      super().run_command(command)
    File "/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 992, in run_command
      cmd_obj.run()
    File "/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/command/install.py", line 68, in run
      return orig.install.run(self)
    File "/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/command/install.py", line 692, in run
      self.run_command('build')
    File "/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
      self.distribution.run_command(command)
    File "/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command
      super().run_command(command)
    File "/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 992, in run_command
      cmd_obj.run()
    File "/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/command/build.py", line 24, in run
      super().run()
    File "/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/command/build.py", line 132, in run
      self.run_command(cmd_name)
    File "/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
      self.distribution.run_command(command)
    File "/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command
      super().run_command(command)
    File "/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 992, in run_command
      cmd_obj.run()
    File "/home/dtu-neurorobotics-desk2/Research_Work/Video2Events_ESIM/rpg_vid2e/esim_py/setup.py", line 31, in run
      self.build_extension(ext)
    File "/home/dtu-neurorobotics-desk2/Research_Work/Video2Events_ESIM/rpg_vid2e/esim_py/setup.py", line 49, in build_extension
      subprocess.check_call(['cmake', ext.sourcedir] + cmake_args, cwd=self.build_temp, env=env)
    File "/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/lib/python3.9/subprocess.py", line 373, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['cmake', '/home/dtu-neurorobotics-desk2/Research_Work/Video2Events_ESIM/rpg_vid2e/esim_py', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/home/dtu-neurorobotics-desk2/Research_Work/Video2Events_ESIM/rpg_vid2e/esim_py/build/lib.linux-x86_64-cpython-39', '-DPYTHON_EXECUTABLE=/home/dtu-neurorobotics-desk2/anaconda3/envs/vid2e/bin/python', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1.
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> esim-py

esim_py and esim_torch are not working on Windows 10 nor Ubuntu

Hi,

I followed the same steps in READMe and I could not install esim_py and esim_torch. I tried both Windows 10 nor Ubuntu and still getting errors.

Kindly advise.

error buiding wheel for esim_py

I got a confusing error when buiding wheel for esim_py.
I followed the README:
$ cd esim_py
$ pip install .

And I got the following error message:

Building wheels for collected packages: esim-py
Building wheel for esim-py (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: /home/bob/anaconda3/envs/vid2e/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-jg63bbet/setup.py'"'"'; file='"'"'/tmp/pip-req-build-jg63bbet/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-s032vb4j
cwd: /tmp/pip-req-build-jg63bbet/
Complete output (44 lines):
running bdist_wheel
running build
running build_ext
CMake Error: The current CMakeCache.txt directory /tmp/pip-req-build-jg63bbet/build/temp.linux-x86_64-3.9/CMakeCache.txt is different than the directory /media/bob/Seagate/z_event_camera/rpg_vid2e-master/esim_py/build/temp.linux-x86_64-3.9 where CMakeCache.txt was created. This may result in binaries being created in the wrong place. If you are not sure, reedit the CMakeCache.txt
CMake Error: The source "/tmp/pip-req-build-jg63bbet/CMakeLists.txt" does not match the source "/media/bob/Seagate/z_event_camera/rpg_vid2e-master/esim_py/CMakeLists.txt" used to generate cache. Re-run cmake with a different source directory.
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-req-build-jg63bbet/setup.py", line 53, in
setup(
File "/home/bob/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/init.py", line 87, in setup
return distutils.core.setup(**attrs)
File "/home/bob/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 148, in setup
return run_commands(dist)
File "/home/bob/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
dist.run_commands()
File "/home/bob/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
self.run_command(cmd)
File "/home/bob/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/dist.py", line 1214, in run_command
super().run_command(command)
File "/home/bob/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
cmd_obj.run()
File "/home/bob/anaconda3/envs/vid2e/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 299, in run
self.run_command('build')
File "/home/bob/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/home/bob/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/dist.py", line 1214, in run_command
super().run_command(command)
File "/home/bob/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
cmd_obj.run()
File "/home/bob/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/command/build.py", line 135, in run
self.run_command(cmd_name)
File "/home/bob/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/home/bob/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/dist.py", line 1214, in run_command
super().run_command(command)
File "/home/bob/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
cmd_obj.run()
File "/tmp/pip-req-build-jg63bbet/setup.py", line 31, in run
self.build_extension(ext)
File "/tmp/pip-req-build-jg63bbet/setup.py", line 49, in build_extension
subprocess.check_call(['cmake', ext.sourcedir] + cmake_args, cwd=self.build_temp, env=env)
File "/home/bob/anaconda3/envs/vid2e/lib/python3.9/subprocess.py", line 373, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '/tmp/pip-req-build-jg63bbet', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/tmp/pip-req-build-jg63bbet/build/lib.linux-x86_64-3.9', '-DPYTHON_EXECUTABLE=/home/bob/anaconda3/envs/vid2e/bin/python', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1.

ERROR: Failed building wheel for esim-py
Running setup.py clean for esim-py
Failed to build esim-py
Installing collected packages: esim-py
Running setup.py install for esim-py ... error
ERROR: Command errored out with exit status 1:
command: /home/bob/anaconda3/envs/vid2e/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-jg63bbet/setup.py'"'"'; file='"'"'/tmp/pip-req-build-jg63bbet/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-ua79m_eq/install-record.txt --single-version-externally-managed --compile --install-headers /home/bob/anaconda3/envs/vid2e/include/python3.9/esim-py
cwd: /tmp/pip-req-build-jg63bbet/
Complete output (82 lines):
running install
/home/bob/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running build
running build_ext
-- The C compiler identification is GNU 9.4.0
-- The CXX compiler identification is GNU 9.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found PythonInterp: /home/bob/anaconda3/envs/vid2e/bin/python (found version "3.9.12")
-- Found PythonLibs: /home/bob/anaconda3/envs/vid2e/lib/libpython3.9.so
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Found pybind11: /home/bob/anaconda3/envs/vid2e/include (found version "2.6.1" )
CMake Error at CMakeLists.txt:5 (find_package):
By not providing "FindOpenCV.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "OpenCV", but
CMake did not find one.

  Could not find a package configuration file provided by "OpenCV" with any
  of the following names:

    OpenCVConfig.cmake
    opencv-config.cmake

  Add the installation prefix of "OpenCV" to CMAKE_PREFIX_PATH or set
  "OpenCV_DIR" to a directory containing one of the above files.  If "OpenCV"
  provides a separate development package or SDK, be sure it has been
  installed.


-- Configuring incomplete, errors occurred!
See also "/tmp/pip-req-build-jg63bbet/build/temp.linux-x86_64-3.9/CMakeFiles/CMakeOutput.log".
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-req-build-jg63bbet/setup.py", line 53, in <module>
    setup(
  File "/home/bob/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/__init__.py", line 87, in setup
    return distutils.core.setup(**attrs)
  File "/home/bob/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 148, in setup
    return run_commands(dist)
  File "/home/bob/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
    dist.run_commands()
  File "/home/bob/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
    self.run_command(cmd)
  File "/home/bob/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/dist.py", line 1214, in run_command
    super().run_command(command)
  File "/home/bob/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
    cmd_obj.run()
  File "/home/bob/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/command/install.py", line 68, in run
    return orig.install.run(self)
  File "/home/bob/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/command/install.py", line 664, in run
    self.run_command('build')
  File "/home/bob/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/home/bob/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/dist.py", line 1214, in run_command
    super().run_command(command)
  File "/home/bob/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
    cmd_obj.run()
  File "/home/bob/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/home/bob/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/home/bob/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/dist.py", line 1214, in run_command
    super().run_command(command)
  File "/home/bob/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
    cmd_obj.run()
  File "/tmp/pip-req-build-jg63bbet/setup.py", line 31, in run
    self.build_extension(ext)
  File "/tmp/pip-req-build-jg63bbet/setup.py", line 49, in build_extension
    subprocess.check_call(['cmake', ext.sourcedir] + cmake_args, cwd=self.build_temp, env=env)
  File "/home/bob/anaconda3/envs/vid2e/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '/tmp/pip-req-build-jg63bbet', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/tmp/pip-req-build-jg63bbet/build/lib.linux-x86_64-3.9', '-DPYTHON_EXECUTABLE=/home/bob/anaconda3/envs/vid2e/bin/python', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1.
----------------------------------------

ERROR: Command errored out with exit status 1: /home/bob/anaconda3/envs/vid2e/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-jg63bbet/setup.py'"'"'; file='"'"'/tmp/pip-req-build-jg63bbet/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-ua79m_eq/install-record.txt --single-version-externally-mana

About installing ESIM failed

Hi! Thanks for your excellent works!
But it raises ERROR while using pip install .

The ERROR goes like:

    ERROR: Command errored out with exit status 1:
 command: /hdd8T_3/chensq/anaconda3/envs/vid2e/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-wrmsfhqy/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-wrmsfhqy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-qwp87v2d/install-record.txt --single-version-externally-managed --compile --install-headers /hdd8T_3/chensq/anaconda3/envs/vid2e/include/python3.7m/esim-py
     cwd: /tmp/pip-req-build-wrmsfhqy/
Complete output (113 lines):
running install
running build
running build_ext
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found PythonInterp: /hdd8T_3/chensq/anaconda3/envs/vid2e/bin/python (found version "3.7.9")
-- Found PythonLibs: /hdd8T_3/chensq/anaconda3/envs/vid2e/lib/libpython3.7m.so
-- Performing Test HAS_CPP14_FLAG
-- Performing Test HAS_CPP14_FLAG - Success
-- Found OpenCV: /hdd8T_3/chensq/anaconda3/envs/vid2e (found version "3.4.2")
CMake Warning at /usr/share/cmake-3.5/Modules/FindBoost.cmake:725 (message):
  Imported targets not available for Boost version 106900
Call Stack (most recent call first):
  /usr/share/cmake-3.5/Modules/FindBoost.cmake:763 (_Boost_COMPONENT_DEPENDENCIES)
  /usr/share/cmake-3.5/Modules/FindBoost.cmake:1332 (_Boost_MISSING_DEPENDENCIES)
  CMakeLists.txt:7 (find_package)
CMake Warning at /usr/share/cmake-3.5/Modules/FindBoost.cmake:725 (message):
  Imported targets not available for Boost version 106900
Call Stack (most recent call first):
  /usr/share/cmake-3.5/Modules/FindBoost.cmake:763 (_Boost_COMPONENT_DEPENDENCIES)
  /usr/share/cmake-3.5/Modules/FindBoost.cmake:1332 (_Boost_MISSING_DEPENDENCIES)
  CMakeLists.txt:7 (find_package)


-- Boost version: 1.69.0
-- Found the following Boost libraries:
--   system
--   filesystem
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- LTO enabled
-- Configuring done
CMake Error at /hdd8T_3/chensq/anaconda3/envs/vid2e/share/cmake/pybind11/pybind11Tools.cmake:131 (add_library):
  Target "esim_py" links to target "Boost::filesystem" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?
Call Stack (most recent call first):
  CMakeLists.txt:16 (pybind11_add_module)


CMake Error at /hdd8T_3/chensq/anaconda3/envs/vid2e/share/cmake/pybind11/pybind11Tools.cmake:131 (add_library):
  Target "esim_py" links to target "Boost::system" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?
Call Stack (most recent call first):
  CMakeLists.txt:16 (pybind11_add_module)


CMake Error at /hdd8T_3/chensq/anaconda3/envs/vid2e/share/cmake/pybind11/pybind11Tools.cmake:131 (add_library):
  Target "esim_py" links to target "Boost::filesystem" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?
Call Stack (most recent call first):
  CMakeLists.txt:16 (pybind11_add_module)


CMake Error at /hdd8T_3/chensq/anaconda3/envs/vid2e/share/cmake/pybind11/pybind11Tools.cmake:131 (add_library):
  Target "esim_py" links to target "Boost::system" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?
Call Stack (most recent call first):
  CMakeLists.txt:16 (pybind11_add_module)


-- Generating done
-- Build files have been written to: /tmp/pip-req-build-wrmsfhqy/build/temp.linux-x86_64-3.7
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-req-build-wrmsfhqy/setup.py", line 62, in <module>
    zip_safe=False,
  File "/hdd8T_3/chensq/anaconda3/envs/vid2e/lib/python3.7/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/hdd8T_3/chensq/anaconda3/envs/vid2e/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/hdd8T_3/chensq/anaconda3/envs/vid2e/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/hdd8T_3/chensq/anaconda3/envs/vid2e/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/hdd8T_3/chensq/anaconda3/envs/vid2e/lib/python3.7/site-packages/setuptools/command/install.py", line 61, in run
    return orig.install.run(self)
  File "/hdd8T_3/chensq/anaconda3/envs/vid2e/lib/python3.7/distutils/command/install.py", line 545, in run
    self.run_command('build')
  File "/hdd8T_3/chensq/anaconda3/envs/vid2e/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/hdd8T_3/chensq/anaconda3/envs/vid2e/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/hdd8T_3/chensq/anaconda3/envs/vid2e/lib/python3.7/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/hdd8T_3/chensq/anaconda3/envs/vid2e/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/hdd8T_3/chensq/anaconda3/envs/vid2e/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/tmp/pip-req-build-wrmsfhqy/setup.py", line 31, in run
    self.build_extension(ext)
  File "/tmp/pip-req-build-wrmsfhqy/setup.py", line 49, in build_extension
    subprocess.check_call(['cmake', ext.sourcedir] + cmake_args, cwd=self.build_temp, env=env)
  File "/hdd8T_3/chensq/anaconda3/envs/vid2e/lib/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '/tmp/pip-req-build-wrmsfhqy', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/tmp/pip-req-build-wrmsfhqy/build/lib.linux-x86_64-3.7', '-DPYTHON_EXECUTABLE=/hdd8T_3/chensq/anaconda3/envs/vid2e/bin/python', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1.
----------------------------------------
ERROR: Command errored out with exit status 1: /hdd8T_3/chensq/anaconda3/envs/vid2e/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-wrmsfhqy/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-wrmsfhqy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-qwp87v2d/install-record.txt --single-version-externally-managed --compile --install-headers /hdd8T_3/chensq/anaconda3/envs/vid2e/include/python3.7m/esim-py Check the logs for full command output.

I also tried another command: python setup.py install. And the ERROR seems to be the same:

CMake Error at /hdd8T_3/chensq/anaconda3/envs/vid2e/share/cmake/pybind11/pybind11Tools.cmake:131 (add_library): Target "esim_py" links to target "Boost::filesystem" but the target was not found.  Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing?
Call Stack (most recent call first): CMakeLists.txt:16 (pybind11_add_module)

subprocess.CalledProcessError: Command '['cmake', '/hdd8T_3/chensq/rpg_vid2e/esim_py', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/hdd8T_3/chensq/rpg_vid2e/esim_py/build/lib.linux-x86_64-3.7', '-DPYTHON_EXECUTABLE=/hdd8T_3/chensq/anaconda3/envs/vid2e/bin/python', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1.

I've checked the output of conda list | grep pybind11, and the file /hdd8T_3/chensq/anaconda/envs/vid2e/share/cmake/pybind11/pybind11Config.cmake exists. The output of conda list | grep pybind11 is shown below:
```
pybind11                  2.5.0            py37h99015e2_1    conda-forge
```
Maybe is my fault path in CMakeLists.txt? But I don't know how to modify to my own path.
I am looking forward to your reply!

Building python bindings for ESIM

Hi!

I am having an error by trying to build the python bindings for ESIM.
When I run pip install . in the esim_py folder using the (vid2e) environment, I got the following error:

ERROR: Command errored out with exit status 1:
   command: /home/juan/Software/Anacoda2/envs/vid2e/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-az36ldis/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-az36ldis/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-xflbexrp
       cwd: /tmp/pip-req-build-az36ldis/
  Complete output (155 lines):
  running bdist_wheel
  running build
  running build_ext
  -- The C compiler identification is GNU 5.5.0
  -- The CXX compiler identification is GNU 5.5.0
  -- Check for working C compiler: /usr/bin/cc
  -- Check for working C compiler: /usr/bin/cc -- works
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- Check for working CXX compiler: /usr/bin/c++
  -- Check for working CXX compiler: /usr/bin/c++ -- works
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  -- Found PythonInterp: /home/juan/Software/Anacoda2/envs/vid2e/bin/python (found version "3.7.7")
  -- Found PythonLibs: /home/juan/Software/Anacoda2/envs/vid2e/lib/libpython3.7m.so
  -- Performing Test HAS_CPP14_FLAG
  -- Performing Test HAS_CPP14_FLAG - Success
  -- Found OpenCV: /opt/ros/kinetic (found version "3.3.1")
  -- Found Boost 1.70.0 at /home/juan/Software/Anacoda2/envs/vid2e/lib/cmake/Boost-1.70.0
  --   Requested configuration: QUIET REQUIRED COMPONENTS system;filesystem
  -- Found boost_headers 1.70.0 at /home/juan/Software/Anacoda2/envs/vid2e/lib/cmake/boost_headers-1.70.0
  -- Found boost_system 1.70.0 at /home/juan/Software/Anacoda2/envs/vid2e/lib/cmake/boost_system-1.70.0
  --   libboost_system.a
  -- Adding boost_system dependencies: headers
  -- Found boost_filesystem 1.70.0 at /home/juan/Software/Anacoda2/envs/vid2e/lib/cmake/boost_filesystem-1.70.0
  --   libboost_filesystem.a
  -- Adding boost_filesystem dependencies: headers
  -- Found Boost: /home/juan/Software/Anacoda2/envs/vid2e/lib/cmake/Boost-1.70.0/BoostConfig.cmake (found version "1.70.0") found components:  system filesystem
  -- Performing Test HAS_FLTO
  -- Performing Test HAS_FLTO - Success
  -- LTO enabled
  -- Configuring done
  -- Generating done
  -- Build files have been written to: /tmp/pip-req-build-az36ldis/build/temp.linux-x86_64-3.7
  Scanning dependencies of target libesim
  [ 25%] Building CXX object CMakeFiles/libesim.dir/src/esim.cpp.o
  In file included from /tmp/pip-req-build-az36ldis/src/esim.cpp:1:0:
  /tmp/pip-req-build-az36ldis/include/esim.h: In member function ‘void EventSimulator::read_directory_from_path(const string&, std::vector<std::__cxx11::basic_string<char> >&)’:
  /tmp/pip-req-build-az36ldis/include/esim.h:87:12: error: ‘path_leaf_string’ does not name a type
         auto path_leaf_string = [](const boost::filesystem::directory_entry& entry) -> std::string {return entry.path().string();};
              ^
  In file included from /tmp/pip-req-build-az36ldis/src/esim.cpp:1:0:
  /tmp/pip-req-build-az36ldis/include/esim.h:89:57: error: ‘path_leaf_string’ was not declared in this scope
         std::transform(start, end, std::back_inserter(v), path_leaf_string);
                                                           ^
  /tmp/pip-req-build-az36ldis/src/esim.cpp: In member function ‘Eigen::MatrixXd EventSimulator::generateFromVideo(std::__cxx11::string, std::__cxx11::string)’:
  /tmp/pip-req-build-az36ldis/src/esim.cpp:25:55: error: no matching function for call to ‘std::basic_ifstream<char>::basic_ifstream(std::__cxx11::string&)’
       std::ifstream timestamps_file(timestamps_file_path);
                                                         ^
  In file included from /home/juan/Software/Anacoda2/envs/vid2e/include/boost/filesystem/fstream.hpp:23:0,
                   from /home/juan/Software/Anacoda2/envs/vid2e/include/boost/filesystem/string_file.hpp:14,
                   from /home/juan/Software/Anacoda2/envs/vid2e/include/boost/filesystem.hpp:19,
                   from /tmp/pip-req-build-az36ldis/include/esim.h:5,
                   from /tmp/pip-req-build-az36ldis/src/esim.cpp:1:
  /usr/include/c++/5/fstream:495:7: note: candidate: std::basic_ifstream<_CharT, _Traits>::basic_ifstream(const char*, std::ios_base::openmode) [with _CharT = char; _Traits = std::char_traits<char>; std::ios_base::openmode = std::_Ios_Openmode]
         basic_ifstream(const char* __s, ios_base::openmode __mode = ios_base::in)
         ^
  /usr/include/c++/5/fstream:495:7: note:   no known conversion for argument 1 from ‘std::__cxx11::string {aka std::__cxx11::basic_string<char>}’ to ‘const char*’
  /usr/include/c++/5/fstream:481:7: note: candidate: std::basic_ifstream<_CharT, _Traits>::basic_ifstream() [with _CharT = char; _Traits = std::char_traits<char>]
         basic_ifstream() : __istream_type(), _M_filebuf()
         ^
  /usr/include/c++/5/fstream:481:7: note:   candidate expects 0 arguments, 1 provided
  /usr/include/c++/5/fstream:455:11: note: candidate: std::basic_ifstream<char>::basic_ifstream(const std::basic_ifstream<char>&)
       class basic_ifstream : public basic_istream<_CharT, _Traits>
             ^
  /usr/include/c++/5/fstream:455:11: note:   no known conversion for argument 1 from ‘std::__cxx11::string {aka std::__cxx11::basic_string<char>}’ to ‘const std::basic_ifstream<char>&’
  /tmp/pip-req-build-az36ldis/src/esim.cpp:53:16: error: ‘stod’ is not a member of ‘std’
           time = std::stod(time_str);
                  ^
  /tmp/pip-req-build-az36ldis/src/esim.cpp: In member function ‘Eigen::MatrixXd EventSimulator::generateFromStampedImageSequence(std::vector<std::__cxx11::basic_string<char> >, std::vector<double, std::allocator<double> >)’:
  /tmp/pip-req-build-az36ldis/src/esim.cpp:68:107: error: ‘to_string’ is not a member of ‘std’
           throw std::runtime_error("Number of image paths and number of timestamps should be equal. Got " + std::to_string(image_paths.size()) + " and " + std::to_string(timestamps.size()));
                                                                                                             ^
  /tmp/pip-req-build-az36ldis/src/esim.cpp:68:154: error: ‘to_string’ is not a member of ‘std’
           throw std::runtime_error("Number of image paths and number of timestamps should be equal. Got " + std::to_string(image_paths.size()) + " and " + std::to_string(timestamps.size()));
                                                                                                                                                            ^
  /tmp/pip-req-build-az36ldis/src/esim.cpp: In member function ‘Eigen::MatrixXd EventSimulator::generateFromFolder(std::__cxx11::string, std::__cxx11::string)’:
  /tmp/pip-req-build-az36ldis/src/esim.cpp:108:55: error: no matching function for call to ‘std::basic_ifstream<char>::basic_ifstream(std::__cxx11::string&)’
       std::ifstream timestamps_file(timestamps_file_path);
                                                         ^
  In file included from /home/juan/Software/Anacoda2/envs/vid2e/include/boost/filesystem/fstream.hpp:23:0,
                   from /home/juan/Software/Anacoda2/envs/vid2e/include/boost/filesystem/string_file.hpp:14,
                   from /home/juan/Software/Anacoda2/envs/vid2e/include/boost/filesystem.hpp:19,
                   from /tmp/pip-req-build-az36ldis/include/esim.h:5,
                   from /tmp/pip-req-build-az36ldis/src/esim.cpp:1:
  /usr/include/c++/5/fstream:495:7: note: candidate: std::basic_ifstream<_CharT, _Traits>::basic_ifstream(const char*, std::ios_base::openmode) [with _CharT = char; _Traits = std::char_traits<char>; std::ios_base::openmode = std::_Ios_Openmode]
         basic_ifstream(const char* __s, ios_base::openmode __mode = ios_base::in)
         ^
  /usr/include/c++/5/fstream:495:7: note:   no known conversion for argument 1 from ‘std::__cxx11::string {aka std::__cxx11::basic_string<char>}’ to ‘const char*’
  /usr/include/c++/5/fstream:481:7: note: candidate: std::basic_ifstream<_CharT, _Traits>::basic_ifstream() [with _CharT = char; _Traits = std::char_traits<char>]
         basic_ifstream() : __istream_type(), _M_filebuf()
         ^
  /usr/include/c++/5/fstream:481:7: note:   candidate expects 0 arguments, 1 provided
  /usr/include/c++/5/fstream:455:11: note: candidate: std::basic_ifstream<char>::basic_ifstream(const std::basic_ifstream<char>&)
       class basic_ifstream : public basic_istream<_CharT, _Traits>
             ^
  /usr/include/c++/5/fstream:455:11: note:   no known conversion for argument 1 from ‘std::__cxx11::string {aka std::__cxx11::basic_string<char>}’ to ‘const std::basic_ifstream<char>&’
  /tmp/pip-req-build-az36ldis/src/esim.cpp:120:36: warning: range-based ‘for’ loops only available with -std=c++11 or -std=gnu++11
       for (const std::string& file : image_files)
                                      ^
  /tmp/pip-req-build-az36ldis/src/esim.cpp:135:16: error: ‘stod’ is not a member of ‘std’
           time = std::stod(time_str);
                  ^
  /tmp/pip-req-build-az36ldis/src/esim.cpp: In member function ‘void EventSimulator::imageCallback(const cv::Mat&, double, std::vector<Event>&)’:
  /tmp/pip-req-build-az36ldis/src/esim.cpp:172:12: error: ‘constexpr’ does not name a type
       static constexpr double kTolerance = 1e-6;
              ^
  /tmp/pip-req-build-az36ldis/src/esim.cpp:172:12: note: C++11 ‘constexpr’ only available with -std=c++11 or -std=gnu++11
  /tmp/pip-req-build-az36ldis/src/esim.cpp:183:41: error: ‘kTolerance’ was not declared in this scope
               if (std::fabs (it - itdt) > kTolerance)
                                           ^
  /tmp/pip-req-build-az36ldis/src/esim.cpp:207:40: error: ‘class std::vector<Event>’ has no member named ‘emplace_back’
                               new_events.emplace_back(x, y,t,pol);
                                          ^
  CMakeFiles/libesim.dir/build.make:62: recipe for target 'CMakeFiles/libesim.dir/src/esim.cpp.o' failed
  make[2]: *** [CMakeFiles/libesim.dir/src/esim.cpp.o] Error 1
  CMakeFiles/Makefile2:77: recipe for target 'CMakeFiles/libesim.dir/all' failed
  make[1]: *** [CMakeFiles/libesim.dir/all] Error 2
  Makefile:83: recipe for target 'all' failed
  make: *** [all] Error 2
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-req-build-az36ldis/setup.py", line 62, in <module>
      zip_safe=False,
    File "/home/juan/Software/Anacoda2/envs/vid2e/lib/python3.7/site-packages/setuptools/__init__.py", line 144, in setup
      return distutils.core.setup(**attrs)
    File "/home/juan/Software/Anacoda2/envs/vid2e/lib/python3.7/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/home/juan/Software/Anacoda2/envs/vid2e/lib/python3.7/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/home/juan/Software/Anacoda2/envs/vid2e/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/home/juan/Software/Anacoda2/envs/vid2e/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 223, in run
      self.run_command('build')
    File "/home/juan/Software/Anacoda2/envs/vid2e/lib/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/home/juan/Software/Anacoda2/envs/vid2e/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/home/juan/Software/Anacoda2/envs/vid2e/lib/python3.7/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/home/juan/Software/Anacoda2/envs/vid2e/lib/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/home/juan/Software/Anacoda2/envs/vid2e/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/tmp/pip-req-build-az36ldis/setup.py", line 31, in run
      self.build_extension(ext)
    File "/tmp/pip-req-build-az36ldis/setup.py", line 50, in build_extension
      subprocess.check_call(['cmake', '--build', '.'] + build_args, cwd=self.build_temp)
    File "/home/juan/Software/Anacoda2/envs/vid2e/lib/python3.7/subprocess.py", line 363, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--config', 'Release', '--', '-j']' returned non-zero exit status 2.
  ----------------------------------------
  ERROR: Failed building wheel for esim-py

I followed the installation guide. Perhaps there is a missing dependency for esim_py. Could you please help to solve this issue?

Best,

Upsampling video leads to flashing

Hello,

When I run the code on the video provided there's a flashing effect, is this expected?

output.mp4

"esim_cuda" not found

Even after using the binary file of esim, I'm getting this error even though "esim_cuda" is present in the directory

Traceback (most recent call last):
  File "esim_torch/generate_events.py", line 4, in <module>
    import esim_torch
  File "C:\Users\varoo\EventSeq\rpg_vid2e\esim_torch\esim_torch.py", line 2, in <module>
    import esim_cuda
ModuleNotFoundError: No module named 'esim_cuda'

Can new interpolation model help to get better events?

Is it possible to include upsampler into the install and provide an upsampler that takes images one by one for a for-loop

Hello devs,

Using your esim a lot in my project.
And though if it is possible to include upsampler into the install and provide an upsampler that takes images one by one for a for-loop.

I am developing a esim plugin for MuJoCo and that would help :D

Recursion in the upsampler does not always terminate

Hello,

we are processing the validation set of the REDS 120fps dataset (https://seungjunnah.github.io/Datasets/reds.html). For about half of the sequences there, we encounter a case in which the recursion in upsampling.utils.upsampler.Upsampler._upsample_adaptive does not terminate:

 def _upsample_adaptive(self, I0, I1, t0, t1, num_bisections=-1):
        if num_bisections == 0:
            return [], []

        dt = self.batch_dt = np.full(shape=(1,), fill_value=0.5, dtype=np.float32)
        image, F_0_1, F_1_0 = self.interpolator.interpolate(I0, I1, dt)
        raise Exception()

        if num_bisections < 0:
            flow_mag_0_1_max = ((F_0_1 ** 2).sum(-1) ** .5).max()
            flow_mag_1_0_max = ((F_1_0 ** 2).sum(-1) ** .5).max()
            num_bisections = int(np.ceil(np.log(max([flow_mag_0_1_max, flow_mag_1_0_max]))/np.log(2)))

        left_images, left_timestamps = self._upsample_adaptive(I0, image, t0, (t0+t1)/2, num_bisections=num_bisections-1)
        right_images, right_timestamps = self._upsample_adaptive(image, I1, (t0+t1)/2, t1, num_bisections=num_bisections-1)
        timestamps = left_timestamps + [(t0+t1)/2] + right_timestamps
        images = left_images + [image[0]] + right_images

        return images, timestamps

In the branch if num_bisections < 0: the computation of num_bisections might yield 0, such that recursion becomes infinite (because the recursive calls will hit exactly the same case again) and we either run into Python's recursion limit or run out of memory.

I have a very superficial understanding of this code, but my guess is that adding the following lines as the last instructions inside that if branch fixes the issue:

            if num_bisections == 0:
                return [image[0]], [(t0+t1)/2]

What do you think?

error buiding wheel for esim_py

I got a confusing error when buiding wheel for esim_py. Could you please offer some help? Thanks for your time.
And I got the following error message:

Processing ./rpg_vid2e/esim_py
Preparing metadata (setup.py) ... done
Building wheels for collected packages: esim-py
Building wheel for esim-py (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [59 lines of output]
running bdist_wheel
running build
running build_ext
-- Found pybind11: /mnt/cephfs/home/alvin/anaconda3/envs/vid2e/include (found version "2.6.1" )
CMake Error at CMakeLists.txt:6 (find_package):
Could not find a package configuration file provided by "Eigen3" with any
of the following names:

      Eigen3Config.cmake
      eigen3-config.cmake
  
    Add the installation prefix of "Eigen3" to CMAKE_PREFIX_PATH or set
    "Eigen3_DIR" to a directory containing one of the above files.  If "Eigen3"
    provides a separate development package or SDK, be sure it has been
    installed.
  
  
  -- Configuring incomplete, errors occurred!
  See also "/mnt/cephfs/home/alvin/jialiang/vid2e/rpg_vid2e/esim_py/build/temp.linux-x86_64-cpython-39/CMakeFiles/CMakeOutput.log".
  Traceback (most recent call last):
    File "<string>", line 2, in <module>
    File "<pip-setuptools-caller>", line 34, in <module>
    File "/mnt/cephfs/home/alvin/jialiang/vid2e/rpg_vid2e/esim_py/setup.py", line 53, in <module>
      setup(
    File "/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/__init__.py", line 107, in setup
      return distutils.core.setup(**attrs)
    File "/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
      return run_commands(dist)
    File "/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
      dist.run_commands()
    File "/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
      self.run_command(cmd)
    File "/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/dist.py", line 1244, in run_command
      super().run_command(command)
    File "/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 325, in run
      self.run_command("build")
    File "/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/dist.py", line 1244, in run_command
      super().run_command(command)
    File "/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/command/build.py", line 131, in run
      self.run_command(cmd_name)
    File "/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/dist.py", line 1244, in run_command
      super().run_command(command)
    File "/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/mnt/cephfs/home/alvin/jialiang/vid2e/rpg_vid2e/esim_py/setup.py", line 31, in run
      self.build_extension(ext)
    File "/mnt/cephfs/home/alvin/jialiang/vid2e/rpg_vid2e/esim_py/setup.py", line 49, in build_extension
      subprocess.check_call(['cmake', ext.sourcedir] + cmake_args, cwd=self.build_temp, env=env)
    File "/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/lib/python3.9/subprocess.py", line 373, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['cmake', '/mnt/cephfs/home/alvin/jialiang/vid2e/rpg_vid2e/esim_py', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/mnt/cephfs/home/alvin/jialiang/vid2e/rpg_vid2e/esim_py/build/lib.linux-x86_64-cpython-39', '-DPYTHON_EXECUTABLE=/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/bin/python', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1.
  [end of output]

× Running setup.py install for esim-py did not run successfully.
│ exit code: 1
╰─> [93 lines of output]
running install
/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

          ********************************************************************************
          Please avoid running ``setup.py`` directly.
          Instead, use pypa/build, pypa/installer, pypa/build or
          other standards-based tools.
  
          See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
          ********************************************************************************
  
  !!
    self.initialize_options()
  running build
  running build_ext
  -- The C compiler identification is GNU 9.4.0
  -- The CXX compiler identification is GNU 9.4.0
  -- Check for working C compiler: /usr/bin/cc
  -- Check for working C compiler: /usr/bin/cc -- works
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- Check for working CXX compiler: /usr/bin/c++
  -- Check for working CXX compiler: /usr/bin/c++ -- works
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  -- Found PythonInterp: /mnt/cephfs/home/alvin/anaconda3/envs/vid2e/bin/python (found version "3.9.16")
  -- Found PythonLibs: /mnt/cephfs/home/alvin/anaconda3/envs/vid2e/lib/libpython3.9.so
  -- Performing Test HAS_FLTO
  -- Performing Test HAS_FLTO - Success
  -- Found pybind11: /mnt/cephfs/home/alvin/anaconda3/envs/vid2e/include (found version "2.6.1" )
  -- Found OpenCV: /usr (found version "4.2.0")
  CMake Error at CMakeLists.txt:6 (find_package):
    Could not find a package configuration file provided by "Eigen3" with any
    of the following names:
  
      Eigen3Config.cmake
      eigen3-config.cmake
  
    Add the installation prefix of "Eigen3" to CMAKE_PREFIX_PATH or set
    "Eigen3_DIR" to a directory containing one of the above files.  If "Eigen3"
    provides a separate development package or SDK, be sure it has been
    installed.
  
  
  -- Configuring incomplete, errors occurred!
  See also "/mnt/cephfs/home/alvin/jialiang/vid2e/rpg_vid2e/esim_py/build/temp.linux-x86_64-cpython-39/CMakeFiles/CMakeOutput.log".
  Traceback (most recent call last):
    File "<string>", line 2, in <module>
    File "<pip-setuptools-caller>", line 34, in <module>
    File "/mnt/cephfs/home/alvin/jialiang/vid2e/rpg_vid2e/esim_py/setup.py", line 53, in <module>
      setup(
    File "/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/__init__.py", line 107, in setup
      return distutils.core.setup(**attrs)
    File "/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
      return run_commands(dist)
    File "/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
      dist.run_commands()
    File "/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
      self.run_command(cmd)
    File "/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/dist.py", line 1244, in run_command
      super().run_command(command)
    File "/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/command/install.py", line 74, in run
      return orig.install.run(self)
    File "/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/command/install.py", line 697, in run
      self.run_command('build')
    File "/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/dist.py", line 1244, in run_command
      super().run_command(command)
    File "/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/command/build.py", line 131, in run
      self.run_command(cmd_name)
    File "/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/dist.py", line 1244, in run_command
      super().run_command(command)
    File "/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/mnt/cephfs/home/alvin/jialiang/vid2e/rpg_vid2e/esim_py/setup.py", line 31, in run
      self.build_extension(ext)
    File "/mnt/cephfs/home/alvin/jialiang/vid2e/rpg_vid2e/esim_py/setup.py", line 49, in build_extension
      subprocess.check_call(['cmake', ext.sourcedir] + cmake_args, cwd=self.build_temp, env=env)
    File "/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/lib/python3.9/subprocess.py", line 373, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['cmake', '/mnt/cephfs/home/alvin/jialiang/vid2e/rpg_vid2e/esim_py', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/mnt/cephfs/home/alvin/jialiang/vid2e/rpg_vid2e/esim_py/build/lib.linux-x86_64-cpython-39', '-DPYTHON_EXECUTABLE=/mnt/cephfs/home/alvin/anaconda3/envs/vid2e/bin/python', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1.
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> esim-py

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

Why I always meet such error? I have installed esim_torch.

root@3a4e6b6b83b4:~/rpg_vid2e# pip install esim_torch/
Looking in indexes: https://mirrors.ustc.edu.cn/pypi/web/simple
Processing ./esim_torch
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: esim-cuda
  Building wheel for esim-cuda (setup.py) ... done
  Created wheel for esim-cuda: filename=esim_cuda-0.0.0-cp39-cp39-linux_x86_64.whl size=237193 sha256=e6c205811b6f6c837460eb7ad07cd488cde0cbd33de6cffed86d3e2619215131
  Stored in directory: /tmp/pip-ephem-wheel-cache-l8knfmev/wheels/1a/8b/ca/09c363172d9de1896c08f1bf6f25085684a7962dc84bd09128
Successfully built esim-cuda
Installing collected packages: esim-cuda
Successfully installed esim-cuda-0.0.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
root@3a4e6b6b83b4:~/rpg_vid2e# python
Python 3.9.10 | packaged by conda-forge | (main, Feb  1 2022, 21:24:11) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import esim_cuda
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: libc10.so: cannot open shared object file: No such file or directory

MemoryError: std::bad_alloc

when I run python tests/plot_virtual_events.py an error raise:
Traceback (most recent call last):
File "tests/plot_virtual_events.py", line 56, in
events = esim.generateFromFolder(image_folder, timestamps_file)
MemoryError: std::bad_alloc

Installing esim_py failed

Thanks for your pathbreaking work!
While trying to install esim_py on google colaboratory, they are giving the following output. As suggested, I've run the following lines of code to adapt the CUDA toolkit version:

cuda_version=10.1
!conda create -y -n vid2e python=3.7
!conda activate vid2e
!conda install -y pytorch torchvision cudatoolkit=$cuda_version -c pytorch
!conda install -y -c conda-forge opencv tqdm scikit-video eigen boost boost-cpp pybind11

Can you suggest a possible solution to solve this issue? Thanks in advance.

error building for esim_torch

I was able to install esim successfully, but I am currently experiencing issues (source code?) with installing esim_torch.

Processing ./esim_torch
Preparing metadata (setup.py) ... done
Building wheels for collected packages: esim-torch
Building wheel for esim-torch (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [91 lines of output]
running bdist_wheel
running build
running build_py
running build_ext
/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/torch/utils/cpp_extension.py:371: UserWarning: The detected CUDA version (11.4) has a minor version mismatch with the version that was used to compile PyTorch (11.2). Most likely this shouldn't be a problem.
warnings.warn(CUDA_MISMATCH_WARN.format(cuda_str_version, torch.version.cuda))
/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/torch/utils/cpp_extension.py:381: UserWarning: There are no /home/watk681/anaconda3/envs/vid2e/bin/x86_64-conda-linux-gnu-c++ version bounds defined for CUDA version 11.4
warnings.warn(f'There are no {compiler_name} version bounds defined for CUDA version {cuda_str_version}')
building 'esim_cuda' extension
Emitting ninja build file /qfs/projects/mage/watk681/rpg_vid2e/esim_torch/build/temp.linux-x86_64-cpython-39/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/1] /usr/local/cuda/bin/nvcc -I/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/torch/include -I/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/torch/include/TH -I/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/torch/include/THC -I/usr/local/cuda/include -I/home/watk681/anaconda3/envs/vid2e/include/python3.9 -c -c /qfs/projects/mage/watk681/rpg_vid2e/esim_torch/src/esim_torch/esim_cuda_kernel.cu -o /qfs/projects/mage/watk681/rpg_vid2e/esim_torch/build/temp.linux-x86_64-cpython-39/src/esim_torch/esim_cuda_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1014"' -DTORCH_EXTENSION_NAME=esim_cuda -D_GLIBCXX_USE_CXX11_ABI=1 -gencode=arch=compute_60,code=compute_60 -gencode=arch=compute_60,code=sm_60 -ccbin /home/watk681/anaconda3/envs/vid2e/bin/x86_64-conda-linux-gnu-cc -std=c++14
FAILED: /qfs/projects/mage/watk681/rpg_vid2e/esim_torch/build/temp.linux-x86_64-cpython-39/src/esim_torch/esim_cuda_kernel.o
/usr/local/cuda/bin/nvcc -I/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/torch/include -I/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/torch/include/TH -I/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/torch/include/THC -I/usr/local/cuda/include -I/home/watk681/anaconda3/envs/vid2e/include/python3.9 -c -c /qfs/projects/mage/watk681/rpg_vid2e/esim_torch/src/esim_torch/esim_cuda_kernel.cu -o /qfs/projects/mage/watk681/rpg_vid2e/esim_torch/build/temp.linux-x86_64-cpython-39/src/esim_torch/esim_cuda_kernel.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1014"' -DTORCH_EXTENSION_NAME=esim_cuda -D_GLIBCXX_USE_CXX11_ABI=1 -gencode=arch=compute_60,code=compute_60 -gencode=arch=compute_60,code=sm_60 -ccbin /home/watk681/anaconda3/envs/vid2e/bin/x86_64-conda-linux-gnu-cc -std=c++14
/qfs/projects/mage/watk681/rpg_vid2e/esim_torch/src/esim_torch/esim_cuda_kernel.cu(39): warning: variable "i0" was declared but never referenced
detected during instantiation of "void count_events_cuda_forward_kernel(const scalar_t *, const scalar_t *, scalar_t *, int64_t *, int, int, int, float, float) [with scalar_t=float]"
(152): here

  /home/watk681/anaconda3/envs/vid2e/x86_64-conda-linux-gnu/include/c++/11.3.0/bits/std_function.h:435:145: error: parameter packs not expanded with '...':
    435 |         function(_Functor&& __f)
        |                                                                                                                                                 ^
  /home/watk681/anaconda3/envs/vid2e/x86_64-conda-linux-gnu/include/c++/11.3.0/bits/std_function.h:435:145: note:         '_ArgTypes'
  /home/watk681/anaconda3/envs/vid2e/x86_64-conda-linux-gnu/include/c++/11.3.0/bits/std_function.h:530:146: error: parameter packs not expanded with '...':
    530 |         operator=(_Functor&& __f)
        |                                                                                                                                                  ^
  /home/watk681/anaconda3/envs/vid2e/x86_64-conda-linux-gnu/include/c++/11.3.0/bits/std_function.h:530:146: note:         '_ArgTypes'
  ninja: build stopped: subcommand failed.
  Traceback (most recent call last):
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1884, in _run_ninja_build
      subprocess.run(
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/subprocess.py", line 528, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
  
  The above exception was the direct cause of the following exception:
  
  Traceback (most recent call last):
    File "<string>", line 2, in <module>
    File "<pip-setuptools-caller>", line 34, in <module>
    File "/qfs/projects/mage/watk681/rpg_vid2e/esim_torch/setup.py", line 4, in <module>
      setup(
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/__init__.py", line 108, in setup
      return distutils.core.setup(**attrs)
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
      return run_commands(dist)
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
      dist.run_commands()
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
      self.run_command(cmd)
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/dist.py", line 1221, in run_command
      super().run_command(command)
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 325, in run
      self.run_command("build")
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/dist.py", line 1221, in run_command
      super().run_command(command)
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/command/build.py", line 131, in run
      self.run_command(cmd_name)
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/dist.py", line 1221, in run_command
      super().run_command(command)
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 84, in run
      _build_ext.run(self)
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
      self.build_extensions()
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 827, in build_extensions
      build_ext.build_extensions(self)
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
      self._build_extensions_serial()
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
      self.build_extension(ext)
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 246, in build_extension
      _build_ext.build_extension(self, ext)
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 548, in build_extension
      objects = self.compiler.compile(
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 642, in unix_wrap_ninja_compile
      _write_ninja_file_and_compile_objects(
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1557, in _write_ninja_file_and_compile_objects
      _run_ninja_build(
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1900, in _run_ninja_build
      raise RuntimeError(message) from e
  RuntimeError: Error compiling objects for extension
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for esim-torch
Running setup.py clean for esim-torch
Failed to build esim-torch
Installing collected packages: esim-torch
Running setup.py install for esim-torch ... error
error: subprocess-exited-with-error

× Running setup.py install for esim-torch did not run successfully.
│ exit code: 1
╰─> [103 lines of output]
running install
/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-39
creating build/lib.linux-x86_64-cpython-39/esim_torch
copying src/esim_torch/esim_torch.py -> build/lib.linux-x86_64-cpython-39/esim_torch
copying src/esim_torch/init.py -> build/lib.linux-x86_64-cpython-39/esim_torch
running build_ext
/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/torch/utils/cpp_extension.py:371: UserWarning: The detected CUDA version (11.4) has a minor version mismatch with the version that was used to compile PyTorch (11.2). Most likely this shouldn't be a problem.
warnings.warn(CUDA_MISMATCH_WARN.format(cuda_str_version, torch.version.cuda))
/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/torch/utils/cpp_extension.py:381: UserWarning: There are no /home/watk681/anaconda3/envs/vid2e/bin/x86_64-conda-linux-gnu-c++ version bounds defined for CUDA version 11.4
warnings.warn(f'There are no {compiler_name} version bounds defined for CUDA version {cuda_str_version}')
building 'esim_cuda' extension
creating /qfs/projects/mage/watk681/rpg_vid2e/esim_torch/build/temp.linux-x86_64-cpython-39
creating /qfs/projects/mage/watk681/rpg_vid2e/esim_torch/build/temp.linux-x86_64-cpython-39/src
creating /qfs/projects/mage/watk681/rpg_vid2e/esim_torch/build/temp.linux-x86_64-cpython-39/src/esim_torch
Emitting ninja build file /qfs/projects/mage/watk681/rpg_vid2e/esim_torch/build/temp.linux-x86_64-cpython-39/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/1] /usr/local/cuda/bin/nvcc -I/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/torch/include -I/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/torch/include/TH -I/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/torch/include/THC -I/usr/local/cuda/include -I/home/watk681/anaconda3/envs/vid2e/include/python3.9 -c -c /qfs/projects/mage/watk681/rpg_vid2e/esim_torch/src/esim_torch/esim_cuda_kernel.cu -o /qfs/projects/mage/watk681/rpg_vid2e/esim_torch/build/temp.linux-x86_64-cpython-39/src/esim_torch/esim_cuda_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1014"' -DTORCH_EXTENSION_NAME=esim_cuda -D_GLIBCXX_USE_CXX11_ABI=1 -gencode=arch=compute_60,code=compute_60 -gencode=arch=compute_60,code=sm_60 -ccbin /home/watk681/anaconda3/envs/vid2e/bin/x86_64-conda-linux-gnu-cc -std=c++14
FAILED: /qfs/projects/mage/watk681/rpg_vid2e/esim_torch/build/temp.linux-x86_64-cpython-39/src/esim_torch/esim_cuda_kernel.o
/usr/local/cuda/bin/nvcc -I/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/torch/include -I/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/torch/include/TH -I/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/torch/include/THC -I/usr/local/cuda/include -I/home/watk681/anaconda3/envs/vid2e/include/python3.9 -c -c /qfs/projects/mage/watk681/rpg_vid2e/esim_torch/src/esim_torch/esim_cuda_kernel.cu -o /qfs/projects/mage/watk681/rpg_vid2e/esim_torch/build/temp.linux-x86_64-cpython-39/src/esim_torch/esim_cuda_kernel.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1014"' -DTORCH_EXTENSION_NAME=esim_cuda -D_GLIBCXX_USE_CXX11_ABI=1 -gencode=arch=compute_60,code=compute_60 -gencode=arch=compute_60,code=sm_60 -ccbin /home/watk681/anaconda3/envs/vid2e/bin/x86_64-conda-linux-gnu-cc -std=c++14
/qfs/projects/mage/watk681/rpg_vid2e/esim_torch/src/esim_torch/esim_cuda_kernel.cu(39): warning: variable "i0" was declared but never referenced
detected during instantiation of "void count_events_cuda_forward_kernel(const scalar_t *, const scalar_t *, scalar_t *, int64_t *, int, int, int, float, float) [with scalar_t=float]"
(152): here

  /home/watk681/anaconda3/envs/vid2e/x86_64-conda-linux-gnu/include/c++/11.3.0/bits/std_function.h:435:145: error: parameter packs not expanded with '...':
    435 |         function(_Functor&& __f)
        |                                                                                                                                                 ^
  /home/watk681/anaconda3/envs/vid2e/x86_64-conda-linux-gnu/include/c++/11.3.0/bits/std_function.h:435:145: note:         '_ArgTypes'
  /home/watk681/anaconda3/envs/vid2e/x86_64-conda-linux-gnu/include/c++/11.3.0/bits/std_function.h:530:146: error: parameter packs not expanded with '...':
    530 |         operator=(_Functor&& __f)
        |                                                                                                                                                  ^
  /home/watk681/anaconda3/envs/vid2e/x86_64-conda-linux-gnu/include/c++/11.3.0/bits/std_function.h:530:146: note:         '_ArgTypes'
  ninja: build stopped: subcommand failed.
  Traceback (most recent call last):
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1884, in _run_ninja_build
      subprocess.run(
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/subprocess.py", line 528, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
  
  The above exception was the direct cause of the following exception:
  
  Traceback (most recent call last):
    File "<string>", line 2, in <module>
    File "<pip-setuptools-caller>", line 34, in <module>
    File "/qfs/projects/mage/watk681/rpg_vid2e/esim_torch/setup.py", line 4, in <module>
      setup(
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/__init__.py", line 108, in setup
      return distutils.core.setup(**attrs)
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
      return run_commands(dist)
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
      dist.run_commands()
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
      self.run_command(cmd)
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/dist.py", line 1221, in run_command
      super().run_command(command)
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/command/install.py", line 68, in run
      return orig.install.run(self)
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/command/install.py", line 697, in run
      self.run_command('build')
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/dist.py", line 1221, in run_command
      super().run_command(command)
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/command/build.py", line 131, in run
      self.run_command(cmd_name)
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/dist.py", line 1221, in run_command
      super().run_command(command)
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 84, in run
      _build_ext.run(self)
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
      self.build_extensions()
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 827, in build_extensions
      build_ext.build_extensions(self)
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
      self._build_extensions_serial()
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
      self.build_extension(ext)
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 246, in build_extension
      _build_ext.build_extension(self, ext)
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 548, in build_extension
      objects = self.compiler.compile(
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 642, in unix_wrap_ninja_compile
      _write_ninja_file_and_compile_objects(
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1557, in _write_ninja_file_and_compile_objects
      _run_ninja_build(
    File "/home/watk681/anaconda3/envs/vid2e/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1900, in _run_ninja_build
      raise RuntimeError(message) from e
  RuntimeError: Error compiling objects for extension
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> esim-torch

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

About the trajectory to simulate saccade moments in ESIM

Hello,

I was trying to simulate the saccade moment explained in this vid2e paper as specified in the original paper to generate a small video from a single static event. Here i have given the trajectory that i used. With that I tried to simulate the the three saccades given the paper.
First i converted the angular positions given in degrees to corresponding quaternions. And I set the camera position as i understood these three movements are pure rotations. But the quarternions i got for the trajectory was bit confusing and i could not generate proper frames for a video from the single still image. Can someone from the authors provide the trajectory that was used for saccade moments? Thank you !

timestamp, x, y, z, qx, qy, qz, qw

0,0.0,0.0,0.0,-0.0044,0.0044,0.0000,1.000
100000000,0.0,0.0,0.0,-0.0044,0,0,1.000
200000000,0.0,0.0,0.0,0.0044,0.0044,-0.0000,1.0000
300000000,0.0,0.0,0.0,0.0044,-0.0044,0.0000,1.000

Rgds,
Udayanga

dataset problem

Error when running esim_torch/generate_events.py - cuda runtime error (209) : no kernel image is available

Thank you for this incredible work. I followed all the setup instructions to set up the environment on google colab. I was able to build esim_torch and run upsampling. I can also run upsampling with device=cuda:0. But when I try to run esim_torch/generate_events.py, I get the following error:

Generating events with cn=0.2, cp=0.2 and rp=0
Processing folder example/upsampled/dirname_does_not_matter... Generating events in example/events/dirname_does_not_matter
  0%|                                                                                                                          | 0/56 [00:00<?, ?it/s]THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1573049306803/work/aten/src/THC/THCGeneral.cpp line=371 error=209 : no kernel image is available for execution on the device
Traceback (most recent call last):
  File "esim_torch/generate_events.py", line 72, in <module>
    process_dir(output_folder, path, args)
  File "esim_torch/generate_events.py", line 40, in process_dir
    sub_events = esim.forward(log_image, timestamp_ns)
  File "/content/drive/MyDrive/NN_682/rpg_vid2e/esim_torch/esim_torch.py", line 49, in forward
    events = self.initialized_forward(images, timestamps)
  File "/content/drive/MyDrive/NN_682/rpg_vid2e/esim_torch/esim_torch.py", line 73, in initialized_forward
    cumsum = event_counts.view(-1).cumsum(dim=0)
RuntimeError: cuda runtime error (209) : no kernel image is available for execution on the device at /opt/conda/conda-bld/pytorch_1573049306803/work/aten/src/THC/THCGeneral.cpp:371
  0%|                                                                                                                          | 0/56 [00:00<?, ?it/s]

I get following when I run test.py

# python test.py 
Loading images
Loading data to GPU
Generating events
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1573049306803/work/aten/src/THC/THCGeneral.cpp line=371 error=209 : no kernel image is available for execution on the device
Traceback (most recent call last):
  File "test.py", line 31, in <module>
    events = esim_torch.forward(log_images, timestamps_ns)
  File "/content/drive/MyDrive/NN_682/rpg_vid2e/esim_torch/esim_torch.py", line 49, in forward
    events = self.initialized_forward(images, timestamps)
  File "/content/drive/MyDrive/NN_682/rpg_vid2e/esim_torch/esim_torch.py", line 73, in initialized_forward
    cumsum = event_counts.view(-1).cumsum(dim=0)
RuntimeError: cuda runtime error (209) : no kernel image is available for execution on the device at /opt/conda/conda-bld/pytorch_1573049306803/work/aten/src/THC/THCGeneral.cpp:371

I understand this means that the GPU is not supported for the pytorch version being used. Following are the details of GPU on colab:

  
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.63.01    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 00000000:00:04.0 Off |                    0 |
| N/A   72C    P8    34W / 149W |      0MiB / 11441MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

I used miniconda on colab to setup the python environment.

conda 4.10.3
Python 3.7.11

Pytorch version

rpg_vid2e# conda list pytorch
# packages in environment at /usr/local/envs/vid2e:
#
# Name                    Version                   Build  Channel
_pytorch_select           0.1                       cpu_0  
pytorch                   1.3.1           py3.7_cuda10.1.243_cudnn7.6.3_0    pytorch

I have to run on colab because I don't have my own GPU. Some threads mention building pytorch from source but I am not familiar the steps to setup an older pytorch version. I was hoping for an easier resolution.
Please let me know what can be done to resolve this. Let me know what other details/logs are needed for more information.

Also, is there a way to run this without GPU? Just for making sure this runs and everything else is okay.

Thanks!

Unable to build wheels for esim-cuda while installing esim_torch

I am trying to install esim_torch as instructed but unable to build wheels for esim-cuda and getting the following error:

ERROR: Command errored out with exit status 1:
command: /home/rishabh/anaconda3/envs/vid2e/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-tk48ulrq/setup.py'"'"'; file='"'"'/tmp/pip-req-build-tk48ulrq/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-56sjj7xj
cwd: /tmp/pip-req-build-tk48ulrq/
Complete output (73 lines):
running bdist_wheel
running build
running build_ext
building 'esim_cuda' extension
creating build
creating build/temp.linux-x86_64-3.7
/usr/bin/nvcc -I/home/rishabh/anaconda3/envs/vid2e/lib/python3.7/site-packages/torch/include -I/home/rishabh/anaconda3/envs/vid2e/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/rishabh/anaconda3/envs/vid2e/lib/python3.7/site-packages/torch/include/TH -I/home/rishabh/anaconda3/envs/vid2e/lib/python3.7/site-packages/torch/include/THC -I/home/rishabh/anaconda3/envs/vid2e/include/python3.7m -c esim_cuda_kernel.cu -o build/temp.linux-x86_64-3.7/esim_cuda_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=esim_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_50,code=sm_50 -std=c++11
/usr/lib/gcc/x86_64-linux-gnu/5/include/mwaitxintrin.h(36): error: identifier "__builtin_ia32_monitorx" is undefined

/usr/lib/gcc/x86_64-linux-gnu/5/include/mwaitxintrin.h(42): error: identifier "__builtin_ia32_mwaitx" is undefined

/home/rishabh/anaconda3/envs/vid2e/lib/python3.7/site-packages/torch/include/c10/util/Half-inl.h(21): error: identifier "__half_as_short" is undefined

/home/rishabh/anaconda3/envs/vid2e/lib/python3.7/site-packages/torch/include/c10/core/TensorTypeSet.h(44): warning: integer conversion resulted in a change of sign

/home/rishabh/anaconda3/envs/vid2e/lib/python3.7/site-packages/torch/include/c10/util/ArrayRef.h(180): warning: constexpr function return is non-constant

/home/rishabh/anaconda3/envs/vid2e/lib/python3.7/site-packages/torch/include/torch/csrc/jit/argument_spec.h(68): error: static assertion failed with "ArgumentInfo is to be a POD struct"

/home/rishabh/anaconda3/envs/vid2e/lib/python3.7/site-packages/torch/include/torch/csrc/autograd/profiler.h(97): warning: attribute "visibility" does not apply here

/home/rishabh/anaconda3/envs/vid2e/lib/python3.7/site-packages/torch/include/torch/csrc/autograd/profiler.h(112): warning: attribute "visibility" does not apply here

/home/rishabh/anaconda3/envs/vid2e/lib/python3.7/site-packages/torch/include/pybind11/detail/common.h(694): error: function call must have a constant value in a constant expression

/home/rishabh/anaconda3/envs/vid2e/lib/python3.7/site-packages/torch/include/pybind11/detail/descr.h(59): error: expression must have a constant value

/home/rishabh/anaconda3/envs/vid2e/lib/python3.7/site-packages/torch/include/pybind11/cast.h(1889): warning: nonstandard use of "auto" to both deduce the type from an initializer and to announce a trailing return type

/usr/include/c++/5/bits/hashtable.h(1526): error: no instance of overloaded function "std::forward" matches the argument list
argument types are: (std::string)
detected during:
instantiation of "std::pair<std::__detail::_Hashtable_base<_Key, _Value, _ExtractKey, _Equal, _H1, _H2, _Hash, _Traits>::iterator, __nv_bool> std::_Hashtable<_Key, _Value, _Alloc, _ExtractKey, _Equal, _H1, _H2, _Hash, _RehashPolicy, _Traits>::_M_emplace(std::true_type, _Args &&) [with _Key=std::string, _Value=std::pair<const std::string, size_t>, _Alloc=std::allocator<std::pair<const std::string, size_t>>, _ExtractKey=std::__detail::_Select1st, _Equal=std::equal_tostd::string, _H1=std::hashstd::string, _H2=std::__detail::_Mod_range_hashing, _Hash=std::__detail::_Default_ranged_hash, _RehashPolicy=std::__detail::_Prime_rehash_policy, _Traits=std::__umap_traits, _Args=<std::string, std::size_t>]"
(726): here
instantiation of "std::_Hashtable<_Key, _Value, _Alloc, _ExtractKey, _Equal, _H1, _H2, _Hash, _RehashPolicy, _Traits>::__ireturn_type std::_Hashtable<_Key, _Value, _Alloc, _ExtractKey, _Equal, _H1, _H2, _Hash, _RehashPolicy, _Traits>::emplace(_Args &&...) [with _Key=std::string, _Value=std::pair<const std::string, size_t>, _Alloc=std::allocator<std::pair<const std::string, size_t>>, _ExtractKey=std::__detail::_Select1st, _Equal=std::equal_tostd::string, _H1=std::hashstd::string, _H2=std::__detail::_Mod_range_hashing, _Hash=std::__detail::_Default_ranged_hash, _RehashPolicy=std::__detail::_Prime_rehash_policy, _Traits=std::__umap_traits, _Args=<std::string, std::size_t>]"
/usr/include/c++/5/bits/unordered_map.h(380): here
instantiation of "std::pair<std::_Hashtable<_Key, std::pair<const _Key, _Tp>, _Alloc, std::__detail::_Select1st, _Pred, _Hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<std::_not<std::_and<std::__is_fast_hash<_Hash>, std::__detail::__is_noexcept_hash<_Key, _Hash>>>::value, false, true>>::iterator, __nv_bool> std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::emplace(_Args &&...) [with _Key=std::string, _Tp=size_t, _Hash=std::hashstd::string, _Pred=std::equal_tostd::string, _Alloc=std::allocator<std::pair<const std::string, size_t>>, _Args=<std::string, unsigned long>]"
/home/rishabh/anaconda3/envs/vid2e/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/ordered_dict.h(363): here
instantiation of "Value &torch::OrderedDict<Key, Value>::insert(K &&, V &&) [with Key=std::string, Value=std::shared_ptrtorch::nn::Module, K=std::string, V=std::shared_ptrtorch::nn::Module]"
/home/rishabh/anaconda3/envs/vid2e/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/ordered_dict.h(369): here
instantiation of "Value &torch::OrderedDict<Key, Value>::insert(Key, Value &&) [with Key=std::string, Value=std::shared_ptrtorch::nn::Module]"
/home/rishabh/anaconda3/envs/vid2e/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/nn/module.h(615): here
instantiation of "std::shared_ptr torch::nn::Module::register_module(std::string, std::shared_ptr) [with ModuleType=torch::nn::Module]"
/home/rishabh/anaconda3/envs/vid2e/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/nn/modules/container/modulelist.h(92): here

/usr/include/c++/5/bits/hashtable.h(1565): error: no instance of overloaded function "std::forward" matches the argument list
argument types are: (void *)
detected during:
instantiation of "std::_Hashtable<_Key, _Value, _Alloc, _ExtractKey, _Equal, _H1, _H2, _Hash, _RehashPolicy, _Traits>::iterator std::_Hashtable<_Key, _Value, _Alloc, _ExtractKey, _Equal, _H1, _H2, _Hash, _RehashPolicy, _Traits>::_M_emplace(std::_Hashtable<_Key, _Value, _Alloc, _ExtractKey, _Equal, _H1, _H2, _Hash, _RehashPolicy, _Traits>::const_iterator, std::false_type, _Args &&) [with _Key=const void *, _Value=std::pair<const void *const, pybind11::detail::instance *>, _Alloc=std::allocator<std::pair<const void *const, pybind11::detail::instance *>>, _ExtractKey=std::__detail::_Select1st, _Equal=std::equal_to<const void *>, _H1=std::hash<const void *>, _H2=std::__detail::_Mod_range_hashing, _Hash=std::__detail::_Default_ranged_hash, _RehashPolicy=std::__detail::_Prime_rehash_policy, _Traits=std::__ummap_traits, _Args=<void *&, pybind11::detail::instance *&>]"
(671): here
instantiation of "std::_Hashtable<_Key, _Value, _Alloc, _ExtractKey, _Equal, _H1, _H2, _Hash, _RehashPolicy, _Traits>::iterator std::_Hashtable<_Key, _Value, _Alloc, _ExtractKey, _Equal, _H1, _H2, _Hash, _RehashPolicy, _Traits>::_M_emplace(std::false_type, _Args &&...) [with _Key=const void *, _Value=std::pair<const void *const, pybind11::detail::instance *>, _Alloc=std::allocator<std::pair<const void *const, pybind11::detail::instance *>>, _ExtractKey=std::__detail::_Select1st, _Equal=std::equal_to<const void *>, _H1=std::hash<const void *>, _H2=std::__detail::_Mod_range_hashing, _Hash=std::__detail::_Default_ranged_hash, _RehashPolicy=std::__detail::_Prime_rehash_policy, _Traits=std::__ummap_traits, _Args=<void *&, pybind11::detail::instance *&>]"
(726): here
instantiation of "std::_Hashtable<_Key, _Value, _Alloc, _ExtractKey, _Equal, _H1, _H2, _Hash, _RehashPolicy, _Traits>::__ireturn_type std::_Hashtable<_Key, _Value, _Alloc, _ExtractKey, _Equal, _H1, _H2, _Hash, _RehashPolicy, _Traits>::emplace(_Args &&...) [with _Key=const void *, _Value=std::pair<const void *const, pybind11::detail::instance *>, _Alloc=std::allocator<std::pair<const void *const, pybind11::detail::instance *>>, _ExtractKey=std::__detail::_Select1st, _Equal=std::equal_to<const void *>, _H1=std::hash<const void *>, _H2=std::__detail::_Mod_range_hashing, _Hash=std::__detail::_Default_ranged_hash, _RehashPolicy=std::__detail::_Prime_rehash_policy, _Traits=std::__ummap_traits, _Args=<void *&, pybind11::detail::instance *&>]"
/usr/include/c++/5/bits/unordered_map.h(1128): here
instantiation of "std::unordered_multimap<_Key, _Tp, _Hash, _Pred, _Alloc>::iterator std::unordered_multimap<_Key, _Tp, _Hash, _Pred, _Alloc>::emplace(_Args &&...) [with _Key=const void *, _Tp=pybind11::detail::instance *, _Hash=std::hash<const void *>, _Pred=std::equal_to<const void *>, _Alloc=std::allocator<std::pair<const void *const, pybind11::detail::instance *>>, _Args=<void *&, pybind11::detail::instance *&>]"
/home/rishabh/anaconda3/envs/vid2e/lib/python3.7/site-packages/torch/include/pybind11/detail/class.h(218): here

/usr/include/c++/5/bits/hashtable.h(1526): error: no instance of overloaded function "std::forward" matches the argument list
argument types are: (PyTypeObject *)
detected during:
instantiation of "std::pair<std::__detail::_Hashtable_base<_Key, _Value, _ExtractKey, _Equal, _H1, _H2, _Hash, _Traits>::iterator, __nv_bool> std::_Hashtable<_Key, _Value, _Alloc, _ExtractKey, _Equal, _H1, _H2, _Hash, _RehashPolicy, _Traits>::_M_emplace(std::true_type, _Args &&) [with _Key=PyTypeObject *, _Value=std::pair<PyTypeObject *const, std::vector<pybind11::detail::type_info *, std::allocator<pybind11::detail::type_info *>>>, _Alloc=std::allocator<std::pair<PyTypeObject *const, std::vector<pybind11::detail::type_info *, std::allocator<pybind11::detail::type_info *>>>>, _ExtractKey=std::__detail::_Select1st, _Equal=std::equal_to<PyTypeObject *>, _H1=std::hash<PyTypeObject *>, _H2=std::__detail::_Mod_range_hashing, _Hash=std::__detail::_Default_ranged_hash, _RehashPolicy=std::__detail::_Prime_rehash_policy, _Traits=std::__umap_traits, _Args=<PyTypeObject *&, std::vector<pybind11::detail::type_info *, std::allocator<pybind11::detail::type_info *>>>]"
(726): here
instantiation of "std::_Hashtable<_Key, _Value, _Alloc, _ExtractKey, _Equal, _H1, _H2, _Hash, _RehashPolicy, _Traits>::__ireturn_type std::_Hashtable<_Key, _Value, _Alloc, _ExtractKey, _Equal, _H1, _H2, _Hash, _RehashPolicy, _Traits>::emplace(_Args &&...) [with _Key=PyTypeObject *, _Value=std::pair<PyTypeObject *const, std::vector<pybind11::detail::type_info *, std::allocator<pybind11::detail::type_info *>>>, _Alloc=std::allocator<std::pair<PyTypeObject *const, std::vector<pybind11::detail::type_info *, std::allocator<pybind11::detail::type_info *>>>>, _ExtractKey=std::__detail::_Select1st, _Equal=std::equal_to<PyTypeObject *>, _H1=std::hash<PyTypeObject *>, _H2=std::__detail::_Mod_range_hashing, _Hash=std::__detail::_Default_ranged_hash, _RehashPolicy=std::__detail::_Prime_rehash_policy, _Traits=std::__umap_traits, _Args=<PyTypeObject *&, std::vector<pybind11::detail::type_info *, std::allocator<pybind11::detail::type_info *>>>]"
/usr/include/c++/5/bits/unordered_map.h(380): here
instantiation of "std::pair<std::_Hashtable<_Key, std::pair<const _Key, _Tp>, _Alloc, std::__detail::_Select1st, _Pred, _Hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<std::_not<std::_and<std::__is_fast_hash<_Hash>, std::__detail::__is_noexcept_hash<_Key, _Hash>>>::value, false, true>>::iterator, __nv_bool> std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::emplace(_Args &&...) [with _Key=PyTypeObject *, _Tp=std::vector<pybind11::detail::type_info *, std::allocator<pybind11::detail::type_info *>>, _Hash=std::hash<PyTypeObject *>, _Pred=std::equal_to<PyTypeObject *>, _Alloc=std::allocator<std::pair<PyTypeObject *const, std::vector<pybind11::detail::type_info *, std::allocator<pybind11::detail::type_info *>>>>, _Args=<PyTypeObject *&, std::vector<pybind11::detail::type_info *, std::allocator<pybind11::detail::type_info *>>>]"
/home/rishabh/anaconda3/envs/vid2e/lib/python3.7/site-packages/torch/include/pybind11/pybind11.h(1619): here

esim_cuda_kernel.cu(39): warning: variable "i0" was declared but never referenced
detected during instantiation of "void count_events_cuda_forward_kernel(const scalar_t *, const scalar_t *, scalar_t *, int64_t *, int, int, int, float, float) [with scalar_t=float]"
(152): here

9 errors detected in the compilation of "/tmp/tmpxft_0000183f_00000000-7_esim_cuda_kernel.cpp1.ii".
error: command '/usr/bin/nvcc' failed with exit status 2

ERROR: Failed building wheel for esim-cuda

I request @danielgehrig18 and others to please help me out to rectify this issue!

A bug

The images are loaded in the format of "RGB":

rpg_vid2e/upsampling/utils/dataset.py

Line 57 in b795a4d

img = img.convert('RGB')

But they are converted to "Gray" from "BGR" before writing to disk:

rpg_vid2e/upsampling/utils/upsampler.py

Line 98 in b795a4d

img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

When I'm do not generate in a for logp, what does the generated event represents?

Hi, I just want to generate in this manner as below:

events = esim.forward(
    log_images,        # torch tensor with type float32, shape T x H x W
    timestamps_ns  # torch tensor with type int64,   shape T 
)

and I get one event. Does this describe events between $ I_0 $ and $ I_{T-1} $? How do you merge the sub_event that generate in a for loop?

Building esim_torch does not work

Hey,

everything worked so far on win10, but the pip install esim_torch step does not work.

Also, it is not clear to me what is meant by conda config --add pytorch, since the command leads to this output:
conda-script.py config: error: argument --prepend/--add: expected 2 arguments

Thats the error i get after pip install esim_torch/:

Processing c:\users\hello\documents\rpg_vid2e\esim_torch
Building wheels for collected packages: esim-cuda
 Building wheel for esim-cuda (setup.py) ... error
 ERROR: Command errored out with exit status 1:
  command: 'C:\Users\hello\anaconda3\envs\vid2e\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"
'C:\\Users\\hello\\AppData\\Local\\Temp\\pip-req-build-10_fou08\\setup.py'"'"'; __file__='"'"'C:\\Users\\hello\\AppData\
\Local\\Temp\\pip-req-build-10_fou08\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().re
place('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\he
llo\AppData\Local\Temp\pip-wheel-jg9alm38'
      cwd: C:\Users\hello\AppData\Local\Temp\pip-req-build-10_fou08\
 Complete output (92 lines):
 running bdist_wheel
 running build
 running build_ext
 C:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\torch\utils\cpp_extension.py:304: UserWarning: Error checking co
mpiler version for cl: [WinError 2] Das System kann die angegebene Datei nicht finden
   warnings.warn(f'Error checking compiler version for {compiler}: {error}')
 building 'esim_cuda' extension
 creating C:\Users\hello\AppData\Local\Temp\pip-req-build-10_fou08\build
 creating C:\Users\hello\AppData\Local\Temp\pip-req-build-10_fou08\build\temp.win-amd64-3.7
 creating C:\Users\hello\AppData\Local\Temp\pip-req-build-10_fou08\build\temp.win-amd64-3.7\Release
 Emitting ninja build file C:\Users\hello\AppData\Local\Temp\pip-req-build-10_fou08\build\temp.win-amd64-3.7\Release\bu
ild.ninja...
 Compiling objects...
 Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
 [1/1] C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.4\bin\nvcc  --use-local-env -Xcompiler /MD -Xcompiler /wd
4819 -Xcompiler /wd4251 -Xcompiler /wd4244 -Xcompiler /wd4267 -Xcompiler /wd4275 -Xcompiler /wd4018 -Xcompiler /wd4190 -
Xcompiler /EHsc -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcudafe --diag_suppress=field_without_d
ll_interface -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=dll_interface_conflic
t_dllexport_assumed -IC:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\torch\include -IC:\Users\hello\anaconda3\env
s\vid2e\lib\site-packages\torch\include\torch\csrc\api\include -IC:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\t
orch\include\TH -IC:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU
Computing Toolkit\CUDA\v11.4\include" -IC:\Users\hello\anaconda3\envs\vid2e\include -IC:\Users\hello\anaconda3\envs\vid
2e\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\ATLMFC\include" "-
IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\include" "-IC:\Program Files (x
86)\Windows Kits\NETFXSDK\4.7.2\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\P
rogram Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.1
9041.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-IC:\Program Files (x86)\Windows Kits\
10\include\10.0.19041.0\cppwinrt" -c C:\Users\hello\AppData\Local\Temp\pip-req-build-10_fou08\esim_cuda_kernel.cu -o C:\
Users\hello\AppData\Local\Temp\pip-req-build-10_fou08\build\temp.win-amd64-3.7\Release\esim_cuda_kernel.obj -D__CUDA_NO_
HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-
relaxed-constexpr -arch=sm_60 -O3 -use_fast_math -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=esim_cuda -D_GLI
BCXX_USE_CXX11_ABI=0
 FAILED: C:/Users/hello/AppData/Local/Temp/pip-req-build-10_fou08/build/temp.win-amd64-3.7/Release/esim_cuda_kernel.obj
 C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.4\bin\nvcc  --use-local-env -Xcompiler /MD -Xcompiler /wd4819 -
Xcompiler /wd4251 -Xcompiler /wd4244 -Xcompiler /wd4267 -Xcompiler /wd4275 -Xcompiler /wd4018 -Xcompiler /wd4190 -Xcompi
ler /EHsc -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcudafe --diag_suppress=field_without_dll_int
erface -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=dll_interface_conflict_dlle
xport_assumed -IC:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\torch\include -IC:\Users\hello\anaconda3\envs\vid2
e\lib\site-packages\torch\include\torch\csrc\api\include -IC:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\torch\i
nclude\TH -IC:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Compu
ting Toolkit\CUDA\v11.4\include" -IC:\Users\hello\anaconda3\envs\vid2e\include -IC:\Users\hello\anaconda3\envs\vid2e\inc
lude "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\ATLMFC\include" "-IC:\Pr
ogram Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\include" "-IC:\Program Files (x86)\Wi
ndows Kits\NETFXSDK\4.7.2\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Program
Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0
\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\inc
lude\10.0.19041.0\cppwinrt" -c C:\Users\hello\AppData\Local\Temp\pip-req-build-10_fou08\esim_cuda_kernel.cu -o C:\Users\
hello\AppData\Local\Temp\pip-req-build-10_fou08\build\temp.win-amd64-3.7\Release\esim_cuda_kernel.obj -D__CUDA_NO_HALF_O
PERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxe
d-constexpr -arch=sm_60 -O3 -use_fast_math -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=esim_cuda -D_GLIBCXX_U
SE_CXX11_ABI=0
 C:/Users/hello/anaconda3/envs/vid2e/lib/site-packages/torch/include\c10/macros/Macros.h(189): warning C4067: Unerwarte
te Token nach PrÃ¤prozessordirektive - Zeilenvorschub erwartet.
 C:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\torch\include\pybind11\detail/common.h(108): warning C4005: "HAV
E_SNPRINTF": Makro-Neudefinition
 C:\Users\hello\anaconda3\envs\vid2e\include\pyerrors.h(490): note: Siehe vorherige Definition von "HAVE_SNPRINTF"
 C:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\torch\include\pybind11\cast.h(1429): error: too few arguments fo
r template template parameter "Tuple"
           detected during instantiation of class "pybind11::detail::tuple_caster<Tuple, Ts...> [with Tuple=std::pair,
Ts=<T1, T2>]"
 (1507): here

 C:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\torch\include\pybind11\cast.h(1503): error: too few arguments fo
r template template parameter "Tuple"
           detected during instantiation of class "pybind11::detail::tuple_caster<Tuple, Ts...> [with Tuple=std::pair,
Ts=<T1, T2>]"
 (1507): here

 c:\users\hello\appdata\local\temp\pip-req-build-10_fou08\esim_cuda_kernel.cu(39): warning: variable "i0" was declared
but never referenced
           detected during instantiation of "void count_events_cuda_forward_kernel(const scalar_t *, const scalar_t *,
scalar_t *, int64_t *, int, int, int, float, float) [with scalar_t=float]"
 (152): here

 2 errors detected in the compilation of "c:/users/hello/appdata/local/temp/pip-req-build-10_fou08/esim_cuda_kernel.cu"
.
 esim_cuda_kernel.cu
 ninja: build stopped: subcommand failed.
 Traceback (most recent call last):
   File "C:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\torch\utils\cpp_extension.py", line 1673, in _run_ninja_
build
     env=env)
   File "C:\Users\hello\anaconda3\envs\vid2e\lib\subprocess.py", line 512, in run
     output=stdout, stderr=stderr)
 subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

 The above exception was the direct cause of the following exception:

 Traceback (most recent call last):
   File "<string>", line 1, in <module>
   File "C:\Users\hello\AppData\Local\Temp\pip-req-build-10_fou08\setup.py", line 17, in <module>
     'build_ext': BuildExtension
   File "C:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\setuptools\__init__.py", line 153, in setup
     return distutils.core.setup(**attrs)
   File "C:\Users\hello\anaconda3\envs\vid2e\lib\distutils\core.py", line 148, in setup
     dist.run_commands()
   File "C:\Users\hello\anaconda3\envs\vid2e\lib\distutils\dist.py", line 966, in run_commands
     self.run_command(cmd)
   File "C:\Users\hello\anaconda3\envs\vid2e\lib\distutils\dist.py", line 985, in run_command
     cmd_obj.run()
   File "C:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\wheel\bdist_wheel.py", line 299, in run
     self.run_command('build')
   File "C:\Users\hello\anaconda3\envs\vid2e\lib\distutils\cmd.py", line 313, in run_command
     self.distribution.run_command(command)
   File "C:\Users\hello\anaconda3\envs\vid2e\lib\distutils\dist.py", line 985, in run_command
     cmd_obj.run()
   File "C:\Users\hello\anaconda3\envs\vid2e\lib\distutils\command\build.py", line 135, in run
     self.run_command(cmd_name)
   File "C:\Users\hello\anaconda3\envs\vid2e\lib\distutils\cmd.py", line 313, in run_command
     self.distribution.run_command(command)
   File "C:\Users\hello\anaconda3\envs\vid2e\lib\distutils\dist.py", line 985, in run_command
     cmd_obj.run()
   File "C:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\setuptools\command\build_ext.py", line 79, in run
     _build_ext.run(self)
   File "C:\Users\hello\anaconda3\envs\vid2e\lib\distutils\command\build_ext.py", line 340, in run
     self.build_extensions()
   File "C:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\torch\utils\cpp_extension.py", line 708, in build_extens
ions
     build_ext.build_extensions(self)
   File "C:\Users\hello\anaconda3\envs\vid2e\lib\distutils\command\build_ext.py", line 449, in build_extensions
     self._build_extensions_serial()
   File "C:\Users\hello\anaconda3\envs\vid2e\lib\distutils\command\build_ext.py", line 474, in _build_extensions_serial
     self.build_extension(ext)
   File "C:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\setuptools\command\build_ext.py", line 196, in build_ext
ension
     _build_ext.build_extension(self, ext)
   File "C:\Users\hello\anaconda3\envs\vid2e\lib\distutils\command\build_ext.py", line 534, in build_extension
     depends=ext.depends)
   File "C:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\torch\utils\cpp_extension.py", line 690, in win_wrap_nin
ja_compile
     with_cuda=with_cuda)
   File "C:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\torch\utils\cpp_extension.py", line 1359, in _write_ninj
a_file_and_compile_objects
     error_prefix='Error compiling objects for extension')
   File "C:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\torch\utils\cpp_extension.py", line 1683, in _run_ninja_
build
     raise RuntimeError(message) from e
 RuntimeError: Error compiling objects for extension
 Error in atexit._run_exitfuncs:
 Traceback (most recent call last):
   File "C:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\colorama\ansitowin32.py", line 59, in closed
     return stream.closed
 ValueError: underlying buffer has been detached
 ----------------------------------------
 ERROR: Failed building wheel for esim-cuda
 Running setup.py clean for esim-cuda
Failed to build esim-cuda
Installing collected packages: esim-cuda
   Running setup.py install for esim-cuda ... error
   ERROR: Command errored out with exit status 1:
    command: 'C:\Users\hello\anaconda3\envs\vid2e\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"
'"'C:\\Users\\hello\\AppData\\Local\\Temp\\pip-req-build-10_fou08\\setup.py'"'"'; __file__='"'"'C:\\Users\\hello\\AppDat
a\\Local\\Temp\\pip-req-build-10_fou08\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().
replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\User
s\hello\AppData\Local\Temp\pip-record-zogla9qx\install-record.txt' --single-version-externally-managed --compile --insta
ll-headers 'C:\Users\hello\anaconda3\envs\vid2e\Include\esim-cuda'
        cwd: C:\Users\hello\AppData\Local\Temp\pip-req-build-10_fou08\
   Complete output (94 lines):
   running install
   running build
   running build_ext
   C:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\torch\utils\cpp_extension.py:304: UserWarning: Error checking
compiler version for cl: [WinError 2] Das System kann die angegebene Datei nicht finden
     warnings.warn(f'Error checking compiler version for {compiler}: {error}')
   building 'esim_cuda' extension
   creating C:\Users\hello\AppData\Local\Temp\pip-req-build-10_fou08\build
   creating C:\Users\hello\AppData\Local\Temp\pip-req-build-10_fou08\build\temp.win-amd64-3.7
   creating C:\Users\hello\AppData\Local\Temp\pip-req-build-10_fou08\build\temp.win-amd64-3.7\Release
   Emitting ninja build file C:\Users\hello\AppData\Local\Temp\pip-req-build-10_fou08\build\temp.win-amd64-3.7\Release\
build.ninja...
   Compiling objects...
   Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
   [1/1] C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.4\bin\nvcc  --use-local-env -Xcompiler /MD -Xcompiler /
wd4819 -Xcompiler /wd4251 -Xcompiler /wd4244 -Xcompiler /wd4267 -Xcompiler /wd4275 -Xcompiler /wd4018 -Xcompiler /wd4190
-Xcompiler /EHsc -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcudafe --diag_suppress=field_without
_dll_interface -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=dll_interface_confl
ict_dllexport_assumed -IC:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\torch\include -IC:\Users\hello\anaconda3\e
nvs\vid2e\lib\site-packages\torch\include\torch\csrc\api\include -IC:\Users\hello\anaconda3\envs\vid2e\lib\site-packages
\torch\include\TH -IC:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA G
PU Computing Toolkit\CUDA\v11.4\include" -IC:\Users\hello\anaconda3\envs\vid2e\include -IC:\Users\hello\anaconda3\envs\v
id2e\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\ATLMFC\include"
"-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\include" "-IC:\Program Files
(x86)\Windows Kits\NETFXSDK\4.7.2\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:
\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0
.19041.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-IC:\Program Files (x86)\Windows Kit
s\10\include\10.0.19041.0\cppwinrt" -c C:\Users\hello\AppData\Local\Temp\pip-req-build-10_fou08\esim_cuda_kernel.cu -o C
:\Users\hello\AppData\Local\Temp\pip-req-build-10_fou08\build\temp.win-amd64-3.7\Release\esim_cuda_kernel.obj -D__CUDA_N
O_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --exp
t-relaxed-constexpr -arch=sm_60 -O3 -use_fast_math -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=esim_cuda -D_G
LIBCXX_USE_CXX11_ABI=0
   FAILED: C:/Users/hello/AppData/Local/Temp/pip-req-build-10_fou08/build/temp.win-amd64-3.7/Release/esim_cuda_kernel.o
bj
   C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.4\bin\nvcc  --use-local-env -Xcompiler /MD -Xcompiler /wd4819
-Xcompiler /wd4251 -Xcompiler /wd4244 -Xcompiler /wd4267 -Xcompiler /wd4275 -Xcompiler /wd4018 -Xcompiler /wd4190 -Xcom
piler /EHsc -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcudafe --diag_suppress=field_without_dll_i
nterface -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=dll_interface_conflict_dl
lexport_assumed -IC:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\torch\include -IC:\Users\hello\anaconda3\envs\vi
d2e\lib\site-packages\torch\include\torch\csrc\api\include -IC:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\torch
\include\TH -IC:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Com
puting Toolkit\CUDA\v11.4\include" -IC:\Users\hello\anaconda3\envs\vid2e\include -IC:\Users\hello\anaconda3\envs\vid2e\i
nclude "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\ATLMFC\include" "-IC:\
Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\include" "-IC:\Program Files (x86)\
Windows Kits\NETFXSDK\4.7.2\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Progr
am Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041
.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\i
nclude\10.0.19041.0\cppwinrt" -c C:\Users\hello\AppData\Local\Temp\pip-req-build-10_fou08\esim_cuda_kernel.cu -o C:\User
s\hello\AppData\Local\Temp\pip-req-build-10_fou08\build\temp.win-amd64-3.7\Release\esim_cuda_kernel.obj -D__CUDA_NO_HALF
_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-rela
xed-constexpr -arch=sm_60 -O3 -use_fast_math -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=esim_cuda -D_GLIBCXX
_USE_CXX11_ABI=0
   C:/Users/hello/anaconda3/envs/vid2e/lib/site-packages/torch/include\c10/macros/Macros.h(189): warning C4067: Unerwar
tete Token nach PrÃ¤prozessordirektive - Zeilenvorschub erwartet.
   C:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\torch\include\pybind11\detail/common.h(108): warning C4005: "H
AVE_SNPRINTF": Makro-Neudefinition
   C:\Users\hello\anaconda3\envs\vid2e\include\pyerrors.h(490): note: Siehe vorherige Definition von "HAVE_SNPRINTF"
   C:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\torch\include\pybind11\cast.h(1429): error: too few arguments
for template template parameter "Tuple"
             detected during instantiation of class "pybind11::detail::tuple_caster<Tuple, Ts...> [with Tuple=std::pair
, Ts=<T1, T2>]"
   (1507): here

   C:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\torch\include\pybind11\cast.h(1503): error: too few arguments
for template template parameter "Tuple"
             detected during instantiation of class "pybind11::detail::tuple_caster<Tuple, Ts...> [with Tuple=std::pair
, Ts=<T1, T2>]"
   (1507): here

   c:\users\hello\appdata\local\temp\pip-req-build-10_fou08\esim_cuda_kernel.cu(39): warning: variable "i0" was declare
d but never referenced
             detected during instantiation of "void count_events_cuda_forward_kernel(const scalar_t *, const scalar_t *
, scalar_t *, int64_t *, int, int, int, float, float) [with scalar_t=float]"
   (152): here

   2 errors detected in the compilation of "c:/users/hello/appdata/local/temp/pip-req-build-10_fou08/esim_cuda_kernel.c
u".
   esim_cuda_kernel.cu
   ninja: build stopped: subcommand failed.
   Traceback (most recent call last):
     File "C:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\torch\utils\cpp_extension.py", line 1673, in _run_ninj
a_build
       env=env)
     File "C:\Users\hello\anaconda3\envs\vid2e\lib\subprocess.py", line 512, in run
       output=stdout, stderr=stderr)
   subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

   The above exception was the direct cause of the following exception:

   Traceback (most recent call last):
     File "<string>", line 1, in <module>
     File "C:\Users\hello\AppData\Local\Temp\pip-req-build-10_fou08\setup.py", line 17, in <module>
       'build_ext': BuildExtension
     File "C:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\setuptools\__init__.py", line 153, in setup
       return distutils.core.setup(**attrs)
     File "C:\Users\hello\anaconda3\envs\vid2e\lib\distutils\core.py", line 148, in setup
       dist.run_commands()
     File "C:\Users\hello\anaconda3\envs\vid2e\lib\distutils\dist.py", line 966, in run_commands
       self.run_command(cmd)
     File "C:\Users\hello\anaconda3\envs\vid2e\lib\distutils\dist.py", line 985, in run_command
       cmd_obj.run()
     File "C:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\setuptools\command\install.py", line 61, in run
       return orig.install.run(self)
     File "C:\Users\hello\anaconda3\envs\vid2e\lib\distutils\command\install.py", line 545, in run
       self.run_command('build')
     File "C:\Users\hello\anaconda3\envs\vid2e\lib\distutils\cmd.py", line 313, in run_command
       self.distribution.run_command(command)
     File "C:\Users\hello\anaconda3\envs\vid2e\lib\distutils\dist.py", line 985, in run_command
       cmd_obj.run()
     File "C:\Users\hello\anaconda3\envs\vid2e\lib\distutils\command\build.py", line 135, in run
       self.run_command(cmd_name)
     File "C:\Users\hello\anaconda3\envs\vid2e\lib\distutils\cmd.py", line 313, in run_command
       self.distribution.run_command(command)
     File "C:\Users\hello\anaconda3\envs\vid2e\lib\distutils\dist.py", line 985, in run_command
       cmd_obj.run()
     File "C:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\setuptools\command\build_ext.py", line 79, in run
       _build_ext.run(self)
     File "C:\Users\hello\anaconda3\envs\vid2e\lib\distutils\command\build_ext.py", line 340, in run
       self.build_extensions()
     File "C:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\torch\utils\cpp_extension.py", line 708, in build_exte
nsions
       build_ext.build_extensions(self)
     File "C:\Users\hello\anaconda3\envs\vid2e\lib\distutils\command\build_ext.py", line 449, in build_extensions
       self._build_extensions_serial()
     File "C:\Users\hello\anaconda3\envs\vid2e\lib\distutils\command\build_ext.py", line 474, in _build_extensions_seri
al
       self.build_extension(ext)
     File "C:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\setuptools\command\build_ext.py", line 196, in build_e
xtension
       _build_ext.build_extension(self, ext)
     File "C:\Users\hello\anaconda3\envs\vid2e\lib\distutils\command\build_ext.py", line 534, in build_extension
       depends=ext.depends)
     File "C:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\torch\utils\cpp_extension.py", line 690, in win_wrap_n
inja_compile
       with_cuda=with_cuda)
     File "C:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\torch\utils\cpp_extension.py", line 1359, in _write_ni
nja_file_and_compile_objects
       error_prefix='Error compiling objects for extension')
     File "C:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\torch\utils\cpp_extension.py", line 1683, in _run_ninj
a_build
       raise RuntimeError(message) from e
   RuntimeError: Error compiling objects for extension
   Error in atexit._run_exitfuncs:
   Traceback (most recent call last):
     File "C:\Users\hello\anaconda3\envs\vid2e\lib\site-packages\colorama\ansitowin32.py", line 59, in closed
       return stream.closed
   ValueError: underlying buffer has been detached
   ----------------------------------------
ERROR: Command errored out with exit status 1: 'C:\Users\hello\anaconda3\envs\vid2e\python.exe' -u -c 'import sys, setup
tools, tokenize; sys.argv[0] = '"'"'C:\\Users\\hello\\AppData\\Local\\Temp\\pip-req-build-10_fou08\\setup.py'"'"'; __fil
e__='"'"'C:\\Users\\hello\\AppData\\Local\\Temp\\pip-req-build-10_fou08\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"
', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'
"'"'))' install --record 'C:\Users\hello\AppData\Local\Temp\pip-record-zogla9qx\install-record.txt' --single-version-ext
ernally-managed --compile --install-headers 'C:\Users\hello\anaconda3\envs\vid2e\Include\esim-cuda' Check the logs for f
ull command output.  ```

undefined symbol

I build esim-py with "python setup.py install", but I got the error blew when import esim-py
undefined symbol: _ZN2cv12VideoCapture4readERNS_3MatE

When "ldd -r esim_py.cpython-37m-x86_64-linux-gnu.so", many undefined symbol error.

Unable to install "esim_torch"

     command: /home/sk/anaconda3/envs/vid2e/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-1hhvtkhi/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-1hhvtkhi/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-sycca_tk
         cwd: /tmp/pip-req-build-1hhvtkhi/
    Complete output (11 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-1hhvtkhi/setup.py", line 9, in <module>
        'esim_cuda_kernel.cu',
      File "/home/sk/anaconda3/envs/vid2e/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 470, in CUDAExtension
        library_dirs += library_paths(cuda=True)
      File "/home/sk/anaconda3/envs/vid2e/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 548, in library_paths
        paths.append(_join_cuda_home(lib_dir))
      File "/home/sk/anaconda3/envs/vid2e/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1129, in _join_cuda_home
        raise EnvironmentError('CUDA_HOME environment variable is not set. '
    OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.
    ----------------------------------------
WARNING: Discarding file:///home/sk/rpg_vid2e-master/esim_torch. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

dataset problem

I have download dataset from weblink (http://rpg.ifi.uzh.ch/data/VID2E/ncaltech_syn_images.zip),
but didn't find the video sequences used for event conversion.
After unzip ncaltech_syn_images.zip downloaded from former weblink, I obtain folder sim-N-Caltech101, there is only ./sim-N-Caltech101/class/image_%04d.npz files.
The synthetic N-Caltech101 dataset, as well as video sequences used for event conversion can be found [here]. For each sample of each class it contains events in the form class/image_%04d.npz and

Pip install esim_py always fails. Can you release a compiled version?

That probably would have better stability. Thanks a lot!

Latest rename introduced bugs

52bb812 leads to:

(vid2e) tobias@QUTLAB:~/robot/rpg_vid2e/esim_torch$ python test.py
Traceback (most recent call last):
  File "test.py", line 13, in <module>
    refractory_period=0)
TypeError: __init__() got an unexpected keyword argument 'refractory_period'
(vid2e) tobias@QUTLAB:~/robot/rpg_vid2e/esim_torch$ python test_single_pixel.py 
Traceback (most recent call last):
  File "test_single_pixel.py", line 17, in <module>
    refractory_period=refractory_period)
TypeError: __init__() got an unexpected keyword argument 'refractory_period'
(vid2e) tobias@QUTLAB:~/robot/rpg_vid2e/esim_torch$

errors in installing rpg_vid2e

Hi all,
It raises errors (attached error.txt file) when I run pip install . I did follow every step of your instructions at Github, and there is no errors before pip install . Could you please help me figure it out? Thank you very much. I am looking forward to your reply.

python upsample.py error, missing pretrained model

Hi thank you for the release.
If I’m not mistaken no one mentioned this issue but I’ve seen that after running out-of-the-box the command for sampling
python upsampling/upsample.py --input_dir=example/original --output_dir=example/upsampled
it seems to be missing some pretrained models, that I don’t really see where they should be taken.

The following is the output:

Traceback (most recent call last): File "/home/pellerito/rpg_vid2e/upsampling/upsample.py", line 27, in <module> main() File "/home/pellerito/rpg_vid2e/upsampling/upsample.py", line 22, in main upsampler = Upsampler(input_dir=flags.input_dir, output_dir=flags.output_dir) File "/home/pellerito/rpg_vid2e/upsampling/utils/upsampler.py", line 26, in __init__ self.interpolator = Interpolator(path, None) File "/home/pellerito/rpg_vid2e/upsampling/utils/interpolator.py", line 82, in __init__ self._model = tf.compat.v2.saved_model.load(model_path) File "/home/pellerito/anaconda3/envs/vid2e/lib/python3.9/site-packages/tensorflow/python/saved_model/load.py", line 864, in load result = load_internal(export_dir, tags, options)["root"] File "/home/pellerito/anaconda3/envs/vid2e/lib/python3.9/site-packages/tensorflow/python/saved_model/load.py", line 878, in load_internal loader_impl.parse_saved_model_with_debug_info(export_dir)) File "/home/pellerito/anaconda3/envs/vid2e/lib/python3.9/site-packages/tensorflow/python/saved_model/loader_impl.py", line 60, in parse_saved_model_with_debug_info saved_model = _parse_saved_model(export_dir) File "/home/pellerito/anaconda3/envs/vid2e/lib/python3.9/site-packages/tensorflow/python/saved_model/loader_impl.py", line 118, in parse_saved_model raise IOError( OSError: SavedModel file does not exist at: /home/pellerito/rpg_vid2e/upsampling/utils/../../pretrained_models/film_net/Style/saved_model/{saved_model.pbtxt|saved_model.pb}

How to save the output vedios using matplotlib in this program

import argparse
from operator import sub
import os
import esim_torch
import numpy as np
import glob
import cv2
import tqdm
import torch
import matplotlib.pyplot as plt

def is_valid_dir(subdirs, files):
return len(subdirs) == 1 and len(files) == 1 and "timestamps.txt" in files and "imgs" in subdirs

#def close_event():

plt.close() #timer calls this function after 3 seconds and closes the window

def process_dir(outdir, indir, args):
print(f"Processing folder {indir}... Generating events in {outdir}")
os.makedirs(outdir, exist_ok=True)

# constructor
esim = esim_torch.ESIM(args.contrast_threshold_negative,
                                       args.contrast_threshold_positive,
                                       args.refractory_period_ns)

timestamps = np.genfromtxt(os.path.join(indir, "timestamps.txt"), dtype="float64")
timestamps_ns = (timestamps * 1e9).astype("int64")
timestamps_ns = torch.from_numpy(timestamps_ns).cuda()

image_files = sorted(glob.glob(os.path.join(indir, "imgs", "*.png")))

pbar = tqdm.tqdm(total=len(image_files)-1)
num_events = 0

#fig = plt.figure()
fig, ax = plt.subplots()
#timer = fig.canvas.new_timer(interval = 300) #creating a timer object and setting an interval of 3000 milliseconds
#timer.add_callback(close_event)

counter = 0
ims = []
for image_file, timestamp_ns in zip(image_files, timestamps_ns):
    image = cv2.imread(image_file, cv2.IMREAD_GRAYSCALE)
    log_image = np.log(image.astype("float32") / 255 + 1e-5)
    log_image = torch.from_numpy(log_image).cuda()

    sub_events = esim.forward(log_image, timestamp_ns)

    # for the first image, no events are generated, so this needs to be skipped
    if sub_events is None:
        continue

          
    print("Plotting")
    first_few_events = {k: v.cpu().numpy() for k,v in sub_events.items()}
    image_color = np.stack([image,image,image],-1)
    image_color[first_few_events['y'], first_few_events['x'], :] = 0
    image_color[first_few_events['y'], first_few_events['x'], first_few_events['p']] = 255
    #plt.imshow(image_color)
    
    #timer.start()
    #plt.show()
    
    im = ax.imshow(image_color, animated=True)
 		
    ims.append([im])
    
    sub_events = {k: v.cpu() for k, v in sub_events.items()}    
    num_events += len(sub_events['t'])

    # do something with the events
    np.savez(os.path.join(outdir, "%010d.npz" % counter), **sub_events)
    pbar.set_description(f"Num events generated: {num_events}")
    pbar.update(1)
    counter += 1

ani = animation.ArtistAnimation(fig, ims, interval=50, blit=True,repeat_delay=100)

plt.show()

if name == "main":
parser = argparse.ArgumentParser("""Generate events from a high frequency video stream""")
parser.add_argument("--contrast_threshold_negative", "-cn", type=float, default=0.2)
parser.add_argument("--contrast_threshold_positive", "-cp", type=float, default=0.2)
parser.add_argument("--refractory_period_ns", "-rp", type=int, default=0)
parser.add_argument("--input_dir", "-i", default="", required=True)
parser.add_argument("--output_dir", "-o", default="", required=True)
args = parser.parse_args()

print(f"Generating events with cn={args.contrast_threshold_negative}, cp={args.contrast_threshold_positive} and rp={args.refractory_period_ns}")

for path, subdirs, files in os.walk(args.input_dir):
    if is_valid_dir(subdirs, files):
        output_folder = os.path.join(args.output_dir, os.path.relpath(path, args.input_dir))

        process_dir(output_folder, path, args)

error building wheel for esim_py and esim_torch

up until this point all of the setup was successful, but trying to build the esim modules causes a build error, im not sure whats wrong.
error.log

About implementation of segmentation

Hi, Thanks for this amazing work.
I am kind of interested in the application to segmentation, but I did not see the code of segmentation.
Could you please share it?
About EST, did you apply event representation to evsegnet in TensorFlow?
Or you change to PyTorch ?
Thanks for your help.

I can run on gtx1080ti, but can't run on rtx 2080ti or rtx 3090, I always meet the error below.

RuntimeError: CUDA error: no kernel image is available for execution on the device

And my docker environment is: cuda10.2, cudnn7, pytorch1.6, python3.7.

All gpus, include gtx1080ti, rtx2080ti, or rtx3090 driver are all >= 470. I can only run the docker image successfully on gtx1080ti with the same code.

I don't know the reason.

Save upsampled images in video format

"Why store the upsampling result in images:
-Images support random access from a dataloader. A video file, for example, can typically only be accessed sequentally when we try to avoid loading the whole video into RAM."

"Same sequence can be accessed by multiple processes (e.g. PyTorch num_workers > 1).
Well established C++ interface to load images. This is useful to generate events on the fly (needed for contrast threshold randomization) in C++ code without loading data in Python first. If there is a need to store the resulting sequences in a different format, raise an issue (feature request) on this GitHub repository."

Why would you need to load the whole video at once in RAM? I think storing your upsampled video in .mp4/.avi would be a huge space saver! For Online generation of events, you can use a pytorch "iterable dataset" (see example here: https://github.com/etienne87/pytorch-streamloader)

Besides, is online generation even possible? right now when loading a folder or images python does not control the callback right so you effectively do load all events in RAM to be accessed on python side via numpy array no?

Missing the last image of the original images

In the first step, upsampling operation, the last image of the original images is not stored on the upsampled images.
For example/seq2, there are 9 images in the original images. 1 image is interpolated into the original images between two consecutive images. But the upsampled results only have 16 images due to the lack of the last image of the original images. This results in a missing section of the events stream. Its timestamps also confirm this matter.
It is the same for example/seq0.

Why use a negative mean? What is the below function?

        self.negmean = self._move_to_device(torch.Tensor([x * -1 for x in mean]).view(3, 1, 1), self.device)
        revNormalize = transforms.Normalize(mean=negmean, std=std)
        self.TP = transforms.Compose([revNormalize])

Segmentation fault error

When I run the last command, I meet the error:

THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1573049306803/work/aten/src/THC/THCGeneral.cpp line=371 error=98 : invalid device function
Traceback (most recent call last):
  File "esim_torch/generate_events.py", line 72, in <module>
    process_dir(output_folder, path, args)
  File "esim_torch/generate_events.py", line 40, in process_dir
    sub_events = esim.forward(log_image, timestamp_ns)
  File "/code/rpg_vid2e/esim_torch/esim_torch.py", line 49, in forward
    events = self.initialized_forward(images, timestamps)
  File "/code/rpg_vid2e/esim_torch/esim_torch.py", line 73, in initialized_forward
    cumsum = event_counts.view(-1).cumsum(dim=0)
RuntimeError: cuda runtime error (98) : invalid device function at /opt/conda/conda-bld/pytorch_1573049306803/work/aten/src/THC/THCGeneral.cpp:371
  0%|                                                      | 0/56 [00:00<?, ?it/s]
Segmentation fault (core dumped)

What's the reason?

Time taken for converting videos into synthetic events

Hi,

Thank you for the code. I want to convert 10 sec long videos with 30fps into synthetic events. Could you please tell approximate time it will take to do so using vid2e (for one video)?

Thanks and Regards,
Ayush

Installing ESIM Error

Hi!

I am having an error by trying to build the python bindings for ESIM.
When I run pip install . in the esim_py folder, I got the following error:

  ERROR: Command errored out with exit status 1:
   command: /home/anaconda/envs/graph-wyx/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-6disk9bu/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-6disk9bu/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-1ntzf6xt
       cwd: /tmp/pip-req-build-6disk9bu/
  Complete output (67 lines):
  running bdist_wheel
  running build
  running build_ext
  -- The C compiler identification is GNU 7.5.0
  -- The CXX compiler identification is GNU 7.5.0
  -- Check for working C compiler: /usr/bin/cc
  -- Check for working C compiler: /usr/bin/cc -- works
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- Check for working CXX compiler: /usr/bin/c++
  -- Check for working CXX compiler: /usr/bin/c++ -- works
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  CMake Error at CMakeLists.txt:4 (find_package):
    By not providing "Findpybind11.cmake" in CMAKE_MODULE_PATH this project has
    asked CMake to find a package configuration file provided by "pybind11",
    but CMake did not find one.
  
    Could not find a package configuration file provided by "pybind11" with any
    of the following names:
  
      pybind11Config.cmake
      pybind11-config.cmake
  
    Add the installation prefix of "pybind11" to CMAKE_PREFIX_PATH or set
    "pybind11_DIR" to a directory containing one of the above files.  If
    "pybind11" provides a separate development package or SDK, be sure it has
    been installed.
  
  
  -- Configuring incomplete, errors occurred!
  See also "/tmp/pip-req-build-6disk9bu/build/temp.linux-x86_64-3.6/CMakeFiles/CMakeOutput.log".
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-req-build-6disk9bu/setup.py", line 62, in <module>
      zip_safe=False,
    File "/home/anaconda/envs/graph-wyx/lib/python3.6/site-packages/setuptools/__init__.py", line 144, in setup
      return distutils.core.setup(**attrs)
    File "/home/anaconda/envs/graph-wyx/lib/python3.6/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/home/anaconda/envs/graph-wyx/lib/python3.6/distutils/dist.py", line 955, in run_commands
      self.run_command(cmd)
    File "/home/anaconda/envs/graph-wyx/lib/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/home/anaconda/envs/graph-wyx/lib/python3.6/site-packages/wheel/bdist_wheel.py", line 223, in run
      self.run_command('build')
    File "/home/anaconda/envs/graph-wyx/lib/python3.6/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/home/anaconda/envs/graph-wyx/lib/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/home/anaconda/envs/graph-wyx/lib/python3.6/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/home/anaconda/envs/graph-wyx/lib/python3.6/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/home/anaconda/envs/graph-wyx/lib/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/tmp/pip-req-build-6disk9bu/setup.py", line 31, in run
      self.build_extension(ext)
    File "/tmp/pip-req-build-6disk9bu/setup.py", line 49, in build_extension
      subprocess.check_call(['cmake', ext.sourcedir] + cmake_args, cwd=self.build_temp, env=env)
    File "/home/anaconda/envs/graph-wyx/lib/python3.6/subprocess.py", line 311, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['cmake', '/tmp/pip-req-build-6disk9bu', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/tmp/pip-req-build-6disk9bu/build/lib.linux-x86_64-3.6', '-DPYTHON_EXECUTABLE=/home/anaconda/envs/graph-wyx/bin/python', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1.
  ----------------------------------------
  ERROR: Failed building wheel for esim-py
  Running setup.py clean for esim-py
Failed to build esim-py
Installing collected packages: esim-py
    Running setup.py install for esim-py ... error
    ERROR: Command errored out with exit status 1:
     command: /home/anaconda/envs/graph-wyx/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-6disk9bu/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-6disk9bu/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-hm8h6ckx/install-record.txt --single-version-externally-managed --compile --install-headers /home/anaconda/envs/graph-wyx/include/python3.6m/esim-py
         cwd: /tmp/pip-req-build-6disk9bu/
    Complete output (69 lines):
    running install
    running build
    running build_ext
    -- The C compiler identification is GNU 7.5.0
    -- The CXX compiler identification is GNU 7.5.0
    -- Check for working C compiler: /usr/bin/cc
    -- Check for working C compiler: /usr/bin/cc -- works
    -- Detecting C compiler ABI info
    -- Detecting C compiler ABI info - done
    -- Detecting C compile features
    -- Detecting C compile features - done
    -- Check for working CXX compiler: /usr/bin/c++
    -- Check for working CXX compiler: /usr/bin/c++ -- works
    -- Detecting CXX compiler ABI info
    -- Detecting CXX compiler ABI info - done
    -- Detecting CXX compile features
    -- Detecting CXX compile features - done
    CMake Error at CMakeLists.txt:4 (find_package):
      By not providing "Findpybind11.cmake" in CMAKE_MODULE_PATH this project has
      asked CMake to find a package configuration file provided by "pybind11",
      but CMake did not find one.
    
      Could not find a package configuration file provided by "pybind11" with any
      of the following names:
    
        pybind11Config.cmake
        pybind11-config.cmake
    
      Add the installation prefix of "pybind11" to CMAKE_PREFIX_PATH or set
      "pybind11_DIR" to a directory containing one of the above files.  If
      "pybind11" provides a separate development package or SDK, be sure it has
      been installed.
    
    
    -- Configuring incomplete, errors occurred!
    See also "/tmp/pip-req-build-6disk9bu/build/temp.linux-x86_64-3.6/CMakeFiles/CMakeOutput.log".
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-6disk9bu/setup.py", line 62, in <module>
        zip_safe=False,
      File "/home/anaconda/envs/graph-wyx/lib/python3.6/site-packages/setuptools/__init__.py", line 144, in setup
        return distutils.core.setup(**attrs)
      File "/home/anaconda/envs/graph-wyx/lib/python3.6/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/home/anaconda/envs/graph-wyx/lib/python3.6/distutils/dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "/home/anaconda/envs/graph-wyx/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/home/anaconda/envs/graph-wyx/lib/python3.6/site-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "/home/anaconda/envs/graph-wyx/lib/python3.6/distutils/command/install.py", line 545, in run
        self.run_command('build')
      File "/home/anaconda/envs/graph-wyx/lib/python3.6/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/home/anaconda/envs/graph-wyx/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/home/anaconda/envs/graph-wyx/lib/python3.6/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/home/anaconda/envs/graph-wyx/lib/python3.6/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/home/anaconda/envs/graph-wyx/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/tmp/pip-req-build-6disk9bu/setup.py", line 31, in run
        self.build_extension(ext)
      File "/tmp/pip-req-build-6disk9bu/setup.py", line 49, in build_extension
        subprocess.check_call(['cmake', ext.sourcedir] + cmake_args, cwd=self.build_temp, env=env)
      File "/home/anaconda/envs/graph-wyx/lib/python3.6/subprocess.py", line 311, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['cmake', '/tmp/pip-req-build-6disk9bu', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/tmp/pip-req-build-6disk9bu/build/lib.linux-x86_64-3.6', '-DPYTHON_EXECUTABLE=/home/anaconda/envs/graph-wyx/bin/python', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1.
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/anaconda/envs/graph-wyx/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-6disk9bu/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-6disk9bu/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-hm8h6ckx/install-record.txt --single-version-externally-managed --compile --install-headers /home/anaconda/envs/graph-wyx/include/python3.6m/esim-py Check the logs for full command output.

Error when using WEB_APP

Hello, thanks for your release.
I'd like to use the WEB APP. I've successfully installed esim_py and esim_torch. But when I executed the command conda list | grep 'esim , I got esim-torch 0.0.0 pypi_0 pyp instead of "esim_cuda". And the "Streamlit run ..." command got the error "ImportError: cannot import name 'EventSimulator_torch' from 'esim_torch' " . I can't figure out what's the problem and loo forward to your help.
Thanks.

Ask for the code to generate videos from a still image by moving camera

Hi,
Can anyone please share the code that was used to generate set of frames when you have one still image?
In the paper it is mentioned that "We map the still images onto a 2D plane and simulate an event camera moving in front of this plane in a saccadic motion, as was done for the original N-Caltech101 dataset". But i could not find the corresponding code in the code base availabe. Thank you !

Best Rgds,
Kashita Udayanga

uzh-rpg / rpg_vid2e Goto Github PK

rpg_vid2e's Introduction

Video to Events: Recycling Video Datasets for Event Cameras

News

Web App and Interactive Demo

Dataset

Installation

Installation

Adaptive Upsampling

esim_py

esim_torch

Example

rpg_vid2e's People

Contributors

Stargazers

Watchers

Forkers

rpg_vid2e's Issues

timestamp, x, y, z, qx, qy, qz, qw

9 errors detected in the compilation of "/tmp/tmpxft_0000183f_00000000-7_esim_cuda_kernel.cpp1.ii". error: command '/usr/bin/nvcc' failed with exit status 2

plt.close() #timer calls this function after 3 seconds and closes the window

Recommend Projects

Recommend Topics

Recommend Org

9 errors detected in the compilation of "/tmp/tmpxft_0000183f_00000000-7_esim_cuda_kernel.cpp1.ii".
error: command '/usr/bin/nvcc' failed with exit status 2