hovren / crisp Goto Github PK
View Code? Open in Web Editor NEWCamera-to-IMU calibration and synchronization toolbox
License: GNU General Public License v3.0
Camera-to-IMU calibration and synchronization toolbox
License: GNU General Public License v3.0
Hi, I used the calibration toolbox to calibrate the data given by the paper, and the result was different every time.
In the experimental section of the paper, figure 4 shows the error relative to the set of reference parameters. The frequency error is within +/ -0.2, and the time offset error is between -0.008 and 0004. The results obtained within this range can achieve a nicely stabilised video, but the results of my calibration each time exceed the above range, resulting in the use of calibrated parameters can not generate a good stabilised video.
Here are some of the results I got(rotation.mp4):
Frequency(Hz) -1.245274 0.7984233 -1.291282
Offset(s) 0.004855215 0.004855141 0.004858229
I have several doubts, the first is how I calculate the matrix k,
the vector wc and omega,
readout is not 1/fps ¿why?
and finally where the stabilized video is exported (I can run the program perfectly but I do not see results)
I am using a gopro hero5.
Hi, I have some troubles using crisp with my own samples. I took three sets of qvga@30fps video with 30Hz gyroscope and one set of 720p@60fps and 100Hz gyroscope and every one of those is giving me a Division by Zero error in the calibration.initialize(...) . I've tried to tune my setting values in your gopro_dataset_example.py, but the error is still there...
➜ examples python gopro_dataset_example.py dataset/samples/Set1/test.avi
Creating video stream from dataset/samples/Set1/test.avi
Creating gyro stream from dataset/samples/Set1/test_gyro.csv
Post processing gyroscope data to remove frequency spike noise
Creating calibrator
Estimating time offset and camera to gyroscope rotation. Guessing gyro rate = 30.00
/usr/lib/python2.7/dist-packages/numpy/core/_methods.py:55: RuntimeWarning: Mean of empty slice.
warnings.warn("Mean of empty slice.", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/core/_methods.py:67: RuntimeWarning: invalid value encountered in double_scalars
ret = ret.dtype.type(ret / rcount)
Traceback (most recent call last):
File "gopro_dataset_example.py", line 96, in <module>
calibrator.initialize(gyro_rate=GYRO_RATE_GUESS)
File "build/bdist.linux-x86_64/egg/crisp/calibration.py", line 128, in initialize
File "build/bdist.linux-x86_64/egg/crisp/calibration.py", line 230, in find_initial_offset
File "build/bdist.linux-x86_64/egg/crisp/timesync.py", line 107, in sync_camera_gyro
File "build/bdist.linux-x86_64/egg/crisp/znccpyr.py", line 273, in find_shift_pyr
File "build/bdist.linux-x86_64/egg/crisp/znccpyr.py", line 175, in zncc
ZeroDivisionError: division by zero
The debug output is saying :
DEBUG:crisp:Number of slices: 84
DEBUG:crisp:Generating optical flow magnitude. This can take minutes depending on video length
DEBUG:root:pyramid size = 7
DEBUG:root:size of first element 2243
DEBUG:root:size of last element 35
Do you have any idea of what I can be doing wrong?
Hi, I used the result of calibration toolbox to do video stabilization for the video in the paper. I found that according to the time offset in the calibration result, the stabilized video was very bad.
After scaling and offsetting the gyroscope data(as follow), I can get satisfactory results.
coeffies = [1.60190524e-03, -2.79751510e+00] # rotation.mp4
gyro_times = (gyro_times + coeffies[1]) / (1 - coeffies[0])
My question is, the video stabilization in the paper mentioned in the Readme directly use the time offset( 2.8333333333333335 as follow) of the calibration toolbox?
rotation.mp4 calibration result by the calibration toolbox:
gyro_rate = 853.660712254363
time_offset = 2.8333333333333335
gbias_x = -0.01001219994597313
gbias_y = -0.008216488048614853
gbias_z = 0.010610096301797737
rot_x = 0.022640457925138826
rot_y = -0.0068401509599219324
rot_z = -1.5426536394739456
Looking forward to your reply.
Hi, @hovren
Thank you for your excellent work. After calibration, I need to stabilize the video. But I'm not sure which method to use. I want to stabilize the video on a UAV in real time. Do you have some recommendations?
Thank you in advance.
At line 208, I think "final_ok = np.intersect1d(ok_track, retracked_ok)" is wrong ,it should be "final_ok=ok_track[retracked_ok]". Because ok_track is forward_track and backward_track 's index, but retracked_ok is forward_first and backward_last 's index . They are not the same index. So they can't do the "intersect1d" operate. "retracked_ok" is "ok_track" 's index.
Hi, when I run gopro_dataset_example.py with sample video in gopro-gyro-dataset, I got following errors in camera.py in line:
rn = np.sqrt((X[0,:] - wx)**2 + (X[1,:] - wy)**2)
AttributeError: 'numpy.ndarray' object has no attribute 'sqrt'
in function apply.
All three video: rotation.MP4, walk.MP4, rccar.MP4 will fall into this error after some successfully computation for previous calling.
I dumped the X data and found that X will be like below when reporting this error.
print(X) -->
[[array([-0.44746115])]
[array([0.02795225])]
[array([1.])]]
Could you help me found out why?
Thanks a lot!
Trying to calibrate camera - gyroscope pair. Camera runs at 30 fps, Gyro at 100 Hz.
The example file 'gopro_dataset_example.py', when modified to suit our dataset, runs for 2 - 3 minutes and fails with the following message:
runfile('/crisp/gopro_dataset_example.py', args='/Set3/set3.mov', wdir='/crisp')
Creating video stream from /Set3/set3.mov
Creating gyro stream from /Set3/set3_gyro.csv
Post processing L3G4200D gyroscope data to remove frequency spike noise
Creating calibrator
Estimating time offset and camera to gyroscope rotation. Guessing gyro rate = 100.00
Running calibration. This can take a few minutes.
Calibration failed. Reason "Number of calls to function has reached maxfev = 800."
/crisp/gopro_dataset_example.py:101: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
print('Calibration failed. Reason "{}"'.format(e.message))
To exit: use 'exit', 'quit', or Ctrl-D.
An exception has occurred, use %tb to see the full traceback.
SystemExit: -2
Traceback:
Traceback (most recent call last):
File "<ipython-input-12-64f97a132da6>", line 1, in <module>
runfile('/crisp/gopro_dataset_example.py', args='/Set3/set3.mov', wdir='/crisp')
File "/usr/local/lib/python2.7/dist-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 714, in runfile
execfile(filename, namespace)
File "/usr/local/lib/python2.7/dist-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 81, in execfile
builtins.execfile(filename, *where)
File "/crisp/gopro_dataset_example.py", line 102, in <module>
sys.exit(-2)
SystemExit: -2
In the example python file, there are quite a few parameters which are not very obvious:
Care to explain these ones:
Also, in the *_reference.csv, please explain these column headers:
What happens if I do not already know the gyro biases? Do we need to provide an estimate?
I was trying to install this, but it errors on build that it is missing a crisp/fastintegrate/fastintegrate.c
file. I see that in the gitignore there as .c file is excluded. Where am I suppose to download this file so I can compile the project?
I am on ubuntu 14.04 and here is the error:
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/crisp
copying crisp/cli.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/stream.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/rotations.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/calibration.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/znccpyr.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/videoslice.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/l3g4200d.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/remove_slp.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/tracking.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/__init__.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/timesync.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/imu.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/ransac.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/camera.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/pose.py -> build/lib.linux-x86_64-2.7/crisp
running build_ext
building 'crisp.fastintegrate' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/crisp
creating build/temp.linux-x86_64-2.7/crisp/fastintegrate
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c crisp/fastintegrate/fastintegrate.c -o build/temp.linux-x86_64-2.7/crisp/fastintegrate/fastintegrate.o
x86_64-linux-gnu-gcc: error: crisp/fastintegrate/fastintegrate.c: No such file or directory
x86_64-linux-gnu-gcc: fatal error: no input files
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 4
Edit: I am using the master branch.
I'm running crisp with my own camera and gyro data, it raises an error:
/usr/lib/python2.7/dist-packages/pkg_resources/init.py:1224: UserWarning: /home/lili/.python-eggs is writable by group/others and vulnerable to attack when used with get_resource_filename. Consider a more secure location (set with .set_extraction_path or the PYTHON_EGG_CACHE environment variable).
warnings.warn(msg, UserWarning)
/usr/local/lib/python2.7/dist-packages/crisp-2.2-py2.7-linux-x86_64.egg/crisp/camera.py:181: RuntimeWarning: invalid value encountered in true_divide
Traceback (most recent call last):
File "gopro_dataset_example.py", line 75, in
calibrator.initialize(gyro_rate=GYRO_RATE_GUESS)
File "build/bdist.linux-x86_64/egg/crisp/calibration.py", line 127, in initialize
File "build/bdist.linux-x86_64/egg/crisp/calibration.py", line 256, in find_initial_rotation
File "build/bdist.linux-x86_64/egg/crisp/videoslice.py", line 52, in estimate_rotation
File "build/bdist.linux-x86_64/egg/crisp/rotations.py", line 306, in estimate_rotation_procrustes_ransac
File "build/bdist.linux-x86_64/egg/crisp/ransac.py", line 24, in RANSAC
File "build/bdist.linux-x86_64/egg/crisp/rotations.py", line 289, in
File "build/bdist.linux-x86_64/egg/crisp/rotations.py", line 72, in procrustes
File "/home/lili/.local/lib/python2.7/site-packages/numpy/linalg/linalg.py", line 1359, in svd
u, s, vt = gufunc(a, signature=signature, extobj=extobj)
File "/home/lili/.local/lib/python2.7/site-packages/numpy/linalg/linalg.py", line 99, in _raise_linalgerror_svd_nonconvergence
raise LinAlgError("SVD did not converge")
numpy.linalg.linalg.LinAlgError: SVD did not converge
How can I solve this?
Hi, @hovren, I have a few questions:
Can the crisp toolbox obtain the translation between camera and IMU?
How can I obtain the readout time of a camera?
Thank you in advance!
I'm using the OpenCV camera model. After fixing several code issues and running crisp, I get the following error:
Creating video stream from /home/sergiu/Projects/test_sequences/building1.mp4
Creating gyro stream from /home/sergiu/Projects/test_sequences/building1_gyro.csv
Creating calibrator
Estimating time offset and camera to gyroscope rotation. Guessing gyro rate = 200.00
Traceback (most recent call last):
File "parrot_ardrone_dataset.py", line 70, in <module>
calibrator.initialize(gyro_rate=GYRO_RATE_GUESS)
File "/home/sergiu/Projects/crisp/crisp/calibration.py", line 125, in initialize
R = self.find_initial_rotation()
File "/home/sergiu/Projects/crisp/crisp/calibration.py", line 290, in find_initial_rotation
assert data.shape == (6, len(gyro_axes))
AssertionError
gyro_axes
remains empty. I assume that estimating the initial rotation fails resulting in the error. Is this correct?
It would be useful to be able to install using a simple pip install crisp
.
Thanks for making this library public! I was testing it out with the following sample code:
import crisp
gyro_ms = 5.05050505048712
f = 1000.06549452
im_h, im_w = 1080, 1920
cx = im_w*0.5
cy = im_h*0.5
cam_mat = [[f, 0, cx],
[0, f, cy],
[0, 0, 1 ]]
disto = [-0.20100871814959385, 0.048642904244843899, 0, 0, -0.0063799856646590239]
gyro = crisp.GyroStream.from_csv("gyro.csv")
camera_model = crisp.OpenCVCameraModel((im_h, im_w), 59.94, 0, cam_mat, disto)
video = crisp.VideoStream.from_file(camera_model, "vid.mp4")
calibrator = crisp.AutoCalibrator(video, gyro)
calibrator.initialize(gyro_ms)
result = calibrator.calibrate() # Dict of calibrated parameters
print(result)
But I got the following error.
Traceback (most recent call last):
File "calib.py", line 20, in <module>
calibrator.initialize(gyro_ms)
File "/home/matt/.local/lib/python3.8/site-packages/crisp/calibration.py", line 124, in initialize
time_offset = self.find_initial_offset()
File "/home/matt/.local/lib/python3.8/site-packages/crisp/calibration.py", line 226, in find_initial_offset
time_offset = timesync.sync_camera_gyro(flow, frame_times, self.gyro.data.T, gyro_times, levels=pyramids)
File "/home/matt/.local/lib/python3.8/site-packages/crisp/timesync.py", line 107, in sync_camera_gyro
ishift = znccpyr.find_shift_pyr(flow_mag, gyro_mag, levels)
UnboundLocalError: local variable 'flow_mag' referenced before assignment
I am using python3.8
and installed crisp with python3 -m pip install crisp
. I did have to change one function in as matplotlib doesn't have normpdf anymore. I changed it to from scipy.stats import norm
instead.
Any idea how to fix this error?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.