geohot / twitchslam Goto Github PK

View Code? Open in Web Editor NEW

965.0 49.0 208.0 210.03 MB

A toy implementation of monocular SLAM written while livestreaming

License: MIT License

Python 100.00%

twitchslam's Introduction

twitchslam

A toy implementation of monocular SLAM written while livestreaming.

Stream

https://www.twitch.tv/tomcr00s3

Will be streaming again tomorrow. May 28, 2018

By the end of the day, twitchslam will be decent.

Usage

export REVERSE=1   # Hack for reverse video
export F=500       # Focal length (in px)

./slam.py <video.mp4>

# good example
F=525 ./slam.py videos/test_freiburgxyz525.mp4

# ground truth
F=525 ./slam.py videos/test_freiburgrpy525.mp4 videos/test_freiburgrpy525.npz

# kitti example
REVERSE=1 F=984 ./slam.py videos/test_kitti984_reverse.mp4

# extract ground truth
tools/parse_ground_truth.py videos/groundtruth/freiburgrpy.txt videos/test_freiburgrpy525.npz

Classes

Frame -- An image with extracted features
Point -- A 3-D point in the Map and it's 2-D Frame correspondences
Map -- A collection of points and frames
Display2D -- SDL2 display of the current image
Display3D -- Pangolin display of the current map

Libraries Used

SDL2 for 2-D display
cv2 for feature extraction
pangolin for 3-D display
g2opy for optimization (soon!)

Rendered Scene Test

./fakeslam.py

NOTE: The test currently doesn't work reliably. It seems adding a small amount of Gaussian noise to the point positions can cause the optimizer to fall into really bad local minima. This may just be caused by poor initialization, as I'm not sure how stable Essential matricies are.

TODO: Investigate the stability of Essential matrix recovery.

DONE

BUGFIX: Why is window initting small? (fixed? ish, zoom is broken sometimes)
BUGFIX: Turning doesn't work well (fixed?)
Stop using essential matrix for pose estimation once you have a track (done)
Add kinematic model (done)
Run g2o to only optimize the latest pose (done)
Add search by projection to refind old map points (done)
Check if points are in the field of view of the camera (done)
BUGFIX: Fix moving up! (y axis flipped? nah, it's okay)
Add loading and saving of map support

TODO

Investigate if we need KeyFrames!
BUGFIX: Improve lockups to happen less
Improve init to not need REVERSE environment variable
Add optimizer for F
Add multiscale feature extractor
Add Linux libraries and OS check
Profile and speed up more (tomorrow 6/1/18)
Profile with flame!
Search by projection less stupidly
Don't add all points to the optimization graph for pose at least
Add automated test for freiburg running on commit
Check accuracy with ground truth

LICENSE

All my code is MIT licensed. Videos and libraries follow their respective licenses.

twitchslam's People

Stargazers

Watchers

Forkers

jfrux energee roblight adeebshihadeh fahamjv alpslee sentientmachine jonbryan ronaldcotton number0000009 ompugao aaronmill93 szuer2017 jobajuba yanuartadityan allendoss ikthap rabi3elbeji chensihan bvn-architecture 174high shafaypro quantum56 kbu9299 kelvinson codingsmith rragesh letsdodatascience danielcranston karankatiyar92 jesusoctavioas collector-m cypreess mavaras rfic ironiksk jp-dev benjamesbabala rdn86 hzm8341 soonhwan-kwon sloan8633 xuejunxie buanderie xlbetlog migsr22 csq20081052 sapchan slam7f3 afonsohfontes safisec whrfriends mmajewsk mchighlight zhuoligetu123 jonekee byeb fengyuhen prazvan apandher cuulee manoj652 luovang gryph66 pgh79 hanisa1 meh369 weisili2016 richardsj jingweiz konnpaku-youmu capasitore songtaolee busry liyong2123 romapirogov shadowbone kenyals1 joserfjuniorllms longjohncoder crazysimpson jmsktm lhutyra wpv1999 tahirhafeez wukh0312 shyuu fuzzybuddy103 elsuizo danylee19 cstezcan songgc rayje stjordanis swagato-c johanub ayushmankumar7 jameszhangjp dnlrkaan codeinsleep

twitchslam's Issues

should i set PYSDL2_DLL_PATH ?

Traceback (most recent call last):
File "slam.py", line 9, in
from display import Display
File "/home/gh0st/Desktop/twitchslam/display.py", line 1, in
import sdl2
File "/usr/local/lib/python2.7/dist-packages/sdl2/init.py", line 2, in
from .dll import get_dll_file, _bind
File "/usr/local/lib/python2.7/dist-packages/sdl2/dll.py", line 115, in
raise ImportError(exc)
ImportError: could not find any library for SDL2 (PYSDL2_DLL_PATH: unset)

how to import the lib

How to import the .so libraries?

[Question] Shi-Tomasi vs. FAST keypoint detector?

Just wondering, is there a reason why cv2.goodFeaturesToTrack() is being used instead of ORB's built-in FAST keypoint detector? Is it faster or more robust?

how to compile g2o on my machine?

I always get the error, module not found for import g2o

ValueError: ndarray is not C-contiguous

*** frame 2/628 ***
Matches: 1581 -> 1581 -> 1215 -> 1214
Culled: 0 points
Pose: 0.022129
Traceback (most recent call last):
File "./slam.py", line 218, in
slam.process_frame(frame, None if gt_pose is None else np.linalg.inv(gt_pose[i]))
File "./slam.py", line 86, in process_frame
for m_idx in f1.kd.query_ball_point(projs[i], 2):
File "ckdtree.pyx", line 910, in scipy.spatial.ckdtree.cKDTree.query_ball_point
File "stringsource", line 654, in View.MemoryView.memoryview_cwrapper
File "stringsource", line 349, in View.MemoryView.memoryview.cinit
ValueError: ndarray is not C-contiguous

AttributeError: module 'pypangolin' has no attribute 'DrawCameras'

This issue only appear if pangolin is compiled from here. How would I add DrawCameras() to the original repository?

slower and slower as frames are processed

I noticed that the frame processing time gets higher and higher as new images come in, even by disabling the optimization step.
Is there a way to handle this slow down?

ERROR

gur@ugur-Mac ugur@ugur-Mac twitchslam-master % F=525 python3 slam.py videos/test_drone.mp4
pygame 2.0.1 (SDL 2.0.14, Python 3.9.2)
Hello from the pygame community. https://www.pygame.org/contribute.html
using camera 1024x576 with F 140.000000
pygame 2.0.1 (SDL 2.0.14, Python 3.9.2)
Hello from the pygame community. https://www.pygame.org/contribute.html

*** frame 0/628 ***
Traceback (most recent call last):
File "/Users/ugur/Public/twitchslam-master/slam.py", line 218, in
slam.process_frame(frame, None if gt_pose is None else np.linalg.inv(gt_pose[i]))
File "/Users/ugur/Public/twitchslam-master/slam.py", line 31, in process_frame
frame = Frame(self.mapp, img, self.K, verts=verts)
File "/Users/ugur/Public/twitchslam-master/frame.py", line 84, in init
self.id = tid if tid is not None else mapp.add_frame(self)
AttributeError: 'Map' object has no attribute 'add_frame'

PLEASE, HELP ME % F=525 python3 slam.py videos/test_drone.mp4
pygame 2.0.1 (SDL 2.0.14, Python 3.9.2)
Hello from the pygame community. https://www.pygame.org/contribute.html
using camera 1024x576

with F 140.000000

pygame 2.0.1 (SDL 2.0.14, Python 3.9.2)
Hello from the pygame community. https://www.pygame.org/contribute.html

can not work with test_freiburgrpy525.mp4

Colmap algos useful?

Is there any overlap in math for twitchslam

https://colmap.github.io

ValueError: src.shape[0] must be equal or larger than 8.

I keep running into the following error and I'm not sure whats causing it. When I attempting a ransac calculation the following error pops up, and it's not consistent, sometime I can run the code without any errors whatsoever, sometime it's at different time stamps through the video the error will pop up. Any help will be appreciated!!!

/home/runie/.conda/envs/slam/bin/python /home/runie/Desktop/Code/open3d/main.py 
UserWarning: Using SDL2 binaries from pysdl2-dll 2.30.2
/home/runie/.conda/envs/slam/lib/python3.9/site-packages/skimage/measure/fit.py:672: RuntimeWarning: divide by zero encountered in scalar divide
  return np.ceil(np.log(nom) / np.log(denom))
Traceback (most recent call last):
  File "/home/runie/Desktop/Code/open3d/main.py", line 34, in <module>
    main(file=vid)
  File "/home/runie/Desktop/Code/open3d/main.py", line 20, in main
    process_frame(frame)
  File "/home/runie/Desktop/Code/open3d/slam.py", line 56, in process_frame
    idx1, idx2, Rt = match_frames(frames[-1], frames[-2])
  File "/home/runie/Desktop/Code/open3d/frame.py", line 70, in match_frames
    model, inliers = ransac(
  File "/home/runie/.conda/envs/slam/lib/python3.9/site-packages/skimage/_shared/utils.py", line 270, in fixed_func
    return func(*args, **kwargs)
  File "/home/runie/.conda/envs/slam/lib/python3.9/site-packages/skimage/measure/fit.py", line 944, in ransac
    model.estimate(*data_inliers)
  File "/home/runie/.conda/envs/slam/lib/python3.9/site-packages/skimage/transform/_geometric.py", line 529, in estimate
    self._setup_constraint_matrix(src, dst)
  File "/home/runie/.conda/envs/slam/lib/python3.9/site-packages/skimage/transform/_geometric.py", line 319, in _setup_constraint_matrix
    raise ValueError('src.shape[0] must be equal or larger than 8.')
ValueError: src.shape[0] must be equal or larger than 8.

Process finished with exit code 1