Giter Club home page Giter Club logo

3d-eye-tracker's Introduction

3D Eye Tracker

This software aims to provide an easy-to-compile C++ implementation of a 3D eye-tracking method.

Our software is designed for a wearable eye-tracking scnenario where a user wears a headset with an eye camera(s) that is equipped with infrared (IR) illuminations. The camera can capture close-up shots of the user's eyes.

headset_and_image

With this software, you can obtain the following information:

  • 3D information:
    • Gaze (normal) vector
    • Eyeball center w.r.t the camera
    • Pupil diameter [m]
  • 2D information:
    • Pupil ellipse size [pixel] and angle [deg.]
    • (Glint position(s) [pixel], to be merged soon...)

Sample video on youtube (this was taken from a preliminary version and will be updated with a newer version soon).

Prerequisite

Currently, the software is compatible with Visual Studio 2015. The test was done on a 64bit Windows 10 machine only.

Most of external libraries (for vc14, x64) are bundled in ./external, yet you still need to install the following two libraries:

To setup your own Visual Studio project, you need to use CMake. You can find a windows installer here.

How to compile

  1. Clone the repository [email protected]:YutaItoh/3D-Eye-Tracker.git to your local folder (e.g., /your-local-path/3D-Eye-Tracker)
  2. Open ./CMakeLists.txt and edit the OpenCV path and the Boost path depending on your environment. For example, the default values are as follows:
    • set(BOOST_ROOT "C:/SDK/boost_1_62_0")
    • set(OpenCV_DIR "C:/SDK/opencv-3.1.0/build")
  3. Start CMake GUI and set a source code path (e.g., /your-local-path/3D-Eye-Tracker) and a project build path (e.g., /your-local-path/3D-Eye-Tracker/build_vc14_x64)

4. Press `Generate` buttons, then choose a right build environment (Visual Studio 14 2015 Win64)

5. Open `3d_eye_tracker.sln` in the build folder, and compile `main` project (recommend to set it as a startup project)

How to run

The default setting uses a sinlge camera via the DirectShow filter. Check your camera name on a video capture program (e.g., Skype) and set the name in main.cpp:

		case InputMode::CAMERA_MONO:
			eyecams[0]=std::make_unique<eyecamera::EyeCameraDS>("Pupil Cam1 ID0");

In this case, the program tries to find a camera named "Pupil Cam1 ID0".

Once the program started, it initializes a 3D eye model from 2D pupil observatios:

In this initialization step, a user needs to smoothly rotate his/her eye to capture various 2D shapes of the eye.

After the step, we get 3D eye tracking:

Tips:

Some debug keys are pre-assigned for a better control of the software:

  • p: Takes some more 2D pupil observations. Useful when estimated 3D eye model is incorrect due to not-well-distributed 2D observations
  • r: Resets the 3D eye model and 2D observations and restarts the initialization step
  • ESC: Exit the program

Acknowledgements

This program integrated/modified several existing codes. Especially,

3d-eye-tracker's People

Contributors

nuncobdurat avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

3d-eye-tracker's Issues

Unable to connect to Pupil Cam

Hi. I am currently trying to run your code and already have verified that the camera itself is capable of streaming images through Pupil Labs' official application (pupil-capture).
Whenever the program reaches the part that tries to create the class enumerator for DirectShow, it throws null pointer exception.
`
void DirectShowFrameGrabber::initGraph()
{
// Create the System Device Enumerator.
AutoComPtr< ICreateDevEnum > pDevEnum;
AutoComPtr< IEnumMoniker > pEnum;

HRESULT hr = pDevEnum.CoCreateInstance( CLSID_SystemDeviceEnum, NULL, CLSCTX_INPROC_SERVER );
if ( SUCCEEDED( hr ) )
	// Create an enumerator for the video capture category.
	hr = pDevEnum->CreateClassEnumerator(CLSID_VideoInputDeviceCategory, &pEnum.p, 0 );

`

In debugging mode, it throws a read access violation exception after executing the very last line. The reason of the exception says pEnum.p was nullptr.
The only change that I made after receiving the pupil lab device was to install their device drivers, which made two camera devices listed under libusbK Usb Devices in Device Manager window.
I am not really sure if this is either a hardware or software issue, but I really want to get the image streaming under C++ development environment so that I can combine this feature with my other projects.
Please let me know if you have any idea or need any further information.

Thank you in advance

camera features

Hi, you mention that the camera has an infrared LED, so the camera should not have an infrared filter? and is and is a minimum frame rate necessary?

Thanks and regards.

Debug Assertion Error in Visual Studio

Hi.
Whenever I call,
cv::findContours(threshLow, contoursLow, CV_RETR_EXTERNAL, CV_CHAIN_APPROX_NONE);
it produces assertion error after returning from the function.
Do you know why this happens?

Not an issue, just a question regarding accuracy

First of all, I want to address that I really thank for all of your work.
I am really being benefited by using your program since all the program and I use for my research topic runs under C++ and there is no official C++ support by pupil labs.
The reason why I am starting this thread is that I am wondering how accurate or "close" this program tries to resemble the official pupil labs' pupil detector algorithm in terms of detection accuracy.
I know your calibration routine is simply to rotate eyeball in initialization stages while pupil labs ask to follow the marker with the world camera.
Your program runs 2D ellipse fitting with several filters but does it essentially perform the same detection algorithms?
When my eyes are half-way closed, the detector finds a false pupil area which tends to be located outside of my Iris.
I know that this can be filtered out by computing the reliability. Regarding this matter, does it perform same reliability computation?

One last question is that pupil capture program by pupil labs gives a diameter of Pupil in mm unit. Do you know how they calculate this even though I have never calibrated the camera?
I know you provide the camera intrinsic matrix.

Anyway, I really appreciate your work. Thank you

Error from pupilFitter.h

I'm running only the pupilFitter.h file for my current project. When I built the code, it showed the errors:
error: non-const lvalue reference to type 'cv::Mat' cannot bind to a temporary of type 'cv::Mat'

int darkestPixel = getDarkestPixelBetter(gray(cv::Rect(darkestPixelConfirm.x, darkestPixelConfirm.y, size, size))); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ pupilFitter.h:545:32: note: passing argument to parameter 'I' here
int getDarkestPixelBetter(Mat& I)
at line:77. What is the problem and how do I resolve this?
Thanks

Non-Code Related: HMD Eyetracker Suggestion

What eyetracker do you suggest using for hobby projects/home made applications ?

If there you used an off the shelf solution like pupil tracker can you indicate which one?

What do you suggest for someone who wants to build his own eye tracker ?

how can i check pupil diameter?

How are you?
I need pupil diameter for my research.
I'am checking your code for get the pupil diameter.
Can you explain it for easy?

algorithm of this project?

Hello,i stay tuned this project Recently,and study some algorithm about this project by source code.
but,Can you provide an introduction to this algorithms?

When using a web camera to track my eyes, I can't always get the gaze vector, why?

Nice job! But I met a few problems when running your eye-tracker.
I used a web camera (WIN2 USB2.0 PC Camera) to test the eye tracker. However, the result showed that I can't always obtain the gaze (normal) vector of my eye ball. Is there any solutions for this? I want the gaze vector could be obtained stably.

Should I change another camera or update the camera calibration information in the "cameraintrinsics_eye.txt" file? Or I have to keep the positions of the camera and my head in a more stable status?

Another problem is that is there any paper or document that describe your eye tracking algorithm? because I'd like to learn more details about your work.

Many thanks!

Cannot find Boost libraries

Title says it all, but for more info:

CMake Error at C:/Program Files/CMake/share/cmake-3.7/Modules/FindBoost.cmake:1793 (message):
Unable to find the requested Boost libraries.

Boost version: 1.62.0

Boost include path: C:/local/boost_1_62_0

Could not find the following static Boost libraries:

      boost_filesystem
      boost_system
      boost_timer
      boost_thread
      boost_date_time
      boost_chrono
      boost_regex
      boost_serialization

Some (but not all) of the required Boost libraries were found. You may
need to install these additional Boost libraries. Alternatively, set
BOOST_LIBRARYDIR to the directory containing Boost libraries or BOOST_ROOT
to the location of Boost.
Call Stack (most recent call first):
CMakeLists.txt:54 (find_package)

CMake Error at CMakeLists.txt:55 (find_package):
By not providing "FindOpenCV.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "OpenCV", but
CMake did not find one.

Could not find a package configuration file provided by "OpenCV" with any
of the following names:

OpenCVConfig.cmake
opencv-config.cmake

Add the installation prefix of "OpenCV" to CMAKE_PREFIX_PATH or set
"OpenCV_DIR" to a directory containing one of the above files. If "OpenCV"
provides a separate development package or SDK, be sure it has been
installed.

IR hardware

what kinds of IR hardware you are using ? currently I run your code on my window PC, it will use the webcam, it could not provide the correct result. Your input should be eye region, right?

error C2440

Hello! I followed the instruction in readme.txt. while I running 3d_eye_tracker.sln in the build folder, error C2440 occurred. The detail showed as follows:

Error C2440 '': cannot convert from 'initializer list' to 'singleeyefitter::Ellipse2D' main E:\program\gaze tracking\3D-Eye-Tracker master\singleeyefitter\singleeyefitter\singleeyefitter.h 42

How to fix this error? Thanks!

Exception: getDarkestPixelBetter

Hello,

I am a student from the TU of Vienna and we are currently doing some research on nystagmus. For our research we are using the HMD FOVE, which additionally to the screen inside of the HMD has to two cameras checking the eyes. Your software would perfectly fit for our purpose, to analyses the jittering of the eye.
However, when I use your software, I get an odd error. Maybe you have some input, where the error comes from.

Error in command:
debugmessage1

Error in Visual Studio:
debugmessage2

The image/video coming from the FOVE looks likes this:
fove_eyes
Can it be, that the black boarder around the eyes somehow interferes with the pupil detection?

Thanks in advance,
Stefan

directshowframegrabber don't find pupil lab cams

How are you?
I use the window 10, visual studio 2015, MSI VR notebook.
First I installed the usb installer and I checked device manager and running capture program.
But when I run for testing, directshowframegrabber don't find pupil lab cam.
they found only the webcam in notebook.
do you have any idea or advice?

Unable to start program

I'm using Visual Studio 2019 to compile the program and got an error message when trying to debug the program.
image

In the output log, got an output:
1>------ Build started: Project: main, Configuration: Debug x64 ------
1>main.cpp
1>Unknown compiler version - please run the configure tests and report the results
1>Please define _WIN32_WINNT or _WIN32_WINDOWS appropriately.

Running in Visual Studio 2013 (vc120)

I am trying to compile your solution with Visual Studio, but it seems like compiled external libraries are not usable under vc120 platform toolset. (I have only seen platform conflict issues with Ceres for now).
And I am having a hard time to compile ceres libraries under vc12. Do you have any compiled libraries or any link that I can refer to?

The program was stuck at the beginning

Hi.
I'm using 64-bit windows 10, visual studio 2015 and basically meet the requisites you mentioned.
I compiled it with CMake and got the "build" folder with correct files in it. It all seems well and I successfully run the main project.

But the program got stuck at main.cpp:224, in the function "fetchFrame". I checked out the content of the "fetchFrame" function. It is originated from "DirectShowFrameGrabber::getFrame" in DirectShowFrameGrabber.cpp, and the program stopped at line 526 of DirectShowFrameGrabber.cpp ( m_cv.wait(lk) ). The below image shows the result, it just stopped in the middle of the execution, and I didn't see the expected eye tracking frame or something.
image

I wonder if my compilation step was wrong. I open the .sln file in visual studio, set the "main" project as start up project, and press F5 to execute it.

Or it's because I didn't use the pupil-labs cameras? I used the built-in webcam on my laptop. Because I didn't care about the accuracy for now, I just want to run the program successfully first.

Thank you!

How to compile x86

hi, i want build x86 of vs2015,i have Configurated opencv (vc14, x86) and boost boost_1_62_0-msvc-32.0-.exe,but the compilation failed,what else do I need to do next?
and some others .dll and .lib in ./external,does its affect compilation?

Assertion failed

Hello,
I'm trying to run this program on another camera.However, when I use video source as input I get this error "Assertion failed: lambda(0) >= lambda(1) ".
Should I change the camera calibration information in the "cameraintrinsics_eye.txt" file?

Access is Denied

I did all the Prerequisite and requirements, But when run the program
get this error: ...\ALL_BUILD access is denied.
Do you know how can I solve this problem !?

Theoretical problem

Thank you for sharing, is there any relevant essay or document to help understand the corresponding methodology?

Possibility of c# solution

I have tested the solution in c++ it works nicely, however I would like to do some C# programming, is there a possibility to create this as a library (compile it to a static library (.lib)) and then use it in C++/CLI wrapper ?
The issue would how to create the C++/CLI wrapper, As I have seen some complex opencv datatypes as result of some of the computations...

Do you think it's easier to translate c++ code to c# and then use opencvsharp or something similar?

Visual Studio 2017

What opencv and boost versions did you use to run the build on visual studio 2017?
Opencv gives out errors specifically related to undeclared identifiers in single eye fitters which probably is related to the changes made moving from opencv3 to 4. But I can't find opencv3.x/vs141 compatibility.

Is it possible to run the program in linux?

I'll like to run this program in a raspberry pi. can i compile without visual studio in ubuntu?
If it can, what method would you have to use?

thanks for your time, regards.

How to manage camera rotation and position ?

Hello,

I am trying to use your program on another camera with a different angle and distance. I saw in the code that some values are hard coded, like the focal length . Do you have some parameters to change the camera position and rotation too ?

Regards,

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.