Laser pointer detection sample application, demonstrating a minimal setup for

• HoloLens camera image processing using OpenCV for Unity
• unprojection of camera pixels to 3D coordinates in the world space.

Tested on HoloLens 2 and Gen 1.

• Unity 2019.4.x LTS + UWP Build Support + Visual Studio 2019
• MRTK 2.4.0
• OpenCV for Unity 2.3.9

Basic setup of HoloLens dev environment (Emulator is not required): Install the tools

After cloning, open the project in Unity and import the OpenCV for Unity asset.
Setup OpenCVForUnity: Tools > OpenCV for Unity > Set Plugin Import Settings

Note: MRTK 2.4.0 package files are already included and configured in this repository.

For HoloLens 2, configure Unity Build Settings as follows:

Build and deploy the application to HoloLens as described on Building your application to your HoloLens 2

In Unity, open SampleScene and select ImageView. Its components HoloLens Camera Stream To Mat Helper and Hololens Laser Pointer Detection implement the detection of a laser pointer in the camera image.

HololensLaserPointerDetection.cs detects a red laser pointer using OpenCV for Unity, determines its 3D coordinates, and displays the distance in a tooltip.

Note: To aid debugging, camera images are retrieved from the PC's webcam instead of HoloLens, when executed in the Unity editor.

The original version of this component is available on HoloLens With OpenCVForUnity Example

Adjust the options of HoloLens Camera Stream To Mat Helper to balance accurracy and performance.

See Locatable camera for available resolutions and frame rates.

Depending on the environment and type of laser pointer, it may be necessary to adjust the color range for the detected laser light in HololensLaserPointerDetection.cs method FindLaserPointer

            // Acquire a mask image of reddish pixels using the inRange method. 
            // Red is separated in two areas in the HSV color space
            Scalar s_min = new Scalar(0, 30, 220);
            Scalar s_max = new Scalar(10, 240, 255);
            Core.inRange(hsvMat, s_min, s_max, maskMat1);
            s_min = new Scalar(170, 30, 220);
            s_max = new Scalar(180, 240, 255);
            Core.inRange(hsvMat, s_min, s_max, maskMat2);

See the class and sequence diagrams in the Document folder for information on the static structure and dynamic behavior of the code.

This is work in progress. Known issues are listed in Issues of this repository. Please help to improve this work by reporting bugs and areas of improvement. Ideas for laser pointer detection are particulary welcome!

Currently, detection is limited to a single red laser pointer. Red and shiny objects other than laser pointers are likely to be detected as well (false positives).

• HoloLens で OpenCV をどう使うか レーザーポインター 認識で試してみた
  Lightning Talk held at Tokyo HoloLens ミートアップ vol.21, 2020-07-29 (in Japanese)
• https://docs.microsoft.com/en-us/windows/mixed-reality/locatable-camera
• https://docs.microsoft.com/en-us/uwp/api/windows.media.devices.core.cameraintrinsics.unprojectatunitdepth

This code is provided by the AR Team of ABIST AI Solution under Apache License Version 2.0. It is based on the following source code:

• VulcanTechnologies HoloLensCameraStream for Unity (Apache License Version 2.0)

  • Source files from HoloLensCameraStream for Unity are included to Assets/Script/HoloLensCameraStream/ with the the following modifications:

    1. Code referring to Windows Runtime API is wrapped by
       #if ENABLE_WINMD_SUPPORT ... #endif
       to avoid compile errors in the Unity editor.
    2. Added method GetCameraIntrinsics returning the camera intrinsics object of a video media frame.

    file name	modification a)	modification b)
    CameraParameters.cs
    CapturePixelFormat.cs
    LocatableCameraUtils.cs
    Resolution.cs
    ResultType.cs
    VideoCapture.cs	○
    VideoCaptureResult.cs
    VideoCaptureSample.cs	○	○

• EnoxSoftware HoloLens With OpenCVForUnity Example

  • Source file HololensCameraStreamToMatHelper.cs from HoloLens With OpenCVForUnity Example is included to Assets/Script/HoloLensWithOpenCVForUnityExample/ with the following modifications:
    • Retrieve CameraIntrinsics from VidoCaptureSample object and pass it to FameMatAcquiredCallback
  • The structure of HololensLaserPointerDetection.cs is inspired by HoloLensComicFilterExample.cs from HoloLens With OpenCVForUnity Example

• Microsoft MixedRealityToolkit-Unity (MIT License)