I am trying to deploy and test the updated SceneUnderstanding-Samples release on HoloLens 2, OS 20348.1438. The earlier (prior to August 10 2021) example code ran on the HoloLens 2 just fine, but now I am experiencing an application crash on launch. The app gets as far as showing the generic scene object preview that displays during load, but then shuts down.

I should mention that I am not creating an app package as described in the README, but am building and deploying directly to the HoloLens 2 (with Developer Mode turned on) from Visual Studio 2019. Not sure whether that should make any difference. Otherwise I believe I'm following all the instructions correctly.

Attached is a screenshot of the error being generated at the point when the app crashes, which I got by running in debug mode in Visual Studio. If I select Continue from the Visual Studio debug menu, the application starts and seems to run correctly.

I'm fairly new to debugging with the HoloLens, so if there is additional output or logging I need to provide, let me know.

Hi,

Is it possible to run this project on the XR SDK pipeline instead of the Legacy XR pipeline?

I tried referring to the following, but I couldn't.
https://docs.microsoft.com/ja-jp/windows/mixed-reality/mrtk-unity/features/spatial-awareness/scene-understanding

If possible, please tell me how to implement it.

regards.

Currently the Start Display function in the Scene Understanding Manager component is not awaitable, the only way to do things after the scene has finished loading is to subscribe to the 'OnLoadFinished' callback.

Confusion has risen with regards to SU.

How to use it?, how to integrate it into your project?, what is SU's relationship with MRTK.

I'm in the process of writing documentation to address all of these questions.

Sit tight!

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

I'm working on a project built using the following tools:
OS: Windows 10
Unity version: 2020.3.2f1
VIsual Studio Community version: 2019[16.10.4]

My work is for an existing project that I'm contributing some Scene Understanding capabilities to, and it is not possible to downgrade to an earlier version of Unity.

I'm relatively new to Unity, MRTK and HoloLens development, and was hoping to use the sample code in this repo as a springboard. Specifically, I've added SceneUnderstandingManager.cs to the project and want to modify it to periodically save Scene Object OBJs to storage on the HoloLens; the OBJs have to be correctly aligned to the coordinate system being used by the project. In Unity, I get the error:
The type or namespace name 'WorldAnchor' does not exist in the namespace 'UnityEngine.XR.WSA' (are you missing an assembly reference?)
on compiling the project when this script is included. However, if I comment out the reference to WorldAnchor (lines 599-604) it compiles just fine.

These are the questions I have:

1. Is there a timeline for getting these samples updated to work with Unity 2020?
2. If not, is there any guidance on how to replace the lines I commented out?
3. Should I be using DemoSceneUnderstandingController.cs as base code instead? This Scene Understanding observer documentation makes it seem like the MRTK example is more current, but the Scene Understanding SDK Overview brings me here (via the deprecated version).

This is NOT the same as this issue: https://github.com/microsoft/MixedReality-SceneUnderstanding-Samples/issues/28j which has a very similar title.

I am able to build and run my project just fine in the Unity Editor (2021.3.6f1).

However, our project requires the use of the SceneUnderstanding module to run in an 'AppRemoting' x64 build on a desktop.

These are my build settings in Unity (2021.3.6f1)

Build settings in Visual Studio (VS Community 2019 Version 16.11.19)

The Visual Studio build works just fine! No errors, however, when I attempt to run it. I get the following error:

I expected it to be compatible with a desktop build if it could build in Unity and run just fine?

Here's what I have done to attempt to fix it

copied the file '\myProject\Library\PackageCache\com.microsoft.mixedreality.sceneunderstanding@7bfd3a8ea18c\Editor\Plugins\x64\Microsoft.MixedReality.SceneUnderstanding.dll'

To:
\myProject\build\build\bin\x64\Release\AppX\

When I do this, the SceneUnderstanding.dll is found, but then I get the following error:

I will also add that the documentation says its compatible with UWP build: https://learn.microsoft.com/en-us/windows/mixed-reality/develop/unity/scene-understanding-sdk#where-do-i-get-the-sdk

Any help is greatly appreciated :D Is there a way around this?

Many of the Settings in the Scene Understanding Manager Component are global for all the Scene Objects. We need to enable each setting individually for each scene object, allowing users to customize settings for each specific Scene Object

It is for different days that i try to run the application on Hololens but simply it does not work.
I work in Unity 2019.4.11.f1 with MRTK 2.5.4.
I use Built-in XR with the box Virtual Reality Supported checked.
In Virtual Reality SDKs i have Winodows Mixed Reality correctly configured.
I Have configured the sample scene with Mixed Reality Toolkit and imported the sample scene Nav Mesh, Understanding and Placement as Described in the page.
When i build the application to deploy it oh Hololens i check the box Query Data From Device.
The application is correctly built and deployed on the Hololens but when i launch it I see the first menu where it is asked to load a scene.
I select the desired scene with voice commands but the the app start to glitch, the menu become head locked and simply it does anything. I tried with also the new repository with unity 2020.4 and MRTK 2.7.3. Also there nothing. The app correctly display the second menu with the voice commands. It takes the vocie command inputs because i'm able to toggle the Menu but nothing else.
Please Help.

Unity version: 2020.3.2f1
Visual Studio version: 16.10.4
OS: Windows 10

I'm trying to modify the SceneUnderstandingManager code so that in addition to updating the scene geometry, some other processing of the resulting meshes occurs. Specifically, I want to generate OBJ files for particular scene object classes and automatically save them on some user-specified interval.

I've gotten this working on the HoloLens by modifying RetrieveDataContinuously to make a call to SaveObjsToDiskAsync (with the addition of a Thread.Sleep command to add a delay to the retrieval and storage cycle). However, when I test the changes using the Unity player and a serialized Scene on PC, any commands that require accessing the serialized scene contents (which are of type TextAsset and set in the public SUSerializedScenePaths variable) seem to silently fail, and no scene data is ever returned. Using the key command L, which I guess is running on the main Unity thread, saves the OBJs just fine. Moving the call to SaveObjsToDiskAsync from the separate thread to the Unity Update function also succeeds.

Is it expected behavior that referencing SUSerializedScenePaths files from a thread other than the main Unity thread fails? Is there any way to access them outside the main thread? I'm not actually displaying any of the Scene Understanding data, just exporting the geometry for other purposes. There is other data being displayed by Update, though, so I'd rather not add to the load on the main thread if I can avoid it.

I'm trying to construct a SceneBounds to pass to SceneObserver.ComputeWithBoundsAsync. I'm not sure how to create a Microsoft.MixedReality.SceneUnderstanding.SpatialGraph.SpatialOrientedBox to create the SceneBounds. The struct in the metadata appears to have no public fields. I can try to treat the Microsoft.MixedReality.SceneUnderstanding.SpatialGraph.SpatialOrientedBox as a Windows.Perception.Spatial.SpatialOrientedBox , but the compiler complains that I cannot access the fields due to their protection level. Any help here would be appreciated.

Thanks!

I am getting a DLLNotFoundException when pressing play in the project in Unity.

Am i forgetting something in the explanation in the README?

Scene Understanding runs in the editor but then i check Query Scene from device and deploy to Emulator or Hololens2
and get the following exceptions.

Exception thrown at 0x00007FF9DD52FDCC (KernelBase.dll) in _RobotUnderstanding.exe: WinRT originate error - 0x8000FFFF : 'You must wait for this frame to be ready via HolographicSpace's WaitForNextFrameReady API before presenting.'.
analog\input\mirage\publicapi\holographicframe.cpp(664)\Windows.Mirage.dll!00007FF9D4C23294: (caller: 00007FF9C26D3F5E) ReturnHr(16) tid(918) 8000FFFF Catastrophic failure
Exception thrown at 0x00007FF9DD52FDCC in _RobotUnderstanding.exe: Microsoft C++ exception: winrt::hresult_error at memory location 0x000000D5C9B00B00

OpenXR reprojection settings are not supported; missing OpenXR extension XR_MSFT_composition_layer_reprojection

This repository is currently missing a LICENSE file.

A license helps users understand how to use your project in a compliant manner. You can find the standard MIT license Microsoft uses at: https://github.com/microsoft/repo-templates/blob/main/shared/LICENSE.

If you would like to learn more about open source licenses, please visit the document at https://aka.ms/license (Microsoft-internal guidance).

This is a known issue, for now if you want your scene to be aligned properly in the Unity Editor you need to request Quad data when creating the scene.

Good morning, I tried to download the code but it gives me this error.

Assets\SceneUnderstanding\Core\Understanding\Scripts\SceneUnderstandingManager.cs(1360,167): error CS0234: The type or namespace name 'WindowsMREnvironment' does not exist in the namespace 'UnityEngine.XR.WindowsMR' (are you missing an assembly reference?)

How can I solve it?

I get error that [Assets\SceneUnderstanding\Core\Understanding\Scripts\SceneUnderstandingManager.cs(683,58): error CS0104: 'Vector3' is an ambiguous reference between 'System.Numerics.Vector3' and 'UnityEngine.Vector3'] when setup this sample.

Hi,

First of all thank you very much for providing this sample project!

Recently I tried to build and run this project for HoloLens 2, using Unity 2019.4.25f, Visual Studio 2019 and Windows SDK 10.0.18362.0. When 10.0.18362.0 was selected in the "Minimum Platform Version" in Unity Build>UWP settings, I was able to build the project but when I tested it, there was a visualization problem with the overlays, and the segmentation layers were only visible through several sparse points. When I switched to 10.0.10240.0 in Minimum Platform Version, the problem was gone and I was able to obtain the visualizations similar to the ones shown in the demo video. I am taking this note about my solution in case some other people also encounter a similar problem.

Best,
Ayca

Hello,
I'm quite new to MRTK Hololens. I was navigated to use Scene Understanding to get higher mesh resolution (microsoft/MixedRealityToolkit-Unity#10354 (comment))
So, what is the highest resolution could I get from SceneUnderstanding? Could it reach 1cm resolution? Could I use Hololens 2 to get 3D Object by scanning an object e.g a small machine, machine part in multiple times (to get a dense point cloud, which is supposed to provide more detail for mesh generation)?

The .obj file saved successfully after room scanning in WindowsStorage localfolder. I need to display saved obj file in runtime. But I could not access the hololens localfolder. So I tried another way, I uploaded the file to server. But my problem is I could not load obj file in app. Please give me a solution.

I dont understand what is this for and how it works..

do i need to study video game math? geometry?

im new at unity development.
are they doing this because right handed coordinate system is used inside the circuits to render..
but unity the software that we use to build apps is a left handed coordinate system?
is that the purpose of this code? and why do the skip matrix.M33?
e.g. they didnt do matrix.M33 = -matrix.M33

`        /// <summary>
        /// Converts a right handed tranformation matrix into a left handed one
        /// </summary>
        /// <param name="matrix"> Matrix to convert </param>
        private System.Numerics.Matrix4x4 ConvertRightHandedMatrix4x4ToLeftHanded(System.Numerics.Matrix4x4 matrix)
        {
            matrix.M13 = -matrix.M13;
            matrix.M23 = -matrix.M23;
            matrix.M43 = -matrix.M43;

            matrix.M31 = -matrix.M31;
            matrix.M32 = -matrix.M32;
            matrix.M34 = -matrix.M34;

            return matrix;
        }
`

I took a lot of time opening the start menu until I found an old version sample demo on Youtube, then I know I can control the menu by speech command!!

I am using a time series of scene object OBJs saved to the HoloLens 2 internal storage by the Scene Understanding example code for dynamically building maps of interior spaces. However, while the objects are aligned to one another when they are saved together, groups of objects from different calls to SaveObjsToDiskAsync have different orientations. Presumably this is due to the relative orientation of the headset; if the headset doesn't move, objects remain aligned. See screenshot below for a 3D visualization of two sets of (Wall, Floor, CompletelyInferred) objects created about 2 minutes apart in the same space.

Is there a reason that having the saved OBJ orientation in 3D space aligned to the HoloLens rather than a fixed coordinate system is desirable? If this behavior is intended, is there any guidance on how to maintain object alignment when the HoloLens is moving? I am not displaying these objects in a Unity scene of any kind, but am automatically offloading the OBJs generated by SaveObjsToDiskAsync for processing and visual rendering on an external compute node. I would like to display a map that grows as the HoloLens wearer moves around but maintains a fixed position and orientation.

I cloned the repo and followed the documentation steps:
https://learn.microsoft.com/en-us/samples/microsoft/mixedreality-sceneunderstanding-samples/microsoftmixedrealitysceneunderstandingsamples---unitysample/

I'm able to build and sideload the app package without issue or error, but it displays an odd image of a room before crashing

I took a video and put it on my YouTube:
https://youtu.be/ZKlBIe1FhJU

Any help is appreciated! I'd love to get this working in my app

Hi.

I'm having problems with the build of this project.
I'm not familiar with Unity, so I might be asking a rudimentary question.

Build Environment

• Unity2019.4.19f1
• Visual Studio Community 2019 16.8.5
• MixedReality-SceneUnderstanding-Samples main branch

Problem
Multiple errors occur in SceneUnderstandingManager.cs
Line 571.
System.Numerics.Matrix4x4 converted4x4LocationMatrix = ConvertRightHandedMatrix4x4ToLeftHanded(suObject.GetLocationAsMatrix());
error CS7069: Reference to type 'Matrix4x4' claims it is defined in 'System.Numerics', but it could not be found

Similar errors have occurred in 11 places.

How can I prevent the error from occurring?

Hi,

I want to add the occlusion of spatial awareness to this scene understanding sample in order to get a natural experience, but when I imported the MRTK toolkit, enable spatial awareness then add OpenXR spatial mesh observer, I did see occlusion on Hololens2 in the real environment, but all the scene understanding game objects(the quads represent the scene objects) are gone, and no matter how I debug, I can no longer let the results(the quads represent the scene objects) of the scene understanding display in the real scene, this really troubled me. If you know any solutions please tell me.