clarte53 / gaussiansplattingvrviewerunity Goto Github PK

Hi,
Thank you for the great work.
May I ask how can I import multiple 3DGS into this viewer?
I want to show the result after doing segmentation on a 3DGS, so it is gonna be multiple 3DGS.

Thanks.

when i switch different modle files, it happens and stops at initialize.

Hey, love the splat viewer. It's definitely the easiest way to view splats in VR.

I have a feature request: I sometimes find myself working with models that need to be scaled to large sizes to be accurate. The splat might start out with the model being the 10's of cm across, but in reality it needs to be 2+ meters to be accurately sized (like a car or building for example.)
However when I scale things up large, it seems like scaling doesn't work linearly and at a certain point trying to increase scale doesn't do much anymore. Also weird clipping occurs with my 'eyes' when trying to get close to largely scaled models. Almost as though my headset is getting in the way of viewing it close enough. Ideal behavior would be to size something large and then be able to interact with it like it was actually that size. Thanks!

When loading a new model,

1. set the floor. That means you move the model till you can see the floor. Then you point at three separate points on the floor. Based on this, the model is aligned in yaw, pitch and roll and the floor set at the user's floor
2. set height. That means, you point at something that should be at eye level, and the model is scaled accordingly.
3. This is saved at best in the ply, so next time you load the ply, it is already aligned.

All alignments should be correctable.

When loading a known model, load the data from the ply to align.

Hi there,

Is there a way to properly use our own .ply file ?

What a awesome project! So excited to find it! It's a pity that I haven't succeeded in implementing this project yet, so I really hope to get your help. Thanks a lot！

I've gotten the.ply file through Polycam and tried to see gaussian splatting on my pico neo3 headset. After connectted pico neo3 headset to Windows 10 PC using a USB cable, I launch the executable GaussianSplattingVRViewer.exe on Windows 10, I see the Clarte startup screen for 3 seconds then the screen kept going black.

More details:

1. the.ply file had been put in the same folder as the windows executable GaussianSplattingVRViewer.exe
   
2. I also tried launching the GaussianSplattingVRViewer.exe without any ply file like:
   
3. I also tried the NoDepthSub version like the detail 1 and the detail 2, then got the same result（he Clarte startup screen for 3 seconds then the screen kept going black.
   ）
4. I didn't set headset's runtime as the default OpenXR runtime，because I can't find where to set

If you need more details, please feel free to contact me. Looking forward to your reply.

Thank you for the update! I would like to try Gaussian Splatting Relighting in your Unity plugin. Do you have any clue or idea how to work this out?

I duplicate the 'Gaussian Splatting' in the scene GaussianSplattingCam and play in editor to try rendering different ply files.
But the editor hints: illegal memory access.
Does this lib support mutiply thread rendering?

Lightgaussian is significantly faster than the original gaussian splatting (and btw generates scenes ten times smaller)
https://lightgaussian.github.io/

Hi,

Thank you for creating and sharing the project! I attempted to implement the plugin into an HDRP project, but the Gaussian splatting does not appear in the display. Does the project not support HDRP, and if that's the case, what can I do to make it work? Would the shaders need to be rewritten to function with HDRP?

Hi,

Thanks for creating the project!

I have been playing around and it seems that I am running against performance issues that I cannot explain, as I am not certain what component is a bottleneck. The issue I am seeing is that I experience 'stuttering'. So it is not necessarily slow in the sense of low fps, but their is clear stutter. I.e. in the biking scene, whenever I am looking to the bike this stutter happens, when looking away from it, it does not. This means that for a few frames everything is smooth (as far as 30fps is smooth) and then the stutter happens.

The machine I am using has 32GB ram, i9 12900k and RTX 3090ti and I was using a link cable to stream the screen to the Quest 3.

I am getting around 30fps in the biking scene when running from Unity, which seems like a fairly low fps given the hardware. Gpu-z is telling the GPU is used between 70-85%, so there seems to be some room left here. Task-manager shows that cpu is not above 50%, however 2 threads seem to hit the 100% quite often. So this gave me the hint that their might be a CPU bounded bottleneck. However, when running the profiler, ~90% is semaphore.waitforsignal, which to my understanding means the GPU is working on something. Also, changing the resolution does not seem to effect the frame rate that much, which to my understanding would also hint a CPU bounded bottleneck.

All in all:

1. Should I expect higher fps given the hardware?
2. If this is a hardware limitation, given the story what would your guess be on which part is the bottleneck?
3. Is anyone else experiencing this stutter?

Thanks!

Resolution ?

Hi Clarte,

Thank you for your excellent work on implementing Gaussian Splatting in VR.
I'm currently trying to launch two models at the same time to perform some comparison tasks. For example, the first model launched is rendering the main scene while the second model is rendering in a small quad. My initial attempts failed because it was not possible to load both models at the same time. Can you give me some hints on how to solve this problem, e.g. which part of the code should be changed?

I am trying to use the Clarte VR Viewer with a Varjo XR 3 headset on Windows 10 with a point cloud I created using the original gaussian splatting research paper code.

Upon executing the application I briefly see the Clarte splash screen then the program exits, presumably via an error. Is this a problem with my headset or the application itself? Is there any way I can view the unity crash logs? If so, I'll post one here ASAP. Sorry for the lack of detail.

Thank you!

Hi there,
Thank you for creating this plugin. I want to get the left and right eye view in the game view in unity. How do I do that? Currently I select the GaussianSplattingVR scene then add the Gaussia Splatting Camera script using add component > add the render and depth material parameters to get the one camera view of the splats.

Cycle through multiple point_cloud.ply in a list.txt

So you have a list.txt in GaussianSplattingVRViewer.exe's folder and in this there is

path\to\point_cloud.ply
path2\to\point_cloud.ply
...

And you can cycle through them using the joystick, fi by holding trigger and using the joystick.

thank your VRViewerUnity , i like it very much !

how test your depth support and transparency to have a fully integrable plugins for rendering in unity ?

FPS

Hi,
Is there a way to run this project without an HMD?
I am using a Win PC and want to make some 3DGS to show in this Unity Viewer.
Can I see the result directly on my PC or an Android phone?

Many thanks

The project is so awesome! I love it! May I konw the detailed configuration of build setting and project setting? I did succeed running the program while there is no images or scenes in my PC and HMD display. I am pretty sure that it has already generated the tex under Tex/Element0, Element1，and the texture can update when I move(rotate or transform) my HMD. However, I am still so lost why these are no images/display in my HMD and PC screen. I guess it might be related with the configurations? Many thanks. Hope to get your reply soon. Thanks a lot.

Thank you for the nice work! This repository gave me a lot of inspirations.

I wanna test object representation using gaussian splatting with this code.

What I wanna ask you is that, is there any way to get alpha channel of rendered results?
In the unity code, it seems that the rendered image from .dll file is sent to textures connected to XR Origin - Camera Offset - Main Camera. But the transparent parts looks black.

Can I modify it to transparent? (for example, showing unity's skybox or boundary scene when some part is empty...)