aras-p / unitygaussiansplatting Goto Github PK

Hey! Curious if you have tried this on Unity iOS/Android yet and what benchmarks are like?

Could you please enlighten me what i am doing wrong? "gaussianSplatRenderSystem" does not exist in the current context.

A key insight is that rendering a gaussian and then stretching it is essentially the same as stretching and then rendering.

This means you can prerender textures of normalized gaussians with a reasonable selection of possible falloff values and for each splat assign the texture it uses (since falloff per splat doesn't change each frame).

That means each frame is just calculating the directional harmonics per splat, running the bounds calculations for the facing rectangle, and blending the colors with mask textures; which should be extremely fast.

is there any way to select individual splats or segments of point cloud to apply transformations?

if not, is anyone aware of a workaround for this?

Great work, but want some suggestions about code reading

I have tried your work with unity and run it successfully, I am familiar with NeRF, 3D-Gaussian-Splatting and coding with UE4, but never used Unity before, any suggestions on how to get a quick start on your code?

I saw the hashed alpha branch. Are you aware of the MSAA / SV_Coverage trick to use with this?

In case you're not - see below. If you're slapping your head and saying "of course I do - who does this person think they are!" then stop reading now. ;-)

I think this repo gives an example implementation: https://github.com/johnpars/StochasticTransparency but I haven't tested it.

It does the sampling at the sub-pixel giving a much cleaner result. I played with it for Open Brush but it doesn't play nice with Surface Shaders so would have been a ton of work. In your case I don't think there's an issue. Quill used it to get fast OIT that ran on mobile devices: (see p149 https://ubm-twvideo01.s3.amazonaws.com/o1/vault/gdc2017/Presentations/Horne_Oculus%20Story%20Studio.pdf )

Also see https://research.nvidia.com/publication/2011-08_stochastic-transparency

Great work !
With latest commit 19794a3 - Simplify validation script,
using paper's splats samples like Room, bicycle or playroom, the creator outputs an error:

"PLY vertex size mismatch" on the window and nothing is created.
Using U2020.3.7f1, just clone project, open project, import...

Getting this error in Unity 2022.3.9:
NullReferenceException: Object reference not set to an instance of an object
GaussianSplatRenderer.OnPreCullCamera (UnityEngine.Camera cam) (at Assets/GaussianSplatting/Scripts/GaussianSplatRenderer.cs:345)
UnityEngine.Camera.FireOnPreCull (UnityEngine.Camera cam) (at <10871f9e312b442cb78b9b97db88fdcb>:0)
UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr, Boolean&)

I'm using a pointcloud.ply from NerfStudio with the transforms.json renamed to the expected "cameras.json"

Not sure where I went wrong, but any guidance would be much appreciated!

Currently, the com.unity.collections package isn't actually configured as a dependency of the package.json file.
That means that it won't automatically add it when you install it, the user would have to do so manually.

I've been experimenting with VR (thanks to the fix generously provided by @skaale here: https://x.com/DSkaale/status/1712462036146553120?s=20 )

and it works brilliantly.

I tried using the "sort every n frames" option to improve the framerate but there's enough micro-movements of one's head that the effect is a little distracting. But it got me thinking. Sorting is only neccesary when the view has transformed a certain amount. Maybe the logic for sorting could be based around a certain threshold (maybe a separate threshold for orientation and translation?)

Would this also benefit non-VR use cases?

When the "Create GaussianSplatAsset" is click, I get stuck in this screen and had to close unity with task manager.

It worked the first time then it wont work again.

I have searched and there are similar situation people ran into, but every solution seems to be different and never explained how they found the solution.

Hi! appreciate for your hard work in this project.
I'm try to set different setting to train my own gaussian model such as lower sh， so when get your project I try to change the data struct of the Input data struct:
( point out : I set the sh degree as 1, and cut the normal, because the vertex of my point. my ply just have 104, If add normal it requires 116, so I guess that I need to delete this)( of caurse I changed the error codes in the script)

 public struct InputSplatData  
    {  
        public Vector3 pos;  
        //public Vector3 nor;  
        public Vector3 dc0;  
        public Vector3 sh1, sh2, sh3, sh4;  
        public float opacity;  
        public Vector3 scale;  
        public Quaternion rot;  
    } 

but I didn't get the right result, what ever the point clouds or renderer result
The point cloud in expect :

The point cloud that i got:

The render result:

could you give me some advise? waiting for you reply! : )

Hello,

The FBX export of the gaussian splatting unity project I produced is barely a few kb and failed to open on Autodesk Inventor. I guess the unity FBX export isn't finding any scene to export in Unity. (2022.3.10f1)

Any idea on how to Unity could export a gaussian splatting scene into any 3d CAD file?
Or what must be changed to the unity object to ensure exporting?

I followed the user manual:
https://docs.unity3d.com/Packages/[email protected]/manual/exporting.html

It's great to have this running in URP/HDRP but the new method of version 4 for creating the assets, through a ply importer, impedes other useful packages, as PCX for importing pointclouds, from working simultaneously in the scene, which wasn't the case until version 3. Would it be possible to incorporate the GaussianSplatURPFeature or GaussianSplatHDRPPass in version 3 that didn't have this conflict? Thank you!

As of now, this error occurs, when having both packages installed:

i get this error in unity: Multiple scripted importers are targeting the extension 'ply' and have all been rejected: Pcx.PlyImporter (assembly: C:\Unity_New\UnityGaussianSplatting-main\Library\ScriptAssemblies\Pcx.Editor.dll), GaussianPlyImporter (assembly: C:\Unity_New\UnityGaussianSplatting-main\Library\ScriptAssemblies\GaussianSplatting.dll)
UnityEditor.AssetImporters.ScriptedImporter:RegisterScriptedImporters ()

Would the shaders need to be rewritten for URP or HDRP? How could this be achieved?

Is it possible to measure some width or length in the result from Gaussian? Thank you in advance!

Hi
Not a critique, just from a technical technical point of view: what is the difference between this repo and this webgl implementation? It seems that the webgl version runs quite smoothly while this Unity repo is maybe less performant?
https://github.com/antimatter15/splat

In this repo's readme it mentions 31 fps for the bicycle example on a M chip. The webgl locomotive scene runs 60 fps in safari (which is normally poor) on my M2 which made me think that the webgl version is maybe more performant..

Currently the Unity.Mathematics library isn't actually configured as a dependency of the package.json file.
That means that it won't automatically add it when you install it, the user would have to do so manually.
Easy fix though.

Thank you for open sourcing such a great project, I tried it to see how it works and was amazed by the results.
I noticed that some 3d assets in unity can be placed to a 3d gaussians scene taking into account their depth as well as occlusion and I'd like to know how this is done？

How did you unify these two pipelines? Does this mean that 3d gaussians can be used as static game maps in unity? Looking forward to your reply

My local project version is Untiy 2020 + URP10.10, is it possible/easy to port?

Hi! Would it be possible to render this with depth of field or render out a pass with z-depth to exr? Like if the splats know where they are in z the could just be colored lighter or darker depending on where the they sit in z depth? This could be useful for doing compositing in Nuke.

There is z depth fight between multiple GS, dependably on the camera angle, one is totally in the front of the other, but if you move the camera to the side, the other GS moves to the front, with no Blend on the Z axis. Is this an issue of the how GS is made?

At the same time, if you place an object in the scene it Z blends perfectly, and even the GS in scene blends better.

Hi, Thanks a lot for the awesome and interesting project again.
As Unity Viewer currently only supports Windows and MacOS platform. At first, I choose MacOS platform to build Android apk in the Unity, but some issue happened, like below:

Through checked this thread #26 , Found that this issue just happened on MacOS platform. So looked for Windows platfrom to rebuild again. It happened below issue :

"Shader error in 'Gaussian Splatting/Debug/Render Boxes': no matching function for call to 'asfloat' at /zhenglin_workspace/UnityGaussianSplatting/package/Shaders/GaussianSplatting.hlsl(527) (on vulkan)"

Then I commented out below code snippet :

Comment Out from line 527 to line 541.
Compilation Successlly on Windows (I choose "high quality" presets to load model at the beginning and Vulkan as backend)

But when run the apk on the Mobile device, the screen is black and crashed immediately.
By the way , I tried multiple devices like HuaweiMate20 pro and Oppo Find x5. it has the same issue.

would like to ask if anyone run it successfully on Android device or any clues is appreciated.. thank you.

Logcat output is below:

PowerMS : release:L=13877180,F=0x0,T="WindowManager",N=android",WS=WorkSource{10190},U=1000,P=1262
10-30 14:11:07.958 15313 15382 E CRASH : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
10-30 14:11:07.958 15313 15382 E CRASH : Version '2022.3.12f1 (4fe6e059c7ef)', Build type 'Release', Scripting Backend 'il2cpp', CPU 'arm64-v8a'
10-30 14:11:07.958 15313 15382 E CRASH : Build fingerprint: 'HUAWEI/LYA-L09/HWLYA:9/HUAWEILYA-L09/346C432:user/release-keys'
10-30 14:11:07.958 15313 15382 E CRASH : Revision: '0'
10-30 14:11:07.958 15313 15382 E CRASH : ABI: 'arm64'
10-30 14:11:07.958 15313 15382 E CRASH : Timestamp: 2023-10-30 14:11:07.958401493+0800
10-30 14:11:07.958 15313 15382 E CRASH : pid: 15313, tid: 15382, name: Thread-7 >>> com.DefaultCompany.UnityGaussianSplatting <<<
10-30 14:11:07.958 15313 15382 E CRASH : uid: 10190
10-30 14:11:07.958 15313 15382 E CRASH : signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr --------
10-30 14:11:07.958 15313 15382 E CRASH : Cause: null pointer dereference
10-30 14:11:07.958 15313 15382 E CRASH : x0 0000007be0ccc220 x1 0000007be0fc4600 x2 0000007c13bd8020 x3 0000000000010000
10-30 14:11:07.958 15313 15382 E CRASH : x4 0000007b606e4e38 x5 0000007b5f8107a0 x6 00000000001c4774 x7 0000000000000000
10-30 14:11:07.958 15313 15382 E CRASH : x8 0000000000000001 x9 0000000000000000 x10 0000000000000080 x11 0000007be0ccc230
10-30 14:11:07.958 15313 15382 E CRASH : x12 0000000000000001 x13 0000000000000000 x14 0000007c1382ee10 x15 0000007c13bd43b8
10-30 14:11:07.958 15313 15382 E CRASH : x16 0000007c190723b8 x17 0000007cbeea1df0 x18 0000000000000001 x19 0000000000000001
10-30 14:11:07.958 15313 15382 E CRASH : x20 0000007c13bd8180 x21 0000007be0cd4938 x22 0000007c13bd8140 x23 0000007be0cc55c0
10-30 14:11:07.958 15313 15382 E CRASH : x24 0000007be0ccc220 x25 0000007be0cd48d0 x26 0000007c13bd8020 x27 0000000000010000
10-30 14:11:07.958 15313 15382 E CRASH : x28 0000000000000000 x29 0000000000000000
10-30 14:11:07.958 15313 15382 E CRASH : lr 0000007c18e118d4 sp 0000007c13bd8000 pc 0000007c18dc8f18 pst 0000000080000000
10-30 14:11:07.958 15313 15382 E CRASH :
10-30 14:11:07.958 15313 15382 E CRASH : backtrace:
10-30 14:11:07.958 15313 15382 E CRASH : #00 pc 0000000000601f18 /data/app/com.DefaultCompany.UnityGaussianSplatting-tkfGe42b1l-hAft9QYVmuA==/lib/arm64/libunity.so (BuildId: ad9c4a3136038478)
10-30 14:11:07.958 15313 15382 E CRASH : #1 pc 000000000064a8d0 /data/app/com.DefaultCompany.UnityGaussianSplatting-tkfGe42b1l-hAft9QYVmuA==/lib/arm64/libunity.so (BuildId: ad9c4a3136038478)
10-30 14:11:07.958 15313 15382 E CRASH : #2 pc 000000000071c9b8 /data/app/com.DefaultCompany.UnityGaussianSplatting-tkfGe42b1l-hAft9QYVmuA==/lib/arm64/libunity.so (BuildId: ad9c4a3136038478)
10-30 14:11:07.958 15313 15382 E CRASH : #3 pc 000000000071538c /data/app/com.DefaultCompany.UnityGaussianSplatting-tkfGe42b1l-hAft9QYVmuA==/lib/arm64/libunity.so (BuildId: ad9c4a3136038478)
10-30 14:11:07.958 15313 15382 E CRASH : #4 pc 00000000007150d4 /data/app/com.DefaultCompany.UnityGaussianSplatting-tkfGe42b1l-hAft9QYVmuA==/lib/arm64/libunity.so (BuildId: ad9c4a3136038478)
10-30 14:11:07.958 15313 15382 E CRASH : #5 pc 000000000036a85c /data/app/com.DefaultCompany.UnityGaussianSplatting-tkfGe42b1l-hAft9QYVmuA==/lib/arm64/libunity.so (BuildId: ad9c4a3136038478)
10-30 14:11:07.958 15313 15382 E CRASH : #6 pc 0000000000083588 /system/lib64/libc.so (__pthread_start(void*)+36) (BuildId: 3326eee918b5ef1dbb5d558ca672e8e5)
10-30 14:11:07.958 15313 15382 E CRASH : #7 pc 00000000000241dc /system/lib64/libc.so (__start_thread+68) (BuildId: 3326eee918b5ef1dbb5d558ca672e8e5)
10-30 14:11:08.113 15313 15382 E CRASH : Forwarding signal 11
--------- beginning of crash
10-30 14:11:08.114 15313 15382 F libc : Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x84 in tid 15382 (Thread-7), pid 15313 (ussianSplatting)
10-30 14:11:08.179 15457 15457 I crash_dump64: obtaining output fd from tombstoned, type: kDebuggerdTombstone
10-30 14:11:08.179 824 824 I /system/bin/tombstoned: received crash request for pid 15382
10-30 14:11:08.180 15457 15457 I crash_dump64: performing dump of process 15313 (target tid = 15382)
10-30 14:11:08.185 15457 15457 F DEBUG : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
10-30 14:11:08.185 15457 15457 F DEBUG : Build fingerprint: 'HUAWEI/LYA-L09/HWLYA:9/HUAWEILYA-L09/346C432:user/release-keys'
10-30 14:11:08.185 15457 15457 F DEBUG : Revision: '0'
10-30 14:11:08.185 15457 15457 F DEBUG : ABI: 'arm64'
10-30 14:11:08.185 15457 15457 F DEBUG : Happend: 'Mon Oct 30 14:11:08 2023
10-30 14:11:08.185 15457 15457 F DEBUG : '
10-30 14:11:08.185 15457 15457 F DEBUG : SYSVMTYPE: Art
10-30 14:11:08.185 15457 15457 F DEBUG : APPVMTYPE: Art
10-30 14:11:08.185 15457 15457 F DEBUG : pid: 15313, tid: 15382, name: Thread-7 >>> com.DefaultCompany.UnityGaussianSplatting <<<
10-30 14:11:08.185 15457 15457 F DEBUG : signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x84
10-30 14:11:08.185 15457 15457 F DEBUG : Cause: null pointer dereference
10-30 14:11:08.185 15457 15457 F DEBUG : x0 0000007be0ccc220 x1 0000007be0fc4600 x2 0000007c13bd8020 x3 0000000000010000
10-30 14:11:08.185 15457 15457 F DEBUG : x4 0000007b606e4e38 x5 0000007b5f8107a0 x6 00000000001c4774 x7 0000000000000000
10-30 14:11:08.185 15457 15457 F DEBUG : x8 0000000000000001 x9 0000000000000000 x10 0000000000000080 x11 0000007be0ccc230
10-30 14:11:08.185 15457 15457 F DEBUG : x12 0000000000000001 x13 0000000000000000 x14 0000007c1382ee10 x15 0000007c13bd43b8
10-30 14:11:08.185 15457 15457 F DEBUG : x16 0000007c190723b8 x17 0000007cbeea1df0 x18 0000000000000001 x19 0000000000000001
10-30 14:11:08.185 15457 15457 F DEBUG : x20 0000007c13bd8180 x21 0000007be0cd4938 x22 0000007c13bd8140 x23 0000007be0cc55c0
10-30 14:11:08.185 15457 15457 F DEBUG : x24 0000007be0ccc220 x25 0000007be0cd48d0 x26 0000007c13bd8020 x27 0000000000010000
10-30 14:11:08.185 15457 15457 F DEBUG : x28 0000000000000000 x29 0000000000000000
10-30 14:11:08.185 15457 15457 F DEBUG : sp 0000007c13bd8000 lr 0000007c18e118d4 pc 0000007c18dc8f18
10-30 14:11:08.185 15457 15457 F DEBUG :
10-30 14:11:08.185 15457 15457 F DEBUG : backtrace:
10-30 14:11:08.185 15457 15457 F DEBUG : #00 pc 0000000000600f18 /data/app/com.DefaultCompany.UnityGaussianSplatting-tkfGe42b1l-hAft9QYVmuA==/lib/arm64/libunity.so (offset 0x1f7000)
10-30 14:11:08.185 15457 15457 F DEBUG : #1 pc 0000000000000100
10-30 14:11:08.451 809 886 D AwareLog: iawared: WorkingsetProcessCommand subCmd = 357, userId=10190, pid=15313, com.DefaultCompany.UnityGaussianSplatting
10-30 14:11:08.452 809 886 D AwareLog: iawared: WorkingsetPauseCollect com.DefaultCompany.UnityGaussianSplatting
10-30 14:11:08.513 9156 9156 E adbd : failed to connect to socket 'localabstract:renderdoc_39920': Connection refused
10-30 14:11:08.589 1262 15460 W ActivityManager: finishTopCrashedActivityLocked Force finishing activity com.DefaultCompany.UnityGaussianSplatting/com.unity3d.player.UnityPlayerActivity

10-30 14:11:08.589 824 824 E /system/bin/tombstoned: Tombstone written to: /data/tombstones/tombstone_00
10-30 14:11:08.590 1262 15460 V ActivityManager: positionChild stackId=0 to top.
10-30 14:11:08.591 823 3427 E iMonitor: FaultDetect: DUMPTOOL_PRINTF return.
10-30 14:11:08.591 1262 1347 I BootReceiver: Copying /data/tombstones/tombstone_00 to DropBox (SYSTEM_TOMBSTONE)
10-30 14:11:08.592 1262 1340 I StatusBarDisable: setFlags what=0 which=1 pkg=Window{c8e300c u0 com.DefaultCompany.UnityGaussianSplatting/com.unity3d.player.UnityPlayerActivity}
10-30 14:11:08.595 1262 1340 W ActivityManager: Skipping native crash dialog of ProcessRecord{fe717dc 15313:com.DefaultCompany.UnityGaussianSplatting/u0a190}
10-30 14:11:08.600 2103 3532 E AwareLog: SPKSchedFeature: Uid has been sent, ignore.
10-30 14:11:08.612 2014 15269 I BoosterSwitchP: notifyUidState do nothing
10-30 14:11:08.612 2014 15269 I DeepNoDisturbP: notifyUidState
10-30 14:11:08.612 2014 15269 I BrowserChrP: notifyUidState

Hi, Thanks for the awesome project. I could run it via OpenXR and Quest Link (tethered to my desktop computer with a 3090), but when I try to run it on Quest standalone and build an apk, I get 50 errors regarding the shader files like this:

error in 'Gaussian Splatting/Debug/Render Boxes': 'asfloat': no matching 1 parameter intrinsic function; Possible intrinsic functions are: asfloat(float|half|int|uint) at Assets/GaussianSplatting/Shaders/GaussianSplatting.hlsl(469) (on gles3)

Has anyone succeeded in running Gaussian Splatting on Quest standalone?
Thanks

Thanks for sharing your great work.
I've been investigating about how to make this run on web browsers.

WebGL doesn't work as there is no compute shaders, so we need WebGPU backend.
Fortunately, Unity 2023 beta supports experimental WebGPU though some scripting for editor is needed to enable it.

For now I'm struggling with radix sort codes from FidelityFX, to make it work on WebGPU.
Here's some issues:

It seems like WGSL does not have a counterpart of GroupMemoryBarrierWithGroupSync().

• It's being converted to just workgroupBarrier() which will slow down the sorting algorithm.
• Also the shader compilation fails due to calling the function on a non-uniform control bound.
• I couldn't succeeded to modify the original HLSL code to make it work.

wavebasic does not work. it produces an error like: invalid kernel index passed.

• Metal backend works without it, so I can just comment it out but it could be much slower

There are working WebGPU implementations for 3D GS, so hopefully we could modify the current code to make it run without errors.

The inspector for MeshRenderers that have materials is extremely laggy (0.5s lagspikes) and evident that something evil is happening.
I can see splats in the material's autogenerated preview and they move around when the inspector refreshes.
I can hear their screams as they cry out from the 32x32px prison that can't and probably shouldn't contain them.
The moment the material is removed or the inspector window is closed, the lag goes away.

I'm building the example scene on windows and it's showing the scene perfectly but when I try that on webgl is not showing anything
Is because Webgl is not supported or am I missing something?
Thanks, amazing tool man!

I have an obscure issue here that I think would just take a few lines changes to get fixed. I would be happy to test it and put in a PR myself. I'm up against the edge of my knowledge of Unity's command buffers though and I'm having trouble finding the commands I need.

I'm not sure if anyone here is familiar with LookingGlass volumetric displays. They are cool 3D displays that take up to 100 horizontal views of a scene and then display them using a lenticular display that can be viewed without glasses. You can get a feel for what they do from their Looking Glass Blocks website where you can view the images on a regular monitor. It is not as cool as the display in person though since the display has so many views the objects have a depth that you can't reproduce with anything else. It's even better than VR since each eye is getting multiple views leading to multiple focal planes within the scene. I highly recommend their portrait display if you are interested.

I thought it would be cool to mess around and get gaussian splats rendering to them. Looking Glass has a nice Unity plugin already that places a volumetric camera in the scene to render to the looking glass display.

This plugin seems like it is very close to working out of the box with the UnityGaussianSplatting project. Initially, it just renders a blank image to the looking glass display with traditional meshes rendering fine. From digging through your code and the Looking Glass code I discovered that if I change the UnityGaussianSplatting code to never call m_CommandBuffer.SetRenderTarget I can get the image intended for the GaussianSplatRenderer.Props.GaussianSplatRT to render to the display. This looks great spatialy, but the colors are washed out since it is skipping the compositing step.

You can see one of my renders from this trial here:
https://blocks.glass/mr_anderson/22601

I believe the issue is that the looking glass plugin is setting up its camera for multiview by changing the render target directly to the output image they are generating. The UnityGaussianSplatting plugin sets the render target back to BuiltinRenderTextureType.CameraTarget though. The relevant code is here in GaussianSplatRenderer.cs.

I have tried some hacks for setting the render targets back to all of the following render textures. None of these seem to be the correct target though. If I change the composite shader to just output a solid color I can only get that to render to the display if I remove all of the SetRenderTarget calls. I just get a black image if I set it to any of the following.

`RenderTexture.active
BuiltinRenderTextureType.CurrentActive
BuiltinRenderTextureType.CameraTarget
cam.activeTexture
cam.targetTexture

`

Because of this testing, I am fairly sure that I just need to find a way to store the current render target for the camera and then set it back to it before the composite step. I was hoping there would be some type of GetRenderTarget command that would allow me to store the current render target before rendering GaussianSplatRT and then move back to that target before compositing. I'm having trouble finding anything like that though. Any ideas would be appreciated.

I have a thread going for this on the Looking Glass discord as well:
https://discord.com/channels/573562186597662720/1164052492368224376

Today the Low / Very Low quality presets use BC7 texture compression format, which does not work on mobile. Investigate one of these:

• Either make both BC7 and ASTC (e.g. ASTC 4x4) color data files. Load one or another, depending on GPU capabilities. Quality investigation of ASTC needs to be done.
• Alternatively, look into UASTC (from Basis Universal). Make one UASTC file, add transcoding code path to whatever GPU supports at load time. Quality investigation needs to be done, as well as figuring out how to best package it within Unity (native code plugin? something else?)

Currently all of data encoding, SH clusterization, quantization code is in C#/Burst within Unity. That's fine for "within Unity" use cases, but e.g. if #45 is done then it would be easier for "web" use case to have all that code usable outside of Unity. Like a native code library, possibly compiled to WebAssembly too.

Possibly better documentation of the format is needed, outside of what already exists in the blog posts (one, two).

And then, provide changes needed for one of open source Web based viewers to support this format, both as transmission size savings and runtime memory savings. Possible candidates: https://github.com/quadjr/aframe-gaussian-splatting, https://github.com/mkkellogg/GaussianSplats3D, https://github.com/antimatter15/splat

Might need #44 be addressed first.

Hello,
The tools menu only has Capture Screenshot and Validate Rendering options
There are no error messages.

Hi, thanks for this excellent code.

I want to know whether this can be used in the Mac's Unity without any GPU？

When going into ortho mode to try out splat selection / splat editing, I noticed splat sizes are not correct there:

First of all, i have to apologize in advance if i'm wasting your time and i have to mention that i'm not a programmer or game designer and i'm just trying this plugin as a hobby. I love these new technologies and want to learn more and practice.

I have already done small projects in Unity and i know for a fact that Unity behaves completely normally even today, it is important to mention this because when i copy the repository and open it in Unity (projects/ GaussianExample), the Unity UI is just going totally crazy and i have absolutely no idea what can be causing this (since again my other Unity projects still work fine and i have imported GitHub repos of Unity projects in the past).

Again sorry if that is just a waste of time, its the first time i report an issue on GitHub and that might be caused by my material but i don't even know (i use a 3070GTX)

Currently, the com.unity.burst package isn't actually configured as a dependency of the package.json file.
That means that it won't automatically add it when you install it, the user would have to do so manually.

Hi, congrats for the effort on the project!
I've been testing it with a VR headset, and I was able to see splats using OpenXR in Multi-Pass mode (Single pass instanced does not work). However, I tried to switch to URP to get some performance benefits but the rendering is reversed, the Left image renders on the right eye and viceversa. Any ideas on how to fix it?

Hey,
When using 2 gaussian splatting in a scene, one is render on top of the other one. Also when the camera is facing +X the first model in the hierichy is shown on top, and when facing -X it's the other one that is showing on top.
Please let me know if I am doing something wrong.

Maxime

Currently splat sorting uses AMD Fidelity FX based radix sorting. I actually have no idea how "good" or state-of-the-art it is, but some other options might be worth looking into:

• Alter the same sorting routine to not sort on full 32 bits of depth -- maybe 16, 20 or 24 bits would be just fine, especially if depth is quantized to be only within object's range. The current sort code is doing 4 bit radix rounds, so that would cut down on the number of rounds it does.
• Look into other sorting approaches. State of the art seems to be OneSweep radix sort, but that one might not work on all platforms (I think due to lack of globallycoherent on Metal?), plus most implementations of it seem to be hardcoded to nvidia SIMD width.

This is a great project that is really interesting to tinker with. Please add a license file for people to understand what are tinkering limitations.

Currently all splats are rendered (and sorted), just quads that are behind camera are thrown into the sea at vertex shader level.

It should be possible to do some sort of splat culling at distance evaluation state. Both "behind camera" and "really outside of the frustum" type. Either at individual splat level, or whole splat chunk (groups of 256 splats) level.

And some sort of GPU stream compaction so that later on fewer splats need to be sorted, and fewer quads need to be drawn. Both sorting and drawing would need to get changed to do indirect dispatch/draw.

From my tests with the current tooling here's two ideas on how to make it even better:

1. a common usecase I see is downloading a .ply, correcting the transformation, cropping to a nice area and then exporting again. Currently, the transformation correction is not "baked" into the exported file – that would be nice.

2. combining and composing splats into a new one – currently this can be done manually by copy-pasting lines in PLY files, but would be nice if there would be a single step of exporting a "root" object and then all SplatRenderers + their transformations + their cutoffs and edits would be applied into a single one.

Point (2) would also be a useful optimization for runtime use: the resulting splat would be sorted properly and the data is smaller than shipping multiple splats. Could theoretically even be a build step if someone wants that! (Work at "High" resolution with any number of splats and at build time select ones are baked back into one splat at "Low" resolution)

Instead of sorting + alpha blending, investigate whether Hashed Alpha Testing would produce "acceptable" results. Or one of order independent transparency approaches like MLAB. If any of these is decent, perhaps add them as an option.

Although the splats look correct in the game mode, they show upside-down in the viewport. Rotating or scaling by -1 the object doesn't seem to work. Any idea to visualize it in the viewport correctly?

This project was working fantastically for me yesterday, but for some reason Unity is completely unusable now, even when reinstalling Unity 2022.3, Unity Hub, and starting with a fresh copy of this repository.

Upon clicking the "Create GaussianSplat" MenuItem, the OnGUI method is never called in GaussianSplatAssetCreator.cs. Then this window shows up and hangs indefinitely (I've let it sit for a couple of hours and nothing changes). Only way to exit the Unity Editor is with a force quit from Windows task manager.

This started after I deleted some Splats in the Editor, exported the new Ply file, and then tried creating a new GaussianSplat. Every time I've clicked on the MenuItem afterwards, the Editor crashes like in the picture no matter what I do.

Does this project only support DX12 and Vulkan?

Cant add unity geometry like primitives , FBX etc.. to the scene when in URP mode it just get really slow! and then the editor crash
cant debug it

Sometimes, especially when deleting some unwanted splat areas, you're left with like a "hole" in a surface. In many cases it feels like the hole could be patched up by "simply" copying some nearby splats into it.

So similar to how you can select + delete individual splats right now, maybe you could also select + duplicate + move/rotate individual splats.

And maybe edit their colors (tint, blur, change opacity)? Whelp this sounds like a rabbit hole!

After some testing, there seem to be some heavy fps drop (down to 10-40fps from 80+) when the camera is close to a "surface". Reducing if I lower splat scale and getting worst if it's increased. Wondering if it's a know issue with Gaussian splatting or a known bug with this implementation. Thanks!