Client app packet loss over time. about alvr HOT 36 CLOSED

I think I found and resolved the problem. Some more tests are needed, but I played around 1 hour with Arizona Sunshine (Only for scientific reasons of course), and no performace dropping!

I'll explain here for completeness sake:
In a very old commit, the NalQueue class had a "refactor", from a normal queue to a java.util.concurrent.ConcurrentLinkedQueue. The problem here is that queue calls are thread safe only when they are atomic. So I moved back some "synchronized" statements as in the original master branch. This could explain some strange counts on that queue I saw in debugging and profiling. And maybe performance degradation.

Now we need to test it. Please @Susanoo1337 could you test this apk with V6 server app:
https://drive.google.com/open?id=1TEBEWF-_TiKHyv0s4lYa5CQRRBZpOtoT

from alvr.

There where no changes in network code between alpha 5 and the experimental releases.
While the problem you describe is still occurring , its still not reproducable all the time for me.
It happened with alpha 5 as well.

That said, I still suspect a network problem with your wifi if you get this error all the time. There are a few things you could check:

• Some wifi router have problems handling the high throughput of ALVR on 5ghz and get very hot. A firmware could also help.
• If you have the 2.4Ghz Wifi and the 5Ghz wifi on the same SSID, the Quest will switch between them. Use dedicated names for both

The suggestion from John Carmack to maybe sell a dedicated Quest wireless usb adapter for the pc sounds like a good idea

from alvr.

I just tried your alpha5 experimental v1 and (since it was the first one to feature fixed controller offset) and after one hour of play both client and server still had 70~72 frames.
Riftcat running at 50Mbps doesn't have this problem either. Tracking on it is much slower though.

As I mentioned I have a dedicated 5Ghz AP for the Quest, 2.4Ghz radio is completely disabled on it. It's a tplink R450 running on a Qualcomm Atheros QCA9558 chip @720Mhz. It's well ventilated and doesn't seem to get crazy hot. Can't say for sure of course as it doesn't have a chip temp readout.

I'll try my way through v2, 3, 4 and 5 today or tomorrow and see if the issue pops up before experimental v6 so maybe I can narrow it down for you.

I didn't think it's a netcode issue to begin with, it looks more like an issue with the apk build.

from alvr.

So after a day of testing, it turns out experimental v1 is the only one not causing the issue.
v2 I couldn't get running.
v3 dropped frames after a couple of minutes.
v4 app and server versions in the download mismatched, couldn't try.
v5 dropped frames after a couple of minutes.

I continued experimenting with v6, and it turns out the frame drops coincide with packet loss, and the packet loss is caused by sharp or loud noises being streamed. For consistency I've been using h3vr, everything runs fine for 5 to 10 minutes, then tracking starts slowly failing and every gunshot sound causes a couple hundered packets to be lost and fps dip below 50.

Now I've been using my keyboards audio output for VR sound, because my main audio driver is my Astro A50, and that has been causing crackling in VRidge/Riftcat as well as the original ALVR alpha 5.

I changed steam to use my Astro audio again, and it actually worked on experimental v6, however, there was roughly a 2 second delay in audio.
I've tried all other audio adapters, Nvidia hdmi, optical, mainboard aux for streaming, they all cause either packet loss or a 2 second delay, at random.

from alvr.

When not streaming audio at all, the same issue persists.

from alvr.

To be honest, your description still sounds more like a messed up system.
All experimental versions do work, some without the controller fix after the oculus firmware update. They where downloaded hundred of times. If you switch versions like you did, you have to properly uninstall the old versions on the quest and in steamvr.

Sound streaming does not depend on the volume of the sound at all and if you say that the problems persist even when sound is disabled, that can not be the problem.

What are your system specs? Do you get constant 72fps on the pc? Loud noises in games could mean that there is a lot of action, your framerate could dip and causing lost frames.

from alvr.

I have two more Quests laying around, I'll try it on another one. My system is an i9-9900k, 32gb ddr4 and a gtx 1660ti. I guess that could be the bottleneck but why only on some versions and not others... and yea the server always reports 71~73fps, it's just the client fps that are going down.

I've actually tried it in a really calm and quiet scene, I held a gun behind me, where it wasn't being rendered, and when I shot it the packets just dropped.

Also I was really careful to first uninstall the apk from the quest before installing a new one, same with the ALVR driver from SteamVR.

I also have another VR capable PC that I can try.

I realize how crazy this sounds, but I've spent over an hour playing on experimental v1 yesterday and only had maybe 100 lost packets, as opposed to the 10k I would get within minutes on the other versions. Same games, same system, same settings (except for 150% resolution).

On that note, is there any way I can change v1 to stream the correct resolution for the quest? It's the only version that works for me and when using 100% it does look a bit wonky.

from alvr.

I have the exact same problem on all JackD83 versions. I tested it in all and every configuration. Having a perfect WiFi, with a dedicated high performance AP for ALVR, pc Ethernet cabled to it, and ac only 40mhz 5ghz WiFi, just mitigated and postponed the problem. After about 20-30 minutes the performances starts deteriorating.

It’s almost 3 weeks of tests in search of this bug for me now (I’m a VR dev and actually use ALVR to test my unity envoronments, so hours of VR). I enabled all logs on server. No particular error.
Also, I tested the compiled apk using Android Studio Profiler. I am not an Android dev, but from what I could see there is no EVIDENT memory leak. However there are LOTS of allocations with a very steep curve. CG obviously is correctly destroying objects and used memory stays around 265Mb.

I’m deeply studying the app code. I think that especially in the queue system for the frame buffer the performance could be optimized. But I do this analysis in my spare time, so...

from alvr.

@FingrMastr Fixing this bug is pain in the a**. I rewrote many parts of the software in different ways and still ended up haven the same issue. I currently suspect the server side socket polling to be the culprit. I'm still not sure why it was implemented this way. There could be a buildup of old packages on the socket that are leading to the problem if you play longer. The mic streaming did not help either, cause it adds just more packages that should be processed right away.

@Susanoo1337 Your PC is more that capable to play most games without problems. I still think there could be a problem. The only big change starting with v3 is the increase in render resolution that the client has to render. This can put a lot of stress on the GPU. Can you try this: Use ev6 (or upcoming ev7)
and set video to 100% and in steamvr, set super sampling to manual and 50%. Try a few rounds. It will look ugly :)

from alvr.

I really appreciate you taking the time to work on this project at all. Just wanted to say that.

I've tried 100% with 50% SS in steamvr. Same problem occured after a while, but it might have been a bit longer. I used a completely fresh out-of-the-box Oculus Quest with the only app installed being ALVR experimental v6. The next days I'll try a different PC, make sure it's not that.

Next I've tried Virtual Desktop, unfortunately the quality of the tracking and streaming on that is a bit arse, at first it streamed 78Mbps but after a restart it wouldn't go over 30ish. There is sadly no way to actually set the streaming rate in the app, plus it requires a constant internet connection which is a hassle with my setup.
However the same problem never cropped up, even after over an hour of playing.

Is there a way for me to edit(hex or otherwise) experimental v1 to use the correct Quest resolution meanwhile? That seems to be my only option for now.

from alvr.

It seems strange to me that V1 is working without hassle for you, since I tested it (as all the other versions), and for me the performance degradations is still there even with v1. However, in that version JackD83 did not resolve yet the resolution problem. On that version, 1216x1344 per eye is the default 100% resolution used by the Quest. You can change it by entering
adb shell setprop debug.oculus.textureWidth 1440
adb shell setprop debug.oculus.textureHeight 1600
This setting will stay there until you restart the quest.

from alvr.

@JackD83 is the client from V1 equal to that of versions prior to V5? I see in the Client repo that you marked with the “v” notation only by the V5... it would be useful for me to have some changelog for the client app in respect to polygraphene’s alpha5... today I’ll test the “compatibility fixed” alpha5 and your V1 (maybe I missed something)

from alvr.

is the client from V1 equal to that of versions prior to V5

Unfortunately no, I forgot to tag the client repo.

from alvr.

I just made a debug version of ALVR with an async communication option in the "other" tab. I don't have the time to test it properly so here is a direct link: ALVR_debug

Its compatible with the ev7 client on the HMD. I'm on vacation the next weeks, so no updates from me.

Edit: never mind, has the same error :(

from alvr.

Ok I'm a little... irritated at this point. Not by my first point, but by the second.

1. @FingrMastr I've tried your version. The issue still cropped up within roughly the same timeframe.

2. I've tried both FingrMastr's fixed v6 and JackD83's v6 while connected to a different PC and there were no issues whatsoever within a timespan in which I was comfortable staying in VR.

This is the irritating part:
The second pc is a Win7, i7-3770k, 16GB DDR3, GTX970ti
As opposed to my Win10, i9-9900k, 32GB DDR4, GTX1660ti

The Win7 machine has also not been wiped in a long time, so it should be nice and ripe with malware and stuff. It's abysmally slow in most situations, takes a few minutes to boot up.

So the problem with all versions above experimental v1 are either an issue with Windows 10, or an issue with my specific network configuration. And I have no idea which. There is one more thing I can try, but otherwise, short of installing Win 7 on my main machine, I'm all out of experiments.

from alvr.

Incredible. Simply incredible.
However, as I said, I’ll test some more my very little fix and also try some other optimizations I have in mind. Anyway, I have only a win10 PC, pretty high-end, so I cannot try with other PCs at the moment.

I’m sort of “hating and loving” this issue because it’s nasty and maybe not caused by a single problem. ALVR is a complex and technologically awesome piece of software and at the moment the ONE AND ONLY wireless solution with sliced fixed foveated encoding, and a FREE one, even before the wired official Oculus Link. With some more work it’ll also become the most polished solution for Quest streaming PC VR. I’m sure about this.

from alvr.

Confirmed, still occurring, even with my little fix, but a little mitigated.

Very nasty bug. Starting to think that’s not strictly related to software optimization...

from alvr.

So after a day of testing, it turns out experimental v1 is the only one not causing the issue.
v2 I couldn't get running.
v3 dropped frames after a couple of minutes.
v4 app and server versions in the download mismatched, couldn't try.
v5 dropped frames after a couple of minutes.

@Susanoo1337 @JackD83
I'm getting the same result on my system: QUEST - WIN10 (i5 8400 + GTX1070 16 GB DDR4) - WIFI AC 5 GHz (@868Mb/s)

• V1 is working like a charm
  => with manual resolution correction via ADB:
  adb shell setprop debug.oculus.textureWidth 1440
  adb shell setprop debug.oculus.textureHeight 1600

• V2 => V7 ultimately (after 15 minutes) starts deteriorating into a lag fest.

It sure would be nice if one could take the V1 APK and add just the resolution correction.
I'm very much willing to test ALVR and provide feedback.

Kind regards,
TC-VR

from alvr.

So I was just scrolling through the issues list and reading them just because, welp, curiosity :P and found this one quite interesting because I don't have it but I also have Win7 which means that could be completely related with Windows. I also have another computer with the exact same hardware but running Windows 10 instead. I didn't test on the Win10 machine yet but on the Win7 one I play for hours and hours and never had this issue. I will try to give it a try on the Win10 one as soon as possible and report back then we could guess that it's indeed related with Win10. But we also should take into account that drivers may be different too, I don't update my nvidia drivers everytime since all works fine. I am still on ALVR v6 btw.

from alvr.

A small update. I made some more optimizations on the client App, basically static thread priority assignments. None of these optimizations however did resolve the performance degradation issues.

Since also in my case in V1 this problem is not appearing, I’m comparing the commits made to client and server just after that release. The only ”major” change could be the NVENC intra-refresh elimination. I’ll try to compile a V7 with this function back in and I’ll do some test.

from alvr.

@FingrMastr Thanks for the work you are putting in to this ... V7 has amazing visuals so I hope this can be patched. V1 just doesn't do it for me and I'd prefer Virtual Desktop over that.

from alvr.

Virtual desktop is very polished but has a feature that will limit performance over time. It uses TCP for streaming. ALVR uses UDP for all its connections. It limits it to locally, but boosts performance over reliability. The ideal would be TCP for tracking and flow control and UDP for streaming. But implementing this is lots of work because we need to change code very deeply.

from alvr.

I did a lot more testing, and will do some more in order to draw some conclusions, but here are some pre-eliminary results:

I've tested every bitrate starting at 50 Mbps in 10* increments up to 120 Mbps. 13Mbps as a "control" since that's the VRidge default setting. From 120 on I tested 150, tomorow 200 and 250 will follow. All of these were done with a 200kb buffer.
In the following days I'll try increasing buffer size.

13Mbps: 16 minutes until FUBAR. However, the problem was not packet loss, but tracking that went tits up.
50Mbps: 15 minutes until massive packet loss begun.
60: 11 minutes
70: 8 minutes / packets lost ~4348
80: 11 minutes
90+: problem did not occur / packets lost after ~30minutes: 1358

for 110 Mbps I fiddled with my NICs settings, I've enabled Jumbo Frames @9kb MTU
packets lost after ~40 minutes: 200
This is due to the packet size being increased, only a fraction of total packets were sent, the quality was much more consistent.
It did however, crash the ALVR App on the Quest in very regular intervals. The Quest itself also crashed once, which suggestes that it does not support Jumbo Frames.
I'll try lowering the MTU and try again soon.

At 150 Mbps I disabled Jumbo Frames again, after 38 minutes only 1210 packets got dropped (out of approx 19522000). That's pretty standard for UDP.
The round-trip latency at 150 Mbps however, is 80ms, as opposed to 50 Mbps, which is as low as 66ms. The law of diminishing returns also applies to visual quality as bitrate is increased.

Furthermore when closing the ALVR app and restarting it, without restarting the server, I get about 5 minutes of stable streaming until packets start dropping again. Further hinting that the Android App is the issue here.

@KitsuneShan I'm looking forward to your tests with Win 10.
Windows 10 has introduced new systems for scheduling of packets which did not exist in Win 7. There have also been changes to the UDP. I'm trying to figure out if I can revert networking back to Win 7 standard, but because Win 10 is a dumpsterfire, and many Registry entries have been disabled in that regard, it's taking more effort than dual booting Win 7 would.

@FingrMastr you should create a fork here on github, so you don't have to upload your compiled versions to gdrive. I'd be happy to help you with testing again.

from alvr.

Ok reenabling infra-refresh in the NVENC encoder does not resolve the problem.
@Susanoo1337 What an incredible testing!! However, since the beginning of this problem, I always thinked about a problem in the App, indeed. In my case if I continue playing after the FUBAR state (hey I like much this definition ;-)), sometimes the app freezes completely and the entire Quest becomes irresponsive until I stop steamVR (stopping server), and some other times the app simply drops to oculus Quest homeroom, and I need to restart it. In this last case, when I restart it, it goes well for a variable amount of time, and then again FUBAR state.

This tells me two important things:
First, the app main thread seems to be somewhat coupled with receiving data to visualize, or alternatively, a less important thread is going to take too much CPU resources. I bet on the main thread thing, because I profiled the app with Android Studio and the CPU is not taking too much even in FUBAR state. This is absolutely a NO GO for an Android App: If not a crash, an app accessing UI on main thread from parallel threads causes at least freezes and strange graphic behaviors. In this regard, I tried to use thread safe queues in the code, and to assign explicit priority to threads. However I touched only the Java portion of App code, since I'm not experienced in CPP... However my "try it and hope" optimizations did nothing for the FUBAR effect. The java code seems OK...

Second, the various optimizations made to server side until V7 (IDR frame timing, and all the rest) were really enhancing smoothness and responsiveness, but only when not in FUBAR state. Another hint that server portion is doing what needs to be done, and is not causing the "FUBAR state". Even if in different PC system configs and different GPUs the NVENC stream could be a little different, this problem in my opinion is App related.

I'm not enough skilled to analyze CPP code in the app, but I hope this analysis is enough to put someone in the right direction.

from alvr.

Did you check if this issue was triggered by packet loss ? Like if there is suddenly a lot of lost packets the App enters a loop trying to recover the data from the server which makes the whole App freeze and it gets to this overloaded state.

from alvr.

UDP is not working this way. It’s connectionless and there is no “recovery” at all. The App simply uses the last frame arrived and discards the old one, with no queue (there is actually a queue but it has a max size of 1). The app, because of UDP protocol, is not aware of data error or lost frames. Actually, I suspect frame data is arriving correctly, but the app is not able to compute all frames, thus causing stuttering, because of something unwanted happening in the main thread, responsible for visualization. It needs more investigation and I’ll do my best to better debug it. The help from someone with better understanding of the client side app (especially the CPP part) would be very useful.

from alvr.

@FingrMastr I know it's UDP but I saw in ServerConnectionNative.cpp that the client sends lost packets reports to the server whenever it detects it (based on sequence number and synchronized time). Now I didn't check what the server does about those feedbacks but if it replies to some of them (at least for audio/mic data) it could lead to some erratic behavior if it's not properly handled.

I'm just trying to see what can trigger this issue.

from alvr.

After a couple dozen tests it turns out I can go no lower than 88Mbps, regardless of any other factors. That's kind of overkill and gives a bunch of lag, but the packet loss is like... 7 packets every 10 minutes.
I have formatted an old SSD, I'll dual boot into Windows 7 next week and experiment a bit further with that.
I've also played around with the buffer, between 400 and 500 seems to be the sweetspot to the point where there should be no more packet loss. However around 500+ the app seems to build up quite a bit of audio delay.

No amount of messing with the NICs settings and windows 10 settings allowed me to optimize the UDP any further, turns out it's pretty well locked down, which is why Win 10 is trash.

Next along with some testing on Windows 7, I'll f*ck about with the MTU settings for UDP in Win 10.
It'll be nice to be back to using Win 7, so that's something to look forward to.

from alvr.

@Susanoo1337 Did you get to do the Win7 testing?? I'm tempted to setup a partition for Win7 if there's any improvement over Win10

from alvr.

Well, I have news. After a tedious process of hacking together a Win 7 iso that would play ball with my mainboard using hacked chinese drivers, I actually managed to install Win 7 on my PC and test ALVR with it.

Turns out the OS makes no difference. The same issues crop up on Win 7, as they do on Win 10.

This further narrows down the source of the issue, since I have now tested the setup with 3 different networks, 4 different NICs, and 3 different OS installations across 2 different PCs.

The only remaining difference that could possibly matter are the Mainboard Chipset or CPU.
I have an i9-9900k sitting on a ASRock Z390 Extreme 4.
The other PC I tested is an i7-3770k on a ASUS P8Z77.

If everyone and @FingrMastr could share their CPU/MB combo, I think it would be interesting if there's some red thread here in terms of chipset.

For now I'll probably be going back to v1 of ALVR, since that's the only one where I didn't observe the issue on my PC.

from alvr.

I have an i7 4790K on an Asus Sabertooth Z97 Mark 1.
Used to play ALVR with an nVidia 780Ti without issues, playing now with a 2080Ti without issues as well.
I am on Windows 7 and, while I have a different computer with the same hardware but with Windows 10, I couldn't test it yet as that computer has been busy but I still have plans on trying it.

I used to play at like 60MB/s 75% res and H264 when I had the 780Ti without any issue for hours and hours. Now I play at 90MB/s 100% res and H265 with the 2080Ti. I leave all the rest of the settings as default besides enabling the mic.

I use a dedicated high end Asus router with it's own 5GHz WiFi set on a very far channel from the main router since it also have 5 and 2.4GHz WiFi to make sure that both signals does not interfere with each other. Asus router for Quest only under MAC resctriction, no encryption or other kind of protection. This Asus router it's just connected through LAN to the main router and set as access point. This may not be too relevant but just in case.

Also, I am on nvidia drivers 436.30.

from alvr.

See, another interesting thing is the NIC. Your Z97 runs a Realtek 8111-series chip, same as my Z77. On my other PC I've tried a Realtek RTL8153, Intel I219-V and Intel 82574L. Maybe the NIC plays a role here too, but we'd need more people to disclose their Mainboards/Network adapters. The only thing I can try here is to put one of the Intel cards into the other PC and see if the problem occurs.

from alvr.

@Susanoo1337
My ethernet connection: Gigabyte H370 HD3 with Intel® Ethernet Connection I219-V.

Playing Skyrim

• With EV1 no problems but compression (now and then after 20 minutes severe compression and loss of sharpness overall) but a stop and go of ALVR apk does remedy this problem.
• With EV7: superb visuals (RIft S like i dare say...) but after 15 to 20 minutes: degradation galore...

If EV7 could be remedied, we would have a wireless contender for the oculus link.
On a lighter note: I would guess @polygraphene is being hired by Oculus since he hasn't been seen in a while and is working on ALVR by Oculus ;-)

from alvr.

Yeah I am using the Realtek connection, never tried the Intel one but I don't think I would see much difference. One of the only differences that I see between the Win7 and Win10 computer is that, not sure why, my upload speed seems capped at 100MB while the Win10 machine uses the whole bandwidth. But I am not sure why this happens and I don't think it would be something related to Windows itself. I tried updating drivers and switching to the Intel to see if the speed there was different but same result so I suspect that some setting or software (firewall?) may cause this. I never gave this much importance so I simply went with it.

from alvr.

I have some promising news.
Kitsune mentioned something interesting: The 100MB speed cap on their Win 7 machine. This happened to coincide with something I've read about Moonlight streaming having reduced packet loss on 100Mbps networks.

So I've changed my NIC settings "Speed & Duplex" to "100 Mbps Half Duplex" and boom, no more packet loss.
I've ran my standard test, 200kb buffer, 5mil packets, 50Mbps.
Previous result:
Packets sent: 5m
Packets lost: ~8000
Total latency: 66ms

New result:
Packets sent: 5m
Packets lost: 0
Total latency: 73ms

Changing the speed cap to 100 Mbps Half Duplex seems to have doubled my transport latency from 5 to 10 ms, and I experience almost immediate audio lag of about 2s.
100 Mbps Full Duplex was as expected a disaster, resulting in a constant packet loss of about 5% total packets at all times.

So if you have this issue, @TC-VR @FingrMastr I recommend throttling your network adapter to see if that helps. I'll be running more and longer tests myself when time permits.
Good job @KitsuneShan for pointing out such a seemingly innocuous detail.

from alvr.

Please continue the discussion in #70

from alvr.