AWS G4dn.xLarge stuck at 1366 x 768 about parsec-cloud-preparation-tool HOT 81 CLOSED

You guys need to route print in cmd prompt and remove any static/persistant routes if they are present. There is no way the driver will get licensed if it can't talk to Amazon, which it won't be able to do if there are static routes set up by the original machine creation, which are invalid for the AMI you created.

from parsec-cloud-preparation-tool.

I can confirm with route -p delete ip.addr.here for all of the persistant ipv4 routes and a reboot later and I'm back to 1080p.

from parsec-cloud-preparation-tool.

from parsec-cloud-preparation-tool.

I've escalated this issue internally, will comment here again once there's an update.

from parsec-cloud-preparation-tool.

This issue is solved

Copy all text from https://raw.githubusercontent.com/jamesstringerparsec/Cloud-GPU-Updater/master/GPUUpdaterTool.ps1
Open %appdata%\ParsecLoader\GPU Updater Tool.ps1 in Notepad, replace all text in the document with the text you copied from Step 1.
Save the file (do not Save As)
Run GPU updater tool on the desktop

from parsec-cloud-preparation-tool.

I also have this problem when setting up this EC2 instance on EU-west-2.

from parsec-cloud-preparation-tool.

I noticed a new driver added to the s3 bucket last night (442.49). I tried it and it solved this issue for me.
Windows Server 2016, g4dn.xlarge, eu-central-1

The driver is 442.49_grid_win10_64bit_international.exe

Thanks for letting us know, but as far as I can tell, this doesnt fix the issue when using GPU Updater. I've just run it again today g4dn.xlarge, eu-west-1, and its still using the AWS latest driver 442.59, which is the driver thats causing all the problems. So i consider this still something AWS need to fix.

from parsec-cloud-preparation-tool.

Installing 442.49 on g4dn.xlarge in eu-west-2 fixed it. I can now run 4k resolution

from parsec-cloud-preparation-tool.

The code has been once again updated to support AWS G4DN

Copy all text from https://raw.githubusercontent.com/jamesstringerparsec/Cloud-GPU-Updater/master/GPUUpdaterTool.ps1
Open %appdata%\ParsecLoader\GPU Updater Tool.ps1 in Notepad, replace all text in the document with the text you copied from Step 1.
Save the file (do not Save As)
Run GPU updater tool on the desktop

from parsec-cloud-preparation-tool.

I already had an open support ticket with AWS re. this issue and have just asked for a bug to be raised for the driver.

from parsec-cloud-preparation-tool.

The 442.49 driver is not in the nvidia-gaming S3 bucket, but ec2-windows-nvidia-drivers which seems now to support G4 instances as well as G3. It's not a Gaming driver, but a Quadro driver. Previously you could only find G4 drivers in the nvidia-gaming s3 bucket

from parsec-cloud-preparation-tool.

Hey, I can report that I had the same exact issue: I had followed @keithvassallomt's excellent guide in https://github.com/keithvassallomt/parsec-aws-automation, and I could have sworn everything was working fine up until recently when my same exact AMI image started to spin up instances that would refuse to go above 1366x768 or 1280x1024.
I tried GPU Updater and messing around with forcing resolution, but to no avail.

What actually worked for me was the below, thanks @jamesstringerparsec! I removed all 169.254.169.* persistent routes and voila, the driver magically licences itself (don't even need to reboot) and I can immediately change to 1920x1080 again!

I'm running a g4dn.xlarge spot instance in eu-central-1a, with the 445.87 driver.

What is weird is where these routes come from (I suppose "Sets persistent static routes to reach the metadata service and KMS servers." from https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ec2launch.html, and whether removing them affects the Windows licencing? 🤔 - I did happen to see the "Activate Windows" background in another session when I removed the same set of persistent routes. But I don't really mind that as long as the Nvidia driver works properly! 😁

You guys need to route print in cmd prompt and remove any static/persistant routes if they are present. There is no way the driver will get licensed if it can't talk to Amazon, which it won't be able to do if there are static routes set up by the original machine creation, which are invalid for the AMI you created.

from parsec-cloud-preparation-tool.

Thanks for doing that. Should we change the script to install 442.19 in the meantime, as a fix could be a long time coming?

I recommend manually installing the 442.19 driver if you know how to navigate S3, I'm very hesitant to keep updating the script every time AWS drops the ball. This is the 3rd time there has been an issue with g4dn updates due to them changing / breaking things. The long term solution here is to get AWS to fix the issue / update their documentation and stick with it.

from parsec-cloud-preparation-tool.

I noticed a new driver added to the s3 bucket last night (442.49). I tried it and it solved this issue for me.
Windows Server 2016, g4dn.xlarge, eu-central-1

The driver is 442.49_grid_win10_64bit_international.exe

from parsec-cloud-preparation-tool.

I'm using US-East-1

from parsec-cloud-preparation-tool.

Same with ap-southeast-1. It seems the issue may actually lie with the "latest" nvidia driver, specifically 442.59 which this tool and the aws docs tries to install. I did manage to get it working by manually installing 442.19

from parsec-cloud-preparation-tool.

I seem to be having another, possibly related issue. I can't:

a) set a custom resolution (so can't get my monitor resolution of 3440 x 1440)
b) changing the resolution in the NVidia Control Panel actually does nothing at all. The only way to change resolution is via Windows Display Settings.

from parsec-cloud-preparation-tool.

I noticed a new driver added to the s3 bucket last night (442.49). I tried it and it solved this issue for me.
Windows Server 2016, g4dn.xlarge, eu-central-1

The driver is 442.49_grid_win10_64bit_international.exe

Thanks!

from parsec-cloud-preparation-tool.

Hey Charles

My script just follows the instructions provided by the AWS support page.

Instructions to reproduce are located at Option 4 on this support page.
Option 4: NVIDIA Gaming Drivers (G4 Instances)

After following those instructions, the 442.59 driver, located in the nvidia-gaming.s3.amazonaws.com/windows/latest bucket is not licensed and does not allow resolution choices when correctly following setup instructions.

The working driver you're using now (442.49), is listed under "Option 3: GRID Drivers (G3 and G4 Instances)" and is a Quadro driver.

from parsec-cloud-preparation-tool.

@Sam-Kruglov Before I made snapshot from terminated instance and then created AMI from the snapshot.
So, I do AMI based from working instance and I changed Windows from 2016 to 2019 Server.

Also i can see that new instances from my current AMI creates in different area as before. Before it was ap-southeast-1a instead ap-southeast-1b now.

Everything is working perfectly now, 1920x1080 as i wished . Thank you very much for your help!

from parsec-cloud-preparation-tool.

The way I'm solving it these days on new installs is to follow Option 4 from the following guide . It seems to be changes that AWS constantly make on their end and the script can't keep up.

from parsec-cloud-preparation-tool.

The static routes fixed described in the comments above should still work.

from parsec-cloud-preparation-tool.

Did the script request you enter access key and security key information? Did it show any red error messages?

from parsec-cloud-preparation-tool.

Yes, the script need to enter access key and security key, but it do now show any red error message.
However, when I installed GPU driver(Using GPU Update Tool), then run the tool again, it still told me to update the GPU driver:

Checked Now 03/04/2020 02:32:10 - An update is available ( > grid)
Update now? - (!) Machine will automatically reboot if required (!)
(Y/N):

After I update the driver, this message still do not disappear, and the resolution stuck at 1366x768 too.

First Install:

Installed, than update again:

from parsec-cloud-preparation-tool.

Thank you for the information, it seems AWS has changed something on their end again. I will have to resolve it. It's not the scripts fault unfortunately, but more Amazon keeps changing how they store drivers for the G4.

from parsec-cloud-preparation-tool.

Thank you very much. I try to follow the guide on AWS Support (https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/install-nvidia-driver.html), then install the driver without script,
but it's still cannot solve the problem -- So I think there are something wrong with AWS too ;)

from parsec-cloud-preparation-tool.

I just created a G4.dn instance and uses GPU updater to install the driver...After a reboot I was able to select 4K resolution. Are you creating your instance from a base Server 2016 / Server 2019 image? Or an AMI?

from parsec-cloud-preparation-tool.

I just created a G4.dn instance and uses GPU updater to install the driver...After a reboot I was able to select 4K resolution. Are you creating your instance from a base Server 2016 / Server 2019 image? Or an AMI?

I creating my instance from base Server 2016 image (Which provided by AWS). May I ask which region are you using? I think maybe there are some problem in Hong Kong region... ;)

from parsec-cloud-preparation-tool.

I confirmed there are some problem in Hong Kong region -- I can select 4K resolution when I using US-East-1 and ap-southeast-1 region, only Hong Kong region cannot made it.
Thank you very much. ;)

from parsec-cloud-preparation-tool.

Confirming I'm currently having the same issue in AP-Southeast-2

from parsec-cloud-preparation-tool.

Same with ap-southeast-1. It seems the issue may actually lie with the "latest" nvidia driver, specifically 442.59 which this tool and the aws docs tries to install. I did manage to get it working by manually installing 442.19

Thank you ! This workaround is great. It look like "Lastest" nvidia driver can not work on these region's server.

from parsec-cloud-preparation-tool.

Will you guys please contact AWS and tell them that their driver doesn't work?

If all of you do it individually hopefully this will increase the changes of this being fixed quickly.

From what I can tell, nothing in the process of adding the cert to the machine and changing the registry has changed...so this must be down to the driver itself, or the cert they're providing being invalid.

Essentially all this script does is follow the AWS documentation to install the driver per their instructions.

from parsec-cloud-preparation-tool.

Will you guys please contact AWS and tell them that their driver doesn't work?

If all of you do it individually hopefully this will increase the changes of this being fixed quickly.

From what I can tell, nothing in the process of adding the cert to the machine and changing the registry has changed...so this must be down to the driver itself, or the cert they're providing being invalid.

Essentially all this script does is follow the AWS documentation to install the driver per their instructions.

I want to contact AWS to tell them this problem, but their free customer support look like can not support technical issues. Are there any method for contact with AWS ?

from parsec-cloud-preparation-tool.

I already had an open support ticket with AWS re. this issue and have just asked for a bug to be raised for the driver.

Legend, thanks!

from parsec-cloud-preparation-tool.

Thanks for doing that. Should we change the script to install 442.19 in the meantime, as a fix could be a long time coming?

from parsec-cloud-preparation-tool.

I seem to be having another, possibly related issue. I can't:

a) set a custom resolution (so can't get my monitor resolution of 3440 x 1440)
b) changing the resolution in the NVidia Control Panel actually does nothing at all. The only way to change resolution is via Windows Display Settings.

I think it is related issues. I never change my resolution successfully when using G4dn.xlarge in Hong Kong region too.
Can you change the resolution after using @caasig 's workaround?

from parsec-cloud-preparation-tool.

What about using NVIDIA Gaming PC - Windows Server 2016 it has the 442.19 already installed. Ill give it a try shortly.

from parsec-cloud-preparation-tool.

I think it is related issues. I never change my resolution successfully when using G4dn.xlarge in Hong Kong region too.
Can you change the resolution after using @caasig 's workaround?

Nope, running v442.19 but issue remains.

from parsec-cloud-preparation-tool.

What about using NVIDIA Gaming PC - Windows Server 2016 it has the 442.19 already installed. Ill give it a try shortly.

I tried this AMI before create this issues, but it do not work as cannot find the Windows password :(

from parsec-cloud-preparation-tool.

I think it is related issues. I never change my resolution successfully when using G4dn.xlarge in Hong Kong region too.
Can you change the resolution after using @caasig 's workaround?

Nope, running v442.19 but issue remains.

Understand. So it look like this bug should be fixed by AWS or NVDIA too :(

from parsec-cloud-preparation-tool.

What about using NVIDIA Gaming PC - Windows Server 2016 it has the 442.19 already installed. Ill give it a try shortly.

Works fine in eu-west-1, g4dn.xlarge, variety of resolutions now available including 1080 and 4k

Update: only thing that didnt work was the auto login of parsec

from parsec-cloud-preparation-tool.

I'm confused on how there's a region-specific component to this. Shouldn't the 442.19 driver be the same regardless of what region you're in?

from parsec-cloud-preparation-tool.

Can confirm 442.49 fixed my issue with not being able to change resolutions in Nvidia Control Panel although still can't set a custom resolution. Also continues to work in being able to set a resolution over the previously stuck 1366 x 768.

from parsec-cloud-preparation-tool.

The 442.49 driver is not in the nvidia-gaming S3 bucket, but ec2-windows-nvidia-drivers which seems now to support G4 instances as well as G3. It's not a Gaming driver, but a Quadro driver. Previously you could only find G4 drivers in the nvidia-gaming s3 bucket

Is there any difference performance wise between the gaming driver and Quadro driver?

from parsec-cloud-preparation-tool.

I am having the same issue with a g4dn.xlarge instance in Hong Kong region (ap-east-1b)

from parsec-cloud-preparation-tool.

Resolution problem is solved for me too with 442.49

from parsec-cloud-preparation-tool.

Hey Guys,
I also had this issue and the 442.49 driver install fixed it for me. Thanks guys, and thanks to the maintainer(s) of this awesome script!

Region: eu-west-2 (London)
Instance Type: g4dn.xlarge
OS: Windows Server 2019

from parsec-cloud-preparation-tool.

Folks, if you want this issue to be resolved for everyone else, you will need to contribute to this work by contacting Amazon support, and telling them their driver is not licensed for the latest windows nvidia-gaming driver. Your help would be appreciated in helping to close this github issue.

from parsec-cloud-preparation-tool.

Hi @jamesstringerparsec

Happy to do so, however for the sake of us n00bs here, I will suggest how I would describe it to AWS support and maybe you can confirm I haven't got any wires crossed?

Hi Support,
It looks like there is an issue with the current nVidia drivers for the g4dn.xlarge instances using the T4 GPU in some regions. nVidia gaming driver version 442.59 is not licenced and so resolution choices are very limited (max 1366x768) with this driver installed. Installing version 442.49 (available here: https://ec2-windows-nvidia-drivers.s3.us-east-1.amazonaws.com/g4/latest/442.49_grid_win10_64bit_international.exe) fixed the issue.
Can you please update the bucket with the fixed driver?

Region: eu-west-2 (London)
Instance Type: g4dn.xlarge
OS: Windows Server 2019

The only thing I want to add is a 'steps to reproduce' section, but I don't really know how they can reproduce the issue without installing and using Parsec - When I was having this issue, it wasn't restricting my resolutions when connected via RDP.

Suggestions welcome.
Cheers

from parsec-cloud-preparation-tool.

Sigh @jamesstringerparsec

I only have basic support, which apparently isn't enough to report a bug. I went to the Dev forums to post an issue there, but as it's my first visit there my account is too fresh to post yet #rageface

I will try to post a bit later once my account has matured slightly. In the meantime, I have added feedback via the feedback link at the bottom of that docs page.

EDIT: Posted on forum. Link here

from parsec-cloud-preparation-tool.

@orlyandico Hello, are you an employee of AWS? If so, thank you.

from parsec-cloud-preparation-tool.

I am not sure my problem is related to the same issue reported here but using GPU Update Tool after having run the preparation tool did not properly configure the graphic drivers.

The GPU Update Tool displayed some error during the installation but successfully rebooted the system. Unfortunately, the drivers were not installed.

Install the driver manually does not solve the problem. I am running AWS G4dn.xLarge in London

from parsec-cloud-preparation-tool.

I am not sure my problem is related to the same issue reported here but using GPU Update Tool after having run the preparation tool did not properly configure the graphic drivers.

The GPU Update Tool displayed some error during the installation but successfully rebooted the system. Unfortunately, the drivers were not installed.

Install the driver manually does not solve the problem. I am running AWS G4dn.xLarge in London

Hi Alberto,
Can you provide the error message which displayed by GPU Update Tool ? Maybe it can help to find the problem.

from parsec-cloud-preparation-tool.

I think it's the same error

from parsec-cloud-preparation-tool.

It didn't work. Every time I initialize an instance I got stuck in 1366 x 768

from parsec-cloud-preparation-tool.

It didn't work. Every time I initialize an instance I got stuck in 1366 x 768

I updated my comment

from parsec-cloud-preparation-tool.

Hey! I'm facing a similar issue. I'm using g4dn.xlarge spot requests and when I launch a new instance then run this script, I can get everything working and select from higher resolutions.

However, when I create an AMI from this instance, and then spin up an instance from that new AMI, the resolution is again limited to 1366 x 768. Any ideas? I'm in us-east-2 by the way.

Thanks in advance. :)

from parsec-cloud-preparation-tool.

@royaldallas set everything up on the on-demand instance and take ami from there. To guarantee image integrity it needs to be able to reboot which it can't on spot instances

from parsec-cloud-preparation-tool.

Hello. I have same issue as @royaldallas . @Sam-Kruglov I made AMI from on-demand instance which supports high resolution. But i get resolution 1366x768 when i create new instance based on the AMI i made.

This issue is solved

Copy all text from https://raw.githubusercontent.com/jamesstringerparsec/Cloud-GPU-Updater/master/GPUUpdaterTool.ps1
Open %appdata%\ParsecLoader\GPU Updater Tool.ps1 in Notepad, replace all text in the document with the text you copied from Step 1.

Also it doesn't help.

Installing version 442.49 (available here: https://ec2-windows-nvidia-drivers.s3.us-east-1.amazonaws.com/g4/latest/442.49_grid_win10_64bit_international.exe) fixed the issue.

And this one.

Other thing is If i connect via RDP and via Parsec at the same time I got 1920x1080.

My region is ap-southeast-1 (Singapore), g4dn.xlarge, Windows Server 2016.

May you give me any tips how to fix it? Thank you.

from parsec-cloud-preparation-tool.

I had the same issue when I captured AMI from a spot instance and checked the "no reboot" checkbox. After that I created raw windows server 2019 on an on-deman instance, ran this tool, installed the games, and captured the AMI. After that it works every time I spawn a spot instance with my launch template

from parsec-cloud-preparation-tool.

@Sam-Kruglov @Sarets Thanks for the reply. I tried creating an AMI from an on-demand instance in us-east-2 after running this script, and when I launch spot instances from that AMI, it still gives me the 1366 x 768 resolution.

Did you guys do anything other than run the basic script? Did you try any of the things in these comments?

Any other suggestions?

Thanks again!

from parsec-cloud-preparation-tool.

Also what drivers did you use and what region were you guys in?

from parsec-cloud-preparation-tool.

@royaldallas

from parsec-cloud-preparation-tool.

@royaldallas actually today wanted to make a small change to my AMI and got the same issue. I think it might be related to the fact that new AMI builds from the closest EBS snapshot. Try to delete all EBS snapshots related to the instance you're taking AMI from so that the snapshot goes from zero to full.

from parsec-cloud-preparation-tool.

@Sam-Kruglov were you able to fix it? It looks like my instances from the same AMI have the correct resolution in one availability zone, but not others. So I'm not sure if it's the AMI or an issue on Amazon's end.

from parsec-cloud-preparation-tool.

Hello @royaldallas
I did everything according this instruction https://github.com/keithvassallomt/parsec-aws-automation
Except any automation scripts and i did win server 2019 instead 2016.
And important thing, i do AMI image from "started"(not "terminated") instance directly. I mean right click on instance then create image. In this case when i launch instance from AMI - it starts in the same zone as initial instance.

BTW: video is Tesla T4, driver 445.87(installed automatically by instruction in link above)

from parsec-cloud-preparation-tool.

Another data point, I set up a brand new g4dn.2xlarge instance (worked fine). Then created an AMI for it, spun up and down 2x instances from that AMI and both worked fine. Then today I stood up an instance from that same AMI and was stuck at 1366x768. I'm confused why the behavior would change with the same AMI!

from parsec-cloud-preparation-tool.

Same for me. Creating an instance from AMI image. Stuck at 1366x768. Tried GPUUpdaterTool with no luck.

from parsec-cloud-preparation-tool.

Yep. Worked. Thank you, @jamesstringerparsec

from parsec-cloud-preparation-tool.

The script reports Your PC already has the latest NVIDIA GPU Driver (443.18) available from nvidia.com and stops.

But NVIDIA Control Panel reports 445.87 and the resolution is limited to 1366x768 still (g4dn.xlarge us-east-2a, this is from the NVidia AMI).

Even more strange, some of my systems do not have the problem and never needed the script to begin with, and offer the higher resolutions.

from parsec-cloud-preparation-tool.

@TTimo The README isn't really explicit about it, but I believe this tool is designed to be used with the Windows server AMIs, not the Nvidia ones.

from parsec-cloud-preparation-tool.

If I'm starting two instances, one on us-east-2a and one on us-east-2c, the -2a one reports GRID Virtual Applications in nvidia-smi.exe, the resolution is limited, and I can't find a way to fix it. The -2c however reports GRID vGaming and the higher resolutions are available.

This looks like an AWS problem to me. We have repeatedly created instances in -2c that are are fine, and created instances in -2a that are broken, coming out of the same AMI.

from parsec-cloud-preparation-tool.

@TTimo Please read my comment from May 20th.

from parsec-cloud-preparation-tool.

@jamesstringerparsec dangit! I understand your comment now.

nvidia-smi reports License Status : Licensed - but the VM is actually silently failing in some way. That's just nasty :(

from parsec-cloud-preparation-tool.

I faced same issue as before today. Amazon created instance from fine AMI in wrong location ap-southeast-1b, which always affect video settings and geting stuck resolution. I found that seems like possible to choose right "Availability zone" in process of creating instance from AMI. On "Step 3: Configure Instance Details" i checked "Request Spot instances" and then in parameter "Subnet" i choosed subnet from right location, in my case ap-southeast-1a. But maybe it was luck, i don't know. I hope so it will help somebody.

from parsec-cloud-preparation-tool.

@jamesstringerparsec Your ongoing support on this issue has been amazing. One extra question...does your latest script update (above) automatically delete all persistant ipv4 routes or does that still need to be donw manually?
(the persistant route thing is what was causing the issue for me)

from parsec-cloud-preparation-tool.

@Buzzpod You need to do it yourself as I don't know what else deleting those persistent routes does, perhaps it also stops your copy of Windows being licensed, or things of that nature. I don't know.

from parsec-cloud-preparation-tool.

You guys need to route print in cmd prompt and remove any static/persistant routes if they are present. There is no way the driver will get licensed if it can't talk to Amazon, which it won't be able to do if there are static routes set up by the original machine creation, which are invalid for the AMI you created.

Thank you.

from parsec-cloud-preparation-tool.

I'm getting the what appears to be essentially the same issue as described above using the latest Windows Server 2019 Base image from AWS. All this was done in AWS West-2 (Oregon) on a NVIDIA RTX Virtual Workstation - WinServer 2019-vGPU with a g4dn.xlarge instance.

Created the instance in AWS, then ran the ps1 Parsec Prep tool and can't get anything above 1366 x 768.

I've run the GPU Updater tool and have the latest available 466.47 Driver version.

from parsec-cloud-preparation-tool.

So a manual install didn't fix it either. The latest version available from the BitBucket is identical to the one the GPU Updated powershell script grabbed and installed.

At this point, I assume this might just be a problem with this version (466.47) that might be restricting Display resolutions rather than the tool or setup. 🤔

from parsec-cloud-preparation-tool.

I have tried this with Windows Server 2022 and g4dn.xlarge (the drivers have been installed manually as per AWS documentation). Resolution is stuck and can not be changed even after the reboot. No static routes present.

I have also tried Windows Server 2019 but I do not see any static routes, and the resolution is still stuck.

I have also tried Nvidia provided AMI (the Gaming one) for Windows Server 2019 but that one does not start the RDP server so there is no way to connect to it.

The AWS g4dn has a very bad user experience.

from parsec-cloud-preparation-tool.