`AzToolsFramework.Tests.dir` compiles fail with `C1033: cannot open program database` about o3de HOT 3 OPEN

Is this the ticket to investigate why the program database is corrupted?

The error is listed on Microsoft development documentation website: https://learn.microsoft.com/en-us/cpp/error-messages/compiler-errors-1/fatal-error-c1033?view=msvc-170
I think for the O3DE jenkins build server usage, the most likely causes are either 1. the hard disk errors on the EBS volume or 2. more likely we have two .vcproj files writingto the same .pdb file at the same time.

from o3de.

Is this the ticket to investigate why the program database is corrupted?

The error is listed on Microsoft development documentation website: learn.microsoft.com/en-us/cpp/error-messages/compiler-errors-1/fatal-error-c1033?view=msvc-170 I think for the O3DE jenkins build server usage, the most likely causes are either 1. the hard disk errors on the EBS volume or 2. more likely we have two .vcproj files writingto the same .pdb file at the same time.

Yes, this is to investigate causes and remediate. I also suspect (2), but there might be some element of disk latency causing slow file unlocking behavior due to EBS volumes actually being network disks. We could try testing this using the /FS flag though there's that element of significantly increasing build time.

Would something like turning on MTT or other new parallel management features help?

Alternatively, maybe moving to Ninja might yield better behavior overall?

from o3de.

Is this the ticket to investigate why the program database is corrupted?
The error is listed on Microsoft development documentation website: learn.microsoft.com/en-us/cpp/error-messages/compiler-errors-1/fatal-error-c1033?view=msvc-170 I think for the O3DE jenkins build server usage, the most likely causes are either 1. the hard disk errors on the EBS volume or 2. more likely we have two .vcproj files writingto the same .pdb file at the same time.

Yes, this is to investigate causes and remediate. I also suspect (2), but there might be some element of disk latency causing slow file unlocking behavior due to EBS volumes actually being network disks. We could try testing this using the /FS flag though there's that element of significantly increasing build time.

Would something like turning on MTT or other new parallel management features help?

Alternatively, maybe moving to Ninja might yield better behavior overall?

We actually have the MTT feature turned on through the MSBuild Directory.Build.props feature: https://github.com/o3de/o3de/blob/development/cmake/Platform/Common/MSVC/Directory.Build.props#L11-L12

from o3de.