AzToolsFramework.Tests.dir compiles fail with C1033: cannot open program database

[amzn-changml]

Describe the bug
After random merges (lately after large cherry-picks), MSBuild will fail to build AzToolsFramework.Tests.dir unity files while attempting to access vc143.pdb. Subsequent builds will continue to fail until the output directory is deleted.

Assets required
Latest log from Jenkins: https://jenkins.build.o3de.org/blue/rest/organizations/jenkins/pipelines/O3DE/branches/development/runs/4984/nodes/580/log/?start=0

Steps to reproduce
Steps to reproduce the behavior:

1. Build the Profile target in Windows with: cmake --build . --target ALL_BUILD --config profile -- /m /nologo

Expected behavior
The build completes successfully

Actual behavior
The build fails with fatal error C1033: cannot open program database 'D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\AzToolsFramework.Tests.dir\profile\vc143.pdb'

Screenshots/Video
N/A

Found in Branch
Development

Commit ID from o3de/o3de Repository
7ad0894

Desktop/Device (please complete the following information):

• Device: AWS EC2
• OS: Windows Server
• Version: 2022 (eqv to Windows 11)
• CPU: AMD EPYC 7R13
• GPU: N/A
• Memory 32GB

Additional context
Log message snippet is as follows:

[2024-04-25T11:15:32.917Z] D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\CMakeFiles\AzToolsFramework.Tests.dir\Unity\unity_8_cxx.cxx : fatal error C1033: cannot open program database 'D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\AzToolsFramework.Tests.dir\profile\vc143.pdb' [D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\AzToolsFramework.Tests.vcxproj]
[2024-04-25T11:15:32.917Z] D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\CMakeFiles\AzToolsFramework.Tests.dir\Unity\unity_16_cxx.cxx : fatal error C1033: cannot open program database 'D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\AzToolsFramework.Tests.dir\profile\vc143.pdb' [D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\AzToolsFramework.Tests.vcxproj]
[2024-04-25T11:15:32.917Z] D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\CMakeFiles\AzToolsFramework.Tests.dir\Unity\unity_3_cxx.cxx : fatal error C1033: cannot open program database 'D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\AzToolsFramework.Tests.dir\profile\vc143.pdb' [D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\AzToolsFramework.Tests.vcxproj]
[2024-04-25T11:15:32.917Z] D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\CMakeFiles\AzToolsFramework.Tests.dir\Unity\unity_15_cxx.cxx : fatal error C1033: cannot open program database 'D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\AzToolsFramework.Tests.dir\profile\vc143.pdb' [D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\AzToolsFramework.Tests.vcxproj]
[2024-04-25T11:15:32.917Z] D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\CMakeFiles\AzToolsFramework.Tests.dir\Unity\unity_10_cxx.cxx : fatal error C1033: cannot open program database 'D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\AzToolsFramework.Tests.dir\profile\vc143.pdb' [D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\AzToolsFramework.Tests.vcxproj]
[2024-04-25T11:15:32.917Z] D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\CMakeFiles\AzToolsFramework.Tests.dir\Unity\unity_12_cxx.cxx : fatal error C1033: cannot open program database 'D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\AzToolsFramework.Tests.dir\profile\vc143.pdb' [D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\AzToolsFramework.Tests.vcxproj]
[2024-04-25T11:15:32.917Z] D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\CMakeFiles\AzToolsFramework.Tests.dir\Unity\unity_17_cxx.cxx : fatal error C1033: cannot open program database 'D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\AzToolsFramework.Tests.dir\profile\vc143.pdb' [D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\AzToolsFramework.Tests.vcxproj]
[2024-04-25T11:15:32.917Z] D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\CMakeFiles\AzToolsFramework.Tests.dir\Unity\unity_6_cxx.cxx : fatal error C1033: cannot open program database 'D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\AzToolsFramework.Tests.dir\profile\vc143.pdb' [D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\AzToolsFramework.Tests.vcxproj]
[2024-04-25T11:15:32.917Z] D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\CMakeFiles\AzToolsFramework.Tests.dir\Unity\unity_7_cxx.cxx : fatal error C1033: cannot open program database 'D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\AzToolsFramework.Tests.dir\profile\vc143.pdb' [D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\AzToolsFramework.Tests.vcxproj]
[2024-04-25T11:15:32.917Z] D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\CMakeFiles\AzToolsFramework.Tests.dir\Unity\unity_9_cxx.cxx : fatal error C1033: cannot open program database 'D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\AzToolsFramework.Tests.dir\profile\vc143.pdb' [D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\AzToolsFramework.Tests.vcxproj]
[2024-04-25T11:15:32.917Z] D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\CMakeFiles\AzToolsFramework.Tests.dir\Unity\unity_13_cxx.cxx : fatal error C1033: cannot open program database 'D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\AzToolsFramework.Tests.dir\profile\vc143.pdb' [D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\AzToolsFramework.Tests.vcxproj]
[2024-04-25T11:15:32.917Z] D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\CMakeFiles\AzToolsFramework.Tests.dir\Unity\unity_4_cxx.cxx : fatal error C1033: cannot open program database 'D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\AzToolsFramework.Tests.dir\profile\vc143.pdb' [D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\AzToolsFramework.Tests.vcxproj]
[2024-04-25T11:15:32.917Z] D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\CMakeFiles\AzToolsFramework.Tests.dir\Unity\unity_5_cxx.cxx : fatal error C1033: cannot open program database 'D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\AzToolsFramework.Tests.dir\profile\vc143.pdb' [D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\AzToolsFramework.Tests.vcxproj]
[2024-04-25T11:15:32.917Z] D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\CMakeFiles\AzToolsFramework.Tests.dir\Unity\unity_11_cxx.cxx : fatal error C1033: cannot open program database 'D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\AzToolsFramework.Tests.dir\profile\vc143.pdb' [D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\AzToolsFramework.Tests.vcxproj]
[2024-04-25T11:15:32.917Z] D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\CMakeFiles\AzToolsFramework.Tests.dir\Unity\unity_14_cxx.cxx : fatal error C1033: cannot open program database 'D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\AzToolsFramework.Tests.dir\profile\vc143.pdb' [D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\AzToolsFramework.Tests.vcxproj]
[2024-04-25T11:15:33.174Z] D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\CMakeFiles\AzToolsFramework.Tests.dir\Unity\unity_18_cxx.cxx : fatal  error C1033: cannot open program database 'D:\workspace\o3de\build\windows\Code\Framework\AzToolsFramework\AzToolsFramework.Tests.dir\profile\vc143.pdb' 

[lemonade-dm]

Is this the ticket to investigate why the program database is corrupted?

The error is listed on Microsoft development documentation website: https://learn.microsoft.com/en-us/cpp/error-messages/compiler-errors-1/fatal-error-c1033?view=msvc-170
I think for the O3DE jenkins build server usage, the most likely causes are either 1. the hard disk errors on the EBS volume or 2. more likely we have two .vcproj files writingto the same .pdb file at the same time.

[amzn-changml]

Is this the ticket to investigate why the program database is corrupted?

The error is listed on Microsoft development documentation website: learn.microsoft.com/en-us/cpp/error-messages/compiler-errors-1/fatal-error-c1033?view=msvc-170 I think for the O3DE jenkins build server usage, the most likely causes are either 1. the hard disk errors on the EBS volume or 2. more likely we have two .vcproj files writingto the same .pdb file at the same time.

Yes, this is to investigate causes and remediate. I also suspect (2), but there might be some element of disk latency causing slow file unlocking behavior due to EBS volumes actually being network disks. We could try testing this using the /FS flag though there's that element of significantly increasing build time.

Would something like turning on MTT or other new parallel management features help?

Alternatively, maybe moving to Ninja might yield better behavior overall?

[lemonade-dm]

Is this the ticket to investigate why the program database is corrupted?
The error is listed on Microsoft development documentation website: learn.microsoft.com/en-us/cpp/error-messages/compiler-errors-1/fatal-error-c1033?view=msvc-170 I think for the O3DE jenkins build server usage, the most likely causes are either 1. the hard disk errors on the EBS volume or 2. more likely we have two .vcproj files writingto the same .pdb file at the same time.

Yes, this is to investigate causes and remediate. I also suspect (2), but there might be some element of disk latency causing slow file unlocking behavior due to EBS volumes actually being network disks. We could try testing this using the /FS flag though there's that element of significantly increasing build time.

Would something like turning on MTT or other new parallel management features help?

Alternatively, maybe moving to Ninja might yield better behavior overall?

We actually have the MTT feature turned on through the MSBuild Directory.Build.props feature: https://github.com/o3de/o3de/blob/development/cmake/Platform/Common/MSVC/Directory.Build.props#L11-L12