Manifest/Pull Request Testing Environment

[vedantmgoyal9]

#26251 (review) Here's a discussion where we can talk about and suggest a better way to test manifests.

[jedieaston]

The options, as I see them at this point:

• Run it in a VM.

  Pros:

• • The most "real" environment you can get, since you are running on a real Windows install with no file sharing or other things.
• • Scalable. You can run as many as you have ram/cpu/disk for.

• • You can make your own image with winget already installed.

    Cons:

• • Slow! If you want a clean environment every time, you have to wait for a new VM to boot (like 90 seconds from click to desktop, presuming modest specs in Hyper-V, SSD, etc). That's assuming that you didn't do the right thing to get a clean environment by booting from a sysprepped image, which would take minutes to initialize. You could just use the same VM for multiple manifests, but that means you have to wait for uninstalls, which stinks.
• • There's not a lot of great tools for managing ephemeral Windows VMs, at least that I've seen. Vagrant is slow as heck on Windows especially with Hyper-V (not even to bring up the vm, but any Vagrant command seems super slow on my machine). Hyper-V has templates but they are super annoying to import and export. Snapshots work, but they still take 25-30 seconds to boot on my machine.

• Run it in Windows Sandbox

  Pros:

• • Fast! Click to desktop in like 10-15 seconds. Doesn't take up much disk space either.
• • Better memory management, it will free up memory when the host asks for it.
• • Resets to a clean image every time.

  Cons:

• • You can only run one at a time. If you want to test something like JetBrains.dotUltimate which takes 20-30 minutes to install, you can't do anything else while it runs. Doesn't matter if you're setting at 50% ram and disk free, it won't do it. I don't see a technical problem here, given that I can run Application Guard and Sandbox side by side just fine. Does anyone know why this is?
• • Inaccurate. Certain applications behave differently due to the way the Sandbox shares the kernel and system files. (I can't think of one off the top of my head, but I know we've had apps that would only install in a real VM). Also, since the Sandbox has UAC disabled, it's very hard to accurately test user-scoped apps (see The validation in sandbox command should not run as admin by default #26177).
• • No way to customize the image. SandboxTest.ps1 and anyone else who wants to use winget in a Sandbox, even on Windows 11, has to provision it on every run. This takes 10 seconds easy and is super annoying when I just want to check something quick.

• Run it in a Windows Container

  Pros:

• • Fast! Boots in like 10 seconds.
• • You can customize the image via a Dockerfile and create your own image.
• • Since it's docker, all of the typical docker options apply (--rm to delete when you exit, shared folders are easy, etc).
• • You can run as many as you want, and since they are sharing the host's kernel (by default), they take less RAM than VMs.

  Cons:

• • No GUI access. This makes it hard to accurately test most software in the repository, as I can see its ARP entry and that winget exited successfully, but I can't actually run the software since it usually relies on being able to draw something on the screen.
• • winget won't install in microsoft/windows containers for some reason (I literally get an Unknown Error from Add-AppxPackage). You have to extract the appx package and run AppInstallerCLI.exe to use it.

(I only put the Windows Container option on here hoping someone from Microsoft would say whether or not they'll ever let me RDP into a container. A winget moderator can dream :D)

On Linux I'd just docker run and forget about it, but on Windows, there's a lot less flexibility there. Does anyone have any better ideas for automatically standing up and tearing down testing environments locally that don't have all of these trade offs? It's a serious bottleneck for moderators and annoyance for everyone else, I'm sure. Also, once microsoft/winget-cli#1073 hits, we'll need to run a manifest while we are making it to get the ARP entries for every installer, making a quick way to run the manifests all the more important.

p.s: @denelon, can you share anything about how the validation environment is provisioned during the pipeline run? Is a new VM spun up on Azure for every run?

tagging @OfficialEsco @ItzLevvie so they can share their setups.

[OfficialEsco]

I think you are overthinking the process, it don't think we can get to the amount of testing wingetbot does, but we don't actually have to because wingetbot already does that, IT JUST NEEDS TO EXPLAIN THE ISSUE IT SEES 😠

We can catch the clear issues with a installer, wingetbot catches the smaller issues.
We kinda just need to check the metadata and in the future for Dependency's, so you don't actually have to use a brand new Windows Installation, just don't have any dependency's installed.

I am pretty sure both me and Levvie use a VM and reset to a "clean" snapshot environment when we are done, this means we can boot up the VM, have the Community Moderators - 2021-08-27 - experimental.cmd script already ready to go and test a bunch of PRs quickly, then just uninstalling the dependency's that some software installs.

I use Virtualbox which works great because i do not like Hyper-V and i prefer my Host to not act as Virtual Machine.
I also do not like to use Sandbox because of the disabled UAC Prompt which is useful for detecting the software's scope.

[denelon]

I appreciate the list of pro's and con's by @jedieaston for the different choices of environment. I also agree with @OfficialEsco on the validation. Our validation does use a special environment for testing (a VM in a dedicated environment). We are working with several other teams on reporting the findings so we can surface them with the bots. In some cases we simply get a pass/fail or the response from the service times out. There are also concerns about how the data gets reported. We need to make sure the data is formatted in a consistent manner. We are trying to avoid building something completely custom when several other services exist like SmartScreen. Other automated tests produce a high quantity of false negative results. In those cases, we've implemented the waiver system with the .validation files so a person can review the submission against policies to make a determination if a manifest should pass. The engineering team is simply responding to the business decisions when they are adding .validation files with waivers.

The challenges we currently have with matching data in Apps & Features will hopefully be resolvable with the new schema and enhancements to validation. We're also working through dependencies so they can also be tested appropriately. Our next release includes a few other improvements in the client, and then we will begin focusing more on the bugs we've been seeing to improve the experience with the client. Most of the challenges we are encountering are due to legacy installers, and the way they report packages once they have been installed. We knew this was going to be a challenge, and we knew we were going to have to work with Independent Software Vendors and publishers to improve the overall customer experience. This is a huge team effort, and it will continue to require collaboration and feedback. Keep submitting Issues for new Bugs and new Feature requests. They are helping to drive the direction of the product, and they are letting us know where we need to spend our engineering resources.