AutoPkg 3.0 RC3 Test Release

[nmcspadden]

Much has changed, internally!

Since AutoPkg 3.0 RC2, I've rewritten the entire Recipe loading logic. This is currently stored in the dev_new_recipe_logic branch, and is being tested for merging into the dev branch here in #926. Since this is not yet merged into dev, GitHub's automatic comparison probably won't do the right thing and generate proper comparison notes.

For now, I am keeping track of the major issues people are experiencing on this Trello board: https://trello.com/b/NlGjpVJm/autopkg-30
(I will also probably move that somewhere else in the future to reduce the spread of information, which is already fragmented enough).

I have successfully tested the rewritten recipe logic on a bunch of recipes, but I need a lot more data, especially from people whose setups don't look like mine. AutoPkg's beauty is that it's flexible enough to run in all kinds of environments and all kinds of ways, and my curse is that I'm largely insulated from the effects of changes on those - so I need all of you to test it for me and tell what's not working.

The ultimate goal here, of course, is to get the new Recipe Map and the new Recipe loading logic finalized so that I can start tackling the major blockers identified in issues already - where it's not loading recipes the way it did previously. The release of AutoPkg 3.0 should essentially be a big performance improvement without any major changes to the way people use AutoPkg (something I haven't always been particularly good at).

Please, test, and generate Discussions or file Issues for things that are blocking you so I can track them and we can work on them accordingly.

Known Issues

• Recipe trust verification doesn't do anything right now. It doesn't verify, and it doesn't fail - every recipe passes regardless. This is being implemented still.
• (Dev only) Most of the unit tests were disabled as I've split up the monolithic autopkglib into smaller modules, and have not yet moved the tests around accordingly. Many more need to be written.
• There's a significant performance hit for the one time the recipe map is generated, until we think of a better way to manage git hashing.
• The behavior of the GITHUB_TOKEN_PATH isn't great. It works in the specific context of Github-related processors, but it's not usable outside of that. This is being re-implemented.
• The behavior of preferences when you combine both JSON/Plist external files along with the built-in macOS preferences domain does not at all work the way you'd expect. Some preferences are just silently used from macOS prefs, some are used from the external file, but it almost certainly will be a surprise either way. For this test release, I recommend you rely on either/or - delete your existing macOS preferences and only use a prefs file, or go the other way and don't use an external preference file. This is something I intend to fix before 3.0 release.
• This release expects you to have already installed AutoPkg on the machine. I have not tested it on a fresh install at all.

What's Changed

• fix github action unittests tests.yaml by @jgstew in fix github action unittests tests.yaml #897
• Deprecate and neuter MunkiCatalogBuilder processor; update MunkiImporter processor by @gregneagle in Deprecate and neuter MunkiCatalogBuilder processor; update MunkiImporter processor #913

Full Changelog: v3.0.0RC2...v3.0.0RC3
See also #926 for the full change in the recipe logic rewrite.

---

This discussion was created from the release AutoPkg 3.0 RC3 Test Release.

[n8felton]

There's a significant performance hit for the one time the recipe map is generated, until we think of a better way to manage git hashing.

Do you want a quick patch for this to bypass calculating the hashes? The hashes aren't being used in the recipe map today, so I'd say it's a worthy change for RC4

[nmcspadden]

Yeah, let's get that listed as a launch blocker too.

[vmiller]

Did some testing against my recipe list. Some recipes were failing with an "Unknown processor" message. After comparing recipes, the ones that are failing are ones with use non-core processors that are not located in the same repo as the recipe.

An example recipe is https://github.com/autopkg/joshua-d-miller-recipes/blob/master/Adobe/AdobeAcrobatDCUpdate.munki.recipe

autopkg@test-mini ~ % autopkg run -v AdobeAcrobatDCUpdate.munki
Looking for AdobeAcrobatDCUpdate.munki...
Found AdobeAcrobatDCUpdate.munki in recipe map overrides
Looking for com.github.joshua-d-miller.autopkg.munki.AdobeAcrobatDCUpdate...
Found com.github.joshua-d-miller.autopkg.munki.AdobeAcrobatDCUpdate in recipe map
Looking for com.github.novaksam.download.AdobeAcrobatDCUpdate...
Found com.github.novaksam.download.AdobeAcrobatDCUpdate in recipe map
**old_load_recipe time: 0.17996290599967324
Processing AdobeAcrobatDCUpdate.munki...
Failed.
Receipt written to /tmp/receipts/AdobeAcrobatDCUpdate-receipt-20240212-195339.plist

The following recipes failed:
AdobeAcrobatDCUpdate.munki
Unknown processor 'com.github.autopkg.novaksam-recipes.Processors/AdobeAcrobatDcUpdateInfoProvider'.

Nothing downloaded, packaged or imported.

[nmcspadden]

Thanks for that, I'll try that out and take a look!