AutoPkg 3.0 RC3 Test Release

Much has changed, internally!

Since AutoPkg 3.0 RC2, I've rewritten the entire Recipe loading logic. This is currently stored in the dev_new_recipe_logic branch, and is being tested for merging into the dev branch here in #926. Since this is not yet merged into dev, GitHub's automatic comparison probably won't do the right thing and generate proper comparison notes.

For now, I am keeping track of the major issues people are experiencing on this Trello board: https://trello.com/b/NlGjpVJm/autopkg-30
(I will also probably move that somewhere else in the future to reduce the spread of information, which is already fragmented enough).

I have successfully tested the rewritten recipe logic on a bunch of recipes, but I need a lot more data, especially from people whose setups don't look like mine. AutoPkg's beauty is that it's flexible enough to run in all kinds of environments and all kinds of ways, and my curse is that I'm largely insulated from the effects of changes on those - so I need all of you to test it for me and tell what's not working.

The ultimate goal here, of course, is to get the new Recipe Map and the new Recipe loading logic finalized so that I can start tackling the major blockers identified in issues already - where it's not loading recipes the way it did previously. The release of AutoPkg 3.0 should essentially be a big performance improvement without any major changes to the way people use AutoPkg (something I haven't always been particularly good at).

Please, test, and generate Discussions or file Issues for things that are blocking you so I can track them and we can work on them accordingly.

Known Issues

• Recipe trust verification doesn't do anything right now. It doesn't verify, and it doesn't fail - every recipe passes regardless. This is being implemented still.
• (Dev only) Most of the unit tests were disabled as I've split up the monolithic autopkglib into smaller modules, and have not yet moved the tests around accordingly. Many more need to be written.
• There's a significant performance hit for the one time the recipe map is generated, until we think of a better way to manage git hashing.
• The behavior of the GITHUB_TOKEN_PATH isn't great. It works in the specific context of Github-related processors, but it's not usable outside of that. This is being re-implemented.
• The behavior of preferences when you combine both JSON/Plist external files along with the built-in macOS preferences domain does not at all work the way you'd expect. Some preferences are just silently used from macOS prefs, some are used from the external file, but it almost certainly will be a surprise either way. For this test release, I recommend you rely on either/or - delete your existing macOS preferences and only use a prefs file, or go the other way and don't use an external preference file. This is something I intend to fix before 3.0 release.
• This release expects you to have already installed AutoPkg on the machine. I have not tested it on a fresh install at all.

What's Changed

• fix github action unittests tests.yaml by @jgstew in #897
• Deprecate and neuter MunkiCatalogBuilder processor; update MunkiImporter processor by @gregneagle in #913

Full Changelog: v3.0.0RC2...v3.0.0RC3
See also #926 for the full change in the recipe logic rewrite.

AutoPkg 3.0 RC2

Full Changelog: v3.0.0RC1...v3.0.0RC2

ISSUES:

• #886 - it's now possible to correctly run recipes within your search directories that aren't directly part of the recipe map (i.e. from a repo). You no longer have to build the map first to find them, as they are found just-in-time.
• #883 - same as above, the recipe map is now rebuilt on demand with the current search dirs when looking through paths
• #884 - the recipe map is built on-demand. Previous RCs would result in AutoPkg erroring out if you didn't add a repo beforehand.

AutoPkg 3.0 RC1

Given the massive number of fundamental changes in AutoPkg behavior, we have decided to promote this version bump to 3.0.0.

Full Changelog from 2.7.2: master...v3.0.0RC1

What's new in 3.0 RC1 compared to 2.8.1 RC6:

Complete rewrite of GitHub integrations

The Github integrations were previously based on curling the GitHub API directly. For various reasons, that's become both unwieldy and unreliable, especially with how much GitHub has applied rate limiting to API calls from unauthenticated sources. The entire stack has been rewritten to rely on the PyGithub module instead.

autopkglib.github now provides a utility class GitHubSession that will store information about a given repo (as well as the AutoPkg repo and orgs itself by default), and can be used as an interface for working with the PyGithub module. Note that the PyGithub module does require installation of another module, so you will need to use this RC's version of Python (which includes it), or do a pip install on your own, in order to support the new functionality.

GitHubReleasesInfoProvider processor has also been completely rewritten to use the new module and class.

As described in the previous 2.8.1 RC6's notes:
The only option now for reading a GitHub token is via the preference GITHUB_TOKEN_PATH. The old GITHUB_TOKEN preference key is no longer used anywhere. You must now store your GH token in a flat text file.

Known issues: the current implementation of the code does not do any rate limit checking or backoff. If you are denied access to the API calls because of rate limits or lack of access, you will get a Python traceback (which is still similar to the behavior from before, but now the error does not come from curl). I hope to add a little bit better of an experience for API rate limiting issues in a future version of AutoPkg.

Previous RC notable items:

Recipe Map

AutoPkg no longer searches through the folders on disk every time it looks for a recipe or processor. Instead, it now builds a recipe map (stored on disk as a JSON file) that tells AutoPkg where to find recipes by shortname and identifier.
The recipe map contains four top level keys:

• identifiers
• overrides
• overrides-identifiers
• shortnames

Each top level key contains a list of key/value pairs corresponding to the full path of a recipe file. Example:

 "identifiers": {
    "com.github.autopkg.AutoPkg1GitMaster": "/Users/nmcspadden/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes/AutoPkg/AutoPkg1GitMaster.pkg.recipe",
...
  "shortnames": {
    ...
    "Adium.download": "/Users/nmcspadden/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes/Adium/Adium.download.recipe",
...

This recipe map is automatically built whenever you add or delete a repo, or create a new recipe. The recipe map automatically maps everything in your RECIPE_SEARCH_DIRS preferences. The map is consulted only on verbs that require it (such as anything involving listing, auditing, generating, or handling recipes and repos).

For existing clients, simply re-adding an existing repo will build your map: autopkg repo-add recipes

IMPORTANT NOTE: Searching for recipes on disk no longer happens - only the recipe map is consulted! If you experience issues with locating recipes, please file a Github Issue with steps to reproduce the problem.

autopkglib separation

This has no effect on AutoPkg usage whatsoever, but it's an important note for anyone familiar with the code. Rather than throw everything into primarily two files (autopkg and init.py), it's now separated into some smaller libraries - at the moment, prefs and common.

autopkglib.common contains some basic common functions - determining platform, and all default path constants.
autopkglib.prefs has all the Preference objects and getting/saving preferences code.

WARNING: if you have any custom processors that loaded functions from autopkglib, such as log or is_mac(), they will need to be adjusted for this change in order to work. You may need to wrap your import in a try/except block while we make this transition. All Core processors have already been fixed. If you are a custom processor author and are not sure how to address this, please contact the AutoPkg maintainers and/or ask in Slack for advice.

More sub-libraries will come as I continue to break the monolith apart and add more tests. AutoPkg still generally assumes "autopkglib" is one big folder of processors, so the Core processors will likely move to their own subfolder once the logic is fully rewritten.

AutoPkg 2.8.1 RC6

https://github.com/autopkg/autopkg/compare/v2.8.1RC5...dev?expand=1

Recipe Map Changes

The Recipe map is now consulted only on verbs that require it (such as anything involving listing, auditing, generating, or handling recipes and repos).
The Recipe map is built when you add or remove a repo, or if you generate a new recipe.

If you do not have a recipe map, or the recipe map is invalid when it is needed, autopkg now tells you what to do next (add or remove a repo to fix it). In the future, I'll add a preference or a command line argument to automatically rebuild vs. failing. (In a CI environment, you probably want it to fail rather than to attempt to build one, as it may have incomplete state at that point. In a regular desktop environment, you probably want it to automatically build and proceed).

autopkglib separation

Rather than throw everything into primarily two files (autopkg and init.py), it's now separated into some smaller libraries - at the moment, prefs and common.

autopkglib.common contains some basic common functions - determining platform, and all default path constants.
autopkglib.prefs has all the Preference objects and getting/saving preferences code.

WARNING: if you have any custom processors that loaded functions from autopkglib, such as log or is_mac(), they will need to be adjusted for this change in order to work. You may need to wrap your import in a try/except block while we make this transition. All Core processors have already been fixed.

More sub-libraries will come as I continue to break the monolith apart and add more tests. AutoPkg still generally assumes "autopkglib" is one big folder of processors, so the Core processors will likely move to their own subfolder once I fix the logic everywhere.

GitHub integration and token usage

The only option now for reading a GitHub token is via the preference GITHUB_TOKEN_PATH. The old GITHUB_TOKEN preference key is no longer used anywhere. You must now store your GH token in a flat text file.

The GitHub logic is being completely rewritten in a different branch, so I've only barely touched it here, and there may be things that break. I don't intend to spend any more effort on the existing GitHub code except to keep it from breaking as I touch things around it. Once this dev branch is done, I'll merge in those changes and work on that next. In the meantime, if you do have existing issues with anything GitHub-related specifically, please file an Issue and I'll make sure that is tested against the new branch.

AutoPkg 2.8.1 RC5

The recipe map is now created automatically when you use any autopkg verb aside from "help" or "version". This also handles situations where we attempted to read an invalid recipe map (such as would be created in between RC3 and RC4) without rebuilding one first.

This makes the autopkg recipe search feature much more seamless, and is a lot closer to how the existing recipe location mechanism works in the release version (2.7.2).

Fixes #874.

AutoPkg 2.8.1 RC4

• AutoPkg can now locate and run overrides by their identifiers. The recipe map now contains both override shortnames and override identifiers to map to the full file paths.

AutoPkg 2.8.1 RC3

Fixes:

• Recipe map now correctly rebuilds after adding a new repo (#868)
• Unit tests now run on the dev branch first
• Fixed some various lint issues
• Fixed an issue where you could you not add or remove recipes if RECIPE_SEARCH_DIRS was missing from the preferences; it now populates the default folders.

Known Issues:

• Preferences store relative/local paths inconsistently. Some are stored in preferences as ~/Library/AutoPkg, others convert to an absolute path upon preference storage like /Users/nmscpadden/Library/AutoPkg.
• You cannot run a recipe not in the recipe map by using a shortname in the local directory. You must use the absolute path to a recipe not in the recipe map.
• AutoPkg search is still wonky; this will be addressed in the next release.

AutoPkg 2.8.1 RC2

Fixes:

• Recipe map now correctly rebuilds after adding a new repo (#868)
• Unit tests now run on the dev branch first
• Fixed some various lint issues

AutoPkg 2.8.0 RC1

New features:

• The recipe map branch has now been merged in. This means that AutoPkg will build a recipe map (JSON on disk) of all repos and recipes when you add or delete repos, and use that for searching for recipes. This drastically speeds up recipe search, especially if you have lots of repos checked out. The old on-disk search method is a fallback, but will be removed in the future.
• The recipe map feature was merged in after a long hiatus, and I can't currently guarantee everything was merged correctly. Please test this and let us know if there are any problems or bugs.
• MunkiImporter now creates pkginfo files with a supported architecture in the filename if and only if there is exactly one supported architecture (i.e. "SomePackage-1.0.0-x86_64.pkginfo"). This will make it much easier to distinguish multiple architecture pkginfos for the same package. A similar PR was merged for Munki (munki/munki#1185).

Fixes:

• Error handling in URLDownloader to address some curl issues (#850)
• Fixes for URLDownloaderPython (#854)
• Indenting blocks in YAML to match spec (#861)
• PkgCopier now only allows copying of .pkg / .mpkg files (#864)

AutoPkg 2.7.2

2.7.2 (December 07, 2022)

• Fix for SparkleUpdateInfoProvider. (#845)

Full Changelog: https://github.com/autopkg/autopkg/compare/v2.7.1..v2.7.2