AutoPkg 2.7.2

2.7.2 (December 07, 2022)

• Fix for SparkleUpdateInfoProvider. (#845)

Full Changelog: https://github.com/autopkg/autopkg/compare/v2.7.1..v2.7.2

AutoPkg 2.7.1

2.7.1 (December 06, 2022)

• GitHubReleasesInfoProvider -- add support for a "latest_only" input variable (#846)
• GitHubReleasesInfoProvider -- add "asset_created_at" output variable
• PkgPayloadUnpacker -- use "aa" to expand archives if "ditto" fails and "aa" is available (#804)
• Fix for plist serialization when json input contains null values (#810)
• SparkleUpdateInfoProvider -- add support for channels (716dcea)
• Fix for autopkginstalld on recent versions of macOS. Fixes .install recipes. (#838)

Full Changelog: https://github.com/autopkg/autopkg/compare/v2.7..v2.7.1

AutoPkg 2.7

2.7.0 (Aug 22, 2022)

Python 3.10.6

AutoPkg now uses Python 3.10.6, and PyObjc 8.5. Python 3.10 brings several new general improvements and may result in some warnings or errors in processors still using some legacy Python2-conversion-to-3-isms. Please file appropriate issues with recipe authors or bring it to our attention in #autopkg in Slack.

Automated tests on GitHub

Thanks to the incredible work by @jgstew, AutoPkg now has automatic unit tests and a test recipe being run as an automatic GitHub action! This will help provide confidence in AutoPkg's functionality, and empowers contributors to be more confident in testing their code. Here's hoping this opens up more more people interested in contributing!

Similarly, thanks to @homebysix for doing a similar action for linting, ensuring consistent Python style rules are being applied.

Other Changes

• the make_new_release script is a lot easier to use (only really benefits maintainers, but hey)
• remove incompatibility notice non-macs by @jgstew in #795
• add automated UnitTests for AutoPkg with GitHub Actions by @jgstew in #796
• Create linting/auto-formatting workflow in GH Actions by @homebysix in #793
• Always return list when finding matching pkgs by @octomike in #794
• Preserve YAML scalar types when writing YAML to disk by @bfreezy in #770
• MunkiInstallsItemsCreator.py - Add option to derive minimum os version by @macmule in #801
• Update tests.yaml - remove bad schedule by @jgstew in #815
• Fix URLGetter.py when default windows curl is used. by @jgstew in #816

Full Changelog: v2.7.0...v2.5

AutoPkg 2.6.0 Beta

2.6.0 (April 18, 2022)

Python 3.10

AutoPkg now uses Python 3.10, and PyObjc 8.2. Python 3.10 brings several new general improvements and may result in some warnings or errors in processors still using some legacy Python2-conversion-to-3-isms. Please file appropriate issues with recipe authors or bring it to our attention in #autopkg in Slack.

Automated tests on GitHub

Thanks to the incredible work by @jgstew, AutoPkg now has automatic unit tests and a test recipe being run as an automatic GitHub action! This will help provide confidence in AutoPkg's functionality, and empowers contributors to be more confident in testing their code. Here's hoping this opens up more more people interested in contributing!

Similarly, thanks to @homebysix for doing a similar action for linting, ensuring consistent Python style rules are being applied.

Other Changes

• remove incompatibility notice non-macs by @jgstew in #795
• the make_new_release script is a lot easier to use (only really benefits maintainers, but hey)

Full Changelog: v2.5.0...v2.6.0Beta1

AutoPkg 2.5.0 Beta

2.5.0 (April 08, 2022)

FIXES:

• Resolved a bug preventing autopkg repo-update and autopkg repo-delete operations on local file paths (#724; fixes #723 and lindegroup/autopkgr#666)
• Unarchiver now explicitly clears out archive_format variable when finishing execution #777
• Xattr usage on Linux should no longer throw errors #778
• Add a new asset_url output variable to GithubReleasesInfoProvider #779
• Prevent MunkiImporter duplicating imports when multiple architectures. This now lets MunkiImporter correctly support Apple Silicon/Universal versions of existing software in the repo. #781

AutoPkg 2.4.1

The main new feature here is that MunkiImporter now allows importing of multiple "flavors" of a given version of software, differing by what architectures they support. In other words, you can now have a recipe import Foo-2.0 (Intel) and another recipe import Foo-2.0 (Apple Silicon) and MunkiImporter will actually import both.

What's Changed

• Bump pyyaml from 5.3.1 to 5.4 by @dependabot in #730
• AppPkgCreator: New Optional Input: version_key by @primalcurve in #733
• Update FileFinder.py by @macmule in #742
• Allow use of '**' globbing pattern in the Copier Processor. by @Frzk in #711
• Adjust GitHubReleasesInfoProvider to accommodate extra leading dot in version tags by @homebysix in #747
• Update AppPkgCreator.py by @macmule in #755
• Correct typo in CONTRIBUTING.md by @ansara in #762
• Reset archive_format after each Unarchiver process invocation by @homebysix in #777
• Fix for xattr usage on Linux by @gregneagle in #778
• Add asset_url output variable to GitHubReleasesInfoProvider by @haircut in #779
• Prevent MunkiImporter duplicating imports when multiple architectures by @LcTrKiD in #781
• Merging dev into master 3/15/22 by @nmcspadden in #792

New Contributors

• @dependabot made their first contribution in #730
• @primalcurve made their first contribution in #733
• @Frzk made their first contribution in #711
• @ansara made their first contribution in #762
• @haircut made their first contribution in #779

Full Changelog: v2.3.1...v2.4.1

AutoPkg 2.4.1 Beta

Re-release of 2.3.2 beta with one single logic fix preventing the recipe map from generating, and the correct versioning.

What's Changed

• Bump pyyaml from 5.3.1 to 5.4 by @dependabot in #730
• AppPkgCreator: New Optional Input: version_key by @primalcurve in #733
• Update FileFinder.py by @macmule in #742
• Allow use of '**' globbing pattern in the Copier Processor. by @Frzk in #711
• Adjust GitHubReleasesInfoProvider to accommodate extra leading dot in version tags by @homebysix in #747
• Update AppPkgCreator.py by @macmule in #755
• Correct typo in CONTRIBUTING.md by @ansara in #762

New Contributors

• @dependabot made their first contribution in #730
• @primalcurve made their first contribution in #733
• @Frzk made their first contribution in #711
• @ansara made their first contribution in #762

Full Changelog: v2.3.1...v2.4.1-b

What's Changed

• Bump pyyaml from 5.3.1 to 5.4 by @dependabot in #730
• AppPkgCreator: New Optional Input: version_key by @primalcurve in #733
• Update FileFinder.py by @macmule in #742
• Allow use of '**' globbing pattern in the Copier Processor. by @Frzk in #711
• Adjust GitHubReleasesInfoProvider to accommodate extra leading dot in version tags by @homebysix in #747
• Update AppPkgCreator.py by @macmule in #755
• Correct typo in CONTRIBUTING.md by @ansara in #762

New Contributors

• @dependabot made their first contribution in #730
• @primalcurve made their first contribution in #733
• @Frzk made their first contribution in #711
• @ansara made their first contribution in #762

Full Changelog: v2.3.1...v2.4.1-b

AutoPkg 2.3.2 Beta

2.3.2 (December 20, 2021)

I've created a new pre-release version of AutoPkg based on the dev_recipe_map branch. This branch has a radical redesign of the recipe loading logic that dramatically speeds up recipe locating.

Instead of searching for recipes or processors by traversing the file system every time we need to go find something, we instead generate a static map file of all repos and recipes on-disk. This static map (cleverly titled "recipe_map.json") searches the file system during certain operations (adding/removing repos, or adding new recipes, or adding overrides), and stores a list of all recipes in two ways:

1. A list of all identifiers, with the absolute paths to the recipe as the values;
2. A list of all recipe shortnames ("GoogleChrome.download") with the absolute paths to the recipes as values

Whenever. you do any operation in AutoPkg, it consults this recipe map only and knows where to go next. This has reduced load times for running a recipe from multiple seconds (which scales higher based on how many recipes you have on-disk) to fractions of a second.

Graham Pugh provided this sample output to show the very stark difference with this branch:
Master:

% autopkg run -v GoogleChrome.jamf
**load_recipe time: 22.676719694
**verify_parent_trust time: 114.754227371
**process_cli_overrides time: 114.754611623
**verify time: 172.079330759
**process time: 216.01272573499998

With the new dev version:

./autopkg run -v GoogleChrome.jamf
**load_recipe time: 0.03708735200000002
**verify_parent_trust time: 0.696461484
**process_cli_overrides time: 0.696903272
**verify time: 0.711556611
**process time: 44.696209888

22 seconds -> 0.03 seconds to load recipes.

FUTURE DIRECTION
So, what's next for this?

I want to replace just about all Github Search API calls with static maps instead. Rather than relying on the API to give us information sometimes, what if we took this static map idea a step further? The AutoPkg Github Repo itself could store a static mapping of all recipes and all repos across the org, and clients would simply fetch that static map when trying to search for recipes.

To enhance and combine this functionality with the recipe map, we'd also need to change the override to contain a bit more info. Right now, the override just contains the chain of parents and their hashes as was generated at the time, but what if it also referenced the recipe map to also store the list of all things the recipe would ever need to execute successfully? If we had a full repo map as well as a local recipe map, we could easily triangulate exactly where all the resources required to run a recipe are located, and then fetch them if we don't have them. If we store that information in the override itself, then CI environments that are ephemeral would have all the info they need to run any recipe contained within the override itself, rather than having to make a lot of guesses or assumptions.

CONTEXT
Right now, AutoPkg interacts with GitHub in a few ways:

1. You use autopkg search, which generally does what it says;
2. You use autopkg info -p, which tries to search for the parent repos of a recipe and fetch all of them;

The problem is that the Github search API occasionally just.. doesn't. This API is rate limited really heavily, and is especially harsh for large organizations that have lots of outbound traffic from one set of IPs. This means that if a large organization is talking to GitHub API often, you could be rate limited just by sheer volume of traffic. When the API rate limits you, it doesn't return useful results to AutoPkg.

In a CI environment, if you rely on autopkg info -p to automatically pull your repos, this means that occasionally Github just doesn't give you anything. So AutoPkg will fail to pull its parent repos for recipe chains properly, and that means that recipes just occasionally fail for no reason. Trying it again usually just works, without making any changes. At Facebook/Meta, where I use this feature heavily, I see this very frequently.

So frequently, in fact, that it actually reduces the overall reliability of the automatic parent fetching feature.

I still believe that its intended goal is a good one: to avoid having to maintain a hardcoded list of repos to check out in a CI environment (or any environment). I think it makes sense for Autopkg to be able to dynamically figure out what repos you need, and then go get them. The evidence simply shows that we just can't rely on Github's API search for that.

I'd love to hear people's thoughts and see people's test results.

AutoPkg 2.3.1

2.3.1 (March 03, 2021)

FIXES:

• Resolved a bug preventing autopkg repo-update and autopkg repo-delete operations on local file paths (#724; fixes #723 and lindegroup/autopkgr#666)

AutoPkg 2.3

2.3 (March 01, 2021)

NEW FEATURES:

AutoPkg now supports recipes in yaml format (#698, example recipe). Yaml recipes tend to be more human-readable than plist recipes, especially for those who don't work with plists on a daily basis.

AutoPkg can produce new recipes in yaml format using autopkg new-recipe SomeCoolApp.pkg.recipe.yaml and make overrides in yaml format using autopkg make-override --format=yaml SomeCoolApp.pkg. Searching for public yaml recipes on GitHub is also possible using autopkg search.

NOTES FOR RECIPE AUTHORS:

• Because yaml recipes will require AutoPkg 2.3 or later in order to run, and because some members of the AutoPkg community may still be using AutoPkg 1.x, recipe authors are encouraged to be conservative and keep existing public recipes in their current format for a while.
• If you have both plist and yaml recipes for the same app in your repo, you may experience unexpected behavior now that AutoPkg detects and uses yaml recipes.

OTHER CHANGES FROM 2.2:

• Added support for internal GitHub URLs (#649)
• autopkg make-override no longer creates override for deprecated recipes by default (#685)
• Typo fixed in the recipe template created by autopkg new-recipe
• Fixed a bug causing autopkg repo-add and autopkg repo-delete to fail for repos in GitHub organizations with non-alphanumeric characters in their names (#712, #715)
• CodeSignatureVerifier warns when certain incorrect input variables are detected
• MunkiImporter now uses consistent pkginfo matching logic (#671)
• Minor edits to help text
• Improvements to Versioner processor (#600)
• Help is now shown for autopkg list-processors --help, matching behavior of most other verbs (#717)
• The output of autopkg list-recipes --plist is now text instead of binary (this matches previous behavior in AutoPkg 1.x)
• More output when using autopkg repo-add and autopkg repo-delete (#704)
• Fixed a bug in MunkiImporter that caused incorrect uninstaller_item_location path (#702)
• Building a foundation for long term expansion of platform support (#648, #651, #653, #656, #658, #666, #670)

KNOWN ISSUES:

• #710 is currently affecting some autopkg search results (regardless of whether the recipes are plist or yaml)