AutoPkg 2.3 RC 1

This is the release candidate for AutoPkg 2.3.

NEW FEATURES:

AutoPkg now supports recipes in yaml format (#698). Yaml recipes tend to be more human-readable than plist recipes, especially for those who don't work with plists on a daily basis.

AutoPkg can produce new recipes in yaml format using autopkg new-recipe SomeCoolApp.pkg.recipe.yaml and make overrides in yaml format using autopkg make-override --format=yaml SomeCoolApp.pkg. Searching for public yaml recipes on GitHub is also possible using autopkg search.

NOTES FOR RECIPE AUTHORS:

• Because yaml recipes will require AutoPkg 2.3 or later in order to run, and because some members of the AutoPkg community may still be using AutoPkg 1.x, recipe authors are encouraged to be conservative and keep existing public recipes in their current format for a while.
• If you have both plist and yaml recipes for the same app in your repo, you may experience unexpected behavior now that AutoPkg detects and uses yaml recipes.

OTHER CHANGES FROM 2.2:

• Added support for internal GitHub URLs (#649)
• autopkg make-override no longer creates override for deprecated recipes by default (#685)
• Typo fixed in the recipe template created by autopkg new-recipe
• Fixed a bug causing autopkg repo-add and autopkg repo-delete to fail for repos in GitHub organizations with non-alphanumeric characters in their names (#712, #715)
• CodeSignatureVerifier warns when certain incorrect input variables are detected
• MunkiImporter now uses consistent pkginfo matching logic (#671)
• Minor edits to help text
• Improvements to Versioner processor (#600)
• Help is now shown for autopkg list-processors --help, matching behavior of most other verbs (#717)
• The output of autopkg list-recipes --plist is now text instead of binary (this matches previous behavior in AutoPkg 1.x)
• More output when using autopkg repo-add and autopkg repo-delete (#704)
• Fixed a bug in MunkiImporter that caused incorrect uninstaller_item_location path (#702)
• Building a foundation for long term expansion of platform support (#648, #651, #653, #656, #658, #666, #670)

KNOWN ISSUES:

• #710 is currently affecting some autopkg search results (regardless of whether the recipes are plist or yaml)

AutoPkg 2.2

2.2 (August 24, 2020)

NEW FEATURES
MunkiImporter now supports Munki repo plugins, thanks to @tboyko. The default behavior
is still to use FileRepo as the default local behavior, so existing behavior is
unchanged. (#654)

CHANGES FROM 2.1:

• URLDownloader handles Content-Disposition filenames with quotes correctly (#633)
• README and CONTRIBUTING guides updated with correct Python 3 framework info (#638)
• PyYAML updated to 5.3.1 to address PyYAML-CVE-2020-1747 (#642)
• Internal autopkg code structure is being shifted, with more code moving into separate
  shared modules. Lots of various lint fixes, formatting, and safety handling improvements.
• GitHub API queries no longer fail when searching for recipes with spaces in the name,
  and are now quoted correctly (#664)
• Processor subclasses now automatically configure an empty dictionary for self.env if
  none is provided. This doesn't have any practical effect, but makes it easier to create
  and use new Processor subclasses in the future.

AutoPkg 2.1

2.1 (May 19, 2020)

NEW FEATURES
AutoPkg now supports the verbs list-repos and processor-list for convenience (#628)

autopkg info --pull/-p now allows you to fetch all parent repos of a recipe
automatically.

Example:

$ autopkg repo-delete recipes
$ autopkg info -p GoogleChrome.munki
Didn't find a recipe for com.github.autopkg.munki.google-chrome.
Found this recipe in repository: recipes
Attempting git clone...

Adding /Users/nmcspadden/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes to RECIPE_SEARCH_DIRS...
Updated search path:
  '.'
  '~/Library/AutoPkg/Recipes'
  '/Library/AutoPkg/Recipes'
  '/Users/nmcspadden/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes'

Description:         Downloads the latest Google Chrome disk image and imports into Munki.
Identifier:          local.munki.GoogleChrome
Munki import recipe: True
Has check phase:     True
Builds package:      False
Recipe file path:    /Users/nmcspadden/Library/AutoPkg/RecipeOverrides/GoogleChrome.munki.recipe
Parent recipe(s):    /Users/nmcspadden/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes/GoogleChrome/GoogleChrome.munki.recipe
                     /Users/nmcspadden/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes/GoogleChrome/GoogleChrome.download.recipe

The automatic fetching works by looking at the parent identifier of a recipe, and
searching GitHub via API for that file. It fetches that parent file from GitHub
directly, and adds the repo that it belongs to. Then it parses its parent, recursively
until it finds a recipe with no parents.

Note that the only verb to support this is autopkg info. You can use this feature to
dynamically fetch parents on-demand, instead of preconfiguring your environment with a
list of known repos.

CHANGES FROM 2.0.2:

• URLGetter can handle parsing headers without an explicit url in the environment (#605)
• FileCreator now has a unit test (#591)
• AutoPkg warns you more helpfully if you are trying to run it with Python 2 (#610)
• If a recipe generates a Python stacktrace, the traceback output is only provided with verbosity > 2 (#609)
• CodeSignatureVerifier warns you if you attempt to use the deprecated expected_authorities argument (1a3481f)
• Installing packages with AutoPkg .install recipes should no longer generate warnings about failing to close the socket (09a5f5c)
• Updated AppDmgVersioner's description to clarify its limitations (ababfd3)
• Processors can now be run standalone again by accepting variables from a plist read from stdin (#621)
• FileFinder handles recursive searching correctly (#622)
• URLGetter has better error handling (#629)
• Fetching a filename with URLGetter now works more reliably (6d2b941)

AutoPkg 2.0.2

2.0.2 (February 05, 2020)

AutoPkg 2.0 Release

PYTHON 3

This is the official release of a Python 3-only version of AutoPkg. It is no longer compatible with Python 2, and will encounter syntax errors and failures if ran with Python 2.

The release package has an included Python 3 framework that includes all necessary modules to run everything in AutoPkg core, and all of the recipes in autopkg-recipes.

MAJOR HIGHLIGHTS OF THE PYTHON 3 CODE:

• FoundationPlist has been retired. plistlib in Python 3 should be used to handle all plist parsing.
• All Python string interpolation now use Python 3-style f-strings (formatted string literals).
• All references to unicode vs. string types have been refactored to use Python 3's native byte strings whenever possible.
• All unit tests were updated to Python 3.
• All Python code now use a symlink to the embedded Python framework. It is strongly recommended you do not modify or touch anything inside the AutoPkg Python framework (i.e. avoid installing things with pip).

CHANGES FROM RC2:

• Fixed an encoding bug in the make_new_release script (fca4052)
• Removed some orphaned dead code (c90e92b)

AutoPkg 2.0.1 RC 2

2.0 RC2 (January 31, 2020)

CHANGES FROM RC1:

• Fixed some processor docs (3812ca1)
• Added in "APLooseVersion", loosely based on Munki's version comparison, to replace previous version comparison semantics. (7c0676f)
  • This MAY cause a change in behavior for some current version comparisons, but it no longer crashes when comparing
    certain combinations of strings.

AutoPkg 2.0 RC 1

ANNOUNCEMENT:

This is the release candidate for AutoPkg 2.0.

This is the best time and way to test if your recipe list works in Python 3. Barring the discovery of serious problems, we expect to release AutoPkg 2.0 next week.

DEVELOPMENT:

The previous branch for AutoPkg development, "py2-to-3", has been merged into Master.

AutoPkg 1.x has been cut into a new branch ("AutoPkg_1.x"). Short of an emergency scenario, development on AutoPkg 1.x is frozen at the release of 1.4.1.

CHANGES FROM BETA 3:

• Some fixes around URLGetter's behavior and callsites
• URLGetter.execute_curl() was changed to use subprocess.run() instead of Popen (facad8c)
• Update URLDownloader variable descriptions to show up better on the wiki (079c606)
• URLGetter outputs the entire curl command when used with -vvvv verbosity (7c24a05)
• URLGetter now has a convenient download_to_file(url, filename) function, which makes
  it simple to download a file in a custom processor (this was backported to 1.4.1) (2fac695)
• Fixed an extraneous socket close error in InstallFromDmg (hat tip to Allister B) (c31789f)
• make_new_release.py produces more friendly console output indicating what stage it's on (1373b31)

AutoPkg 1.4.1

1.4.1 (December 2, 2019)

FIXES:

• URLGetter now has a download_to_file(url, filename) function that can be used in
  custom processors. It simply downloads a URL to a specific filename, and raises a
  ProcessorError if it fails for any reason.

AutoPkg 2.0 Beta 3

PYTHON 3 ONLY RELEASE

This is the second beta release of a Python 3-only version of AutoPkg. It is no longer compatible with Python 2, and will encounter syntax errors and failures if run with Python 2.

The release package has an included Python 3 framework that includes all necessary modules to run everything in AutoPkg core, and all of the recipes in autopkg-recipes.

This release also coincides with the most recent commit to
autopkg/recipes, so both change notes will be provided here.

CHANGES FROM BETA 2:

• Thanks to @MichalMMac's heroic efforts, URLGetter is now much easier for other processors to use. There are now two ways a custom processor can download things without needing to write any urllib logic:
  • URLGetter.download_with_curl(curl_command, text=True) takes a curl command as an argument (a list of strings that is passed to subprocess). You can use this along with the other helper functions to arrange your own curl command with custom headers and arguments, and parse the output.
  • URLGetter.download(url, headers=None, text=False) takes a URL (and optional headers) and returns the output of the curl command. You can use this to simply retrieve the results of requesting a web page (such as for URLTextSearcher).
    *In both cases, you can pass text mode to determine if you get straight text output.
• All custom processors that need to make a web request of any kind in autopkg/recipes have been switched to using URLGetter's methods. No more urllib in processors!
• Some minor formatting changes in the code itself

CHANGES FROM BETA 1:

• The new URLGetter base Processor class has been merged in. It provides a new centralized way to handle fetching and downloading things from the web. In the future, more convenience functions will be added to allow any custom processor to easily fetch web resources without having to write their own urllib/web-handling code.
• Failing to import a processor due to a Python syntax error (such as due to py2 vs. py3 imports) now has a more specific and clear error message (e52ae69)
• Many, many, many bytes vs. string issues resolved in core processors
• Copier now has a unit test, and produces some more useful output
• autopkgserver shouldn't complain about socket descriptors anymore
• isort now has a seed config that explicitly lists certain third party modules so that they're sorted at the top or bottom of import blocks correctly
• All custom processors in autopkg-recipes now use certifi to set the base SSL certs so that urllib web requests work; this will be removed in the future once URLGetter's convenience functions are written

CHANGES IN INITIAL 2.0 RELEASE:

• FoundationPlist has been retired. plistlib in Python 3 should be used to handle all plist parsing.
• All Python string interpolation should prefer the use of f-strings (formatted string literals).
• All references to unicode vs. string types have been refactored to use Python 3's native byte strings whenever possible.
• All unit tests were updated to Python 3.
• All Python code now use a hardcoded path to the embedded Python framework. This path may change at a later time to incorporate a symlink, for easier cross-platform compatibility.

KNOWN ISSUES:

• There are likely still edge cases in the autopkg/recipes that slipped through testing, so please file issues if you find recipes that don't work as intended.

HOW TO USE THIS BETA RELEASE:

Installing the release package will get you everything you need to run AutoPkg 2.0.

As of this release, all recipes in autopkg/recipes work directly (with commit autopkg/recipes@25518b9).

You will need to update your trust info as many custom processors have changed:

autopkg update-trust-info Firefox.munki
autopkg update-trust-info MakeCatalogs.munki
autopkg run -vv Firefox.munki MakeCatalogs.munki

HOW TO REPORT ISSUES:
Use the "Beta Bug report" GitHub issue template to specifically label the issue as being
beta only. Please make use of the template to convey all information possible in order
to reproduce or diagnose the issue as clearly as possible.

SETTING UP AUTOPKG MANUALLY:
If you do not want to use the AutoPkg release installer, you can manually set up an
AutoPkg 2.0 environment. Setup and place the AutoPkg files as you normally would:

1. Create /Library/AutoPkg/
2. Copy the contents of Code into /Library/AutoPkg/
3. Ensure correct file modes for the autopkgserver components: sudo chmod -R 755 /Library/AutoPkg/autopkgserver/

Build a relocatable python bundle:

1. Use the CONTRIBUTING guide's instructions on building a relocatable python bundle
   that uses the requirements.txt file for pip
2. Move/copy the bundle into /Library/AutoPkg/Python3/Python.framework

AutoPkg 1.4

1.4 (November 25, 2019)

FIXES:

• DmgMounter now correctly handles APFS disk images, especially with EULAs/SLAs (4b77f6d)

ADDITIONS:

• The new URLGetter base Processor class has been merged in. It provides a new centralized way to handle fetching and downloading things from the web. In the future, more convenience functions will be added to allow any custom processor to easily fetch web resources without having to write their own urllib/web-handling code.
• Thanks to @MichalMMac's heroic efforts, URLGetter is now much easier for other processors to use. There are now two ways a custom processor can download things without needing to write any urllib logic:
  • URLGetter.download_with_curl(curl_command,) takes a curl command as an argument (a list of strings that is passed to subprocess). You can use this along with the other helper functions to arrange your own curl command with custom headers and arguments, and parse the output.
  • URLGetter.download(url, headers=None) takes a URL (and optional headers) and returns the output of the curl command. You can use this to simply retrieve the results of requesting a web page (such as for URLTextSearcher).
• All custom processors that need to make a web request of any kind in autopkg/recipes have been switched to using URLGetter's methods. No more urllib in processors!
• URLDownloader now has a new prefetch_filename option that will allow fetching the original name from the redirected URL (#571)
• GitHub calls now successfully function through proxies (#416)

AutoPkg 1.3.1

1.3.1 (November 06, 2019)

FIXES:

• Nested data structures in preferences, such as with JSS_REPOS, should no longer
  cause AutoPkg to fail (1aff762)