change default dmg format and allow modern compression types

[erikng]

• UDZO compression has been marked as deprecated for years now
• UDLO compression is the "new" default and supports macOS 10.11 and higher
• UDMO compression is the "newest" and supports macOS 10.15 and higher

I would personally prefer UDMO as the default DMG format, but there is a significant performance penalty when creating the DMGs. I have only tested the performance on my fully managed macbook pro from 2019, but it was probably a 3x difference in the autopkg runs.

Example:

• Xcode 15
  • 3.18 GB Xcode_15.xip
  • 11.46 GB Xcode.app
  • 4.51 GB Xcode-UDZO.dmg (HFS+)
  • 4.41 GB Xcode-UDLO.dmg (HFS+)
  • 4.09 GB Xcode-UDMO.dmg (HFS+)
  • 4.08 GB Xcode-UDMO.dmg (APFS)

I'm wondering if there is more compression work that can be done here to get similar compression ratios to the original xip format that Apple utilizes.

from the manpage

COMPATIBILITY
     OS X 10.11 introduced lzfse compression in the ULFO format, providing faster, more efficient
     compression and smaller images compared to UDZO.  These images are also supported in-kernel, but will
     not work on any earlier versions of the OS.

     macOS 10.12 included a pre-release version of the Apple File System called APFS which was meant for
     evaluation and development purposes only.  Files stored in APFS-based images may not be accessible in
     future releases of macOS, and won't work in past ones.  All data to be stored in APFS volumes should
     be backed up prior to using APFS and regularly backed up while using APFS.

     macOS 10.15:

           •   Introduced lzma compression in the ULMO format, providing smaller images compared to ULFO.
               These images are not supported in-kernel, and will not work on any earlier versions of the
               OS.

           •   Deprecated OS 9-style dual-fork file support (hdiutil flatten/unflatten).

           •   Removed the deprecated "hdiutil internet-enable" command and the IDME attach flags.

     macOS 11.0:

           •   Removed support for DiskCopy42, DART and NDIF formats.

           •   Removed support for AppleSingle and MacBinary encodings.

           •   Removed the deprecated OS 9-style dual-fork file support (hdiutil flatten/unflatten).

           •   Default file system for new images has changed to APFS (instead of an empty disk image with
               no partition map). To create an empty disk image add "-layout NONE" to the creation flags.
               This change does not apply to images created with -srcfolder or -srcdevice arguments.

     macOS 12.0:

           •   Deprecated UDBZ format (bzip2 compression).

           •   Deprecated segmented UDIF images (hdiutil segment, -segmentSize argument in hdiutil create &
               convert).

           •   Deprecated hdiutil udifrez/udifderez (embed and extract resources).

     macOS 13.0:

           •   Removed the option "encrypted-encoding-version". All new encrypted images are created with
               encrypted encoding version 2.
               
WHAT'S NEW
     macOS 10.15 added ULMO format images compressed with lzma.  These images are smaller than comparable
     ULFO images compressed with lzfse.  These images are not supported in-kernel, and are not usable on
     earlier OSes.

     macOS 10.12 introduced the pre-release APFS for evaluation (see COMPATIBILITY above).  10.12 also
     added an option to disable atomic copying during image from folder operations, -noatomic, which may
     result in slightly faster image creation.  pmap added a new switch, -diagnostic, which captures
     troubleshooting information for Boot Camp configurations.

     OS X 10.11 added ULFO format images compressed with lzfse.  These images are more efficient and
     smaller than comparable UDZO images compressed with zlib, and retain kernel compatibility, but are not
     usable on earlier OSes.

[erikng]

Some further context as to why this would be useful. GitHub Enterprise prohibits git-lfs objects larger than 5gb and charges for ingress/egress usage.

By utilizing better compression methods, companies can reduce their GitHub costs and also decrease the risk of autopkg binaries not being importable.

[erikng]

Some interesting things, but I've found an even better compression when converting to a pkg vs a dmg.

pkgbuild --component /autopkg/Cache/local.munki.Xcode15/Xcode.app --min-os-version 13 --compression latest /autopkg/Cache/local.munki.Xcode15/Xcode_15.0.0.pkg

This results in a 3.23gb pkg.

[jgstew]

I wanted to add that the automated tests for this PR passed on Ubuntu but failed on MacOS due to GitHub API limits exceeded which is not the fault of the PR itself, but is the fault of the fact that the github token currently isn't getting picked up by autopkg like it should at the moment.

[jgstew]

I think adding to the list of acceptable formats is obvious and should happen.

I don't know if changing the default format for everyone makes as much obvious sense, but I also don't use AutoPkg for MacOS software currently, so it is not something I should have much opinion on.

Would this mean the new defaults would cause a failure for anyone not running at least MacOS 13 Ventura if they do not change the new defaults? Or what versions of MacOS would it cut off?

[erikng]

No. I'd love to do that but I opted for the 10.15 and higher version. If people are still running autopkg on insecure versions of macOS that no longer receive security updates I would question their commitment to their job for critical infrastructure like autopkg.

[gregneagle]

I think this is a good idea. My vote for the new default compression format is UDLO, though maybe if I had more performance impact data I'd support UDMO.