Files with shared auxiliary images fail to parse

[leo-barnes]

Filing issue to document email thread with @farindk.

ImageMagick is rejecting some prototype files that I have been creating with the following error:
identify: Invalid input: Unspecified: Too many auxiliary image references (2.0) `file_with_shared_alpha.heic' @ error/heic.c/IsHEIFSuccess/142.

The error is coming from libheif rather than ImageMagick itself. The file in question has two main images (in an 'altr' group) that share an alpha auxiliary image. To me this should be perfectly valid and makes a lot of sense once you start creating files containing 'altr' groups, but I can see why this would be unexpected for simpler files.

The conclusion from the thread is that there is nothing in the spec that explicitly forbids sharing the same auxiliary image between multiple main images and that the restriction in libheif should probably be relaxed. ISOBMFF says that all irefs of a specific type from a source item should be collected into a single ItemReferenceBox, so having multiple irefs is not really an option.
file_with_shared_alpha.heic.zip

[bradh]

It looks like we just need to add the auxiliary image to each of the parents. Might be some complexity with loop detection, but will check that.

One question before I look to PR this: would it make more sense to have the altr group (in this test file, item 4) as the target? Is that at least a possibility that should be covered / supported?

[leo-barnes]

@bradh

would it make more sense to have the altr group (in this test file, item 4) as the target? Is that at least a possibility that should be covered / supported?

Good question. It would only be possible under the unif brand, at which point it falls under the fuzziness of if the semantics of the reference permit it.

While I think the intent of the writer is relatively clear if you let the auxl point to the altr group, I think it's much clearer to let the auxiliary image reference the main image items directly. This avoids all the ambiguity that you get with the unif brand (and works even for parsers that don't implement groups).

I think it's fine to support reading files that have an auxl reference to an altr group, but we should probably discourage such files from being created.

[leo-barnes]

The same issue might also be present for shared metadata and shared thumbnails. Creating a file with shared EXIF payloads seems to result in the following error from ImageMagick:
identify: Invalid input: Unspecified: Metadata not correctly assigned to image (2.0)
(Assuming I didn't mess up creating the file)

[bradh]

@leo-barnes Can you share those additional test files?

[leo-barnes]

Attached a file that shares alpha, thumbnail and XMP between the two main images in an altr group.
file_with_shared_everything.heic.zip

[bradh]

I've got it to a "less broken" stage:

$ ./build/examples/heif-info ./file_with_shared_everything.heic 
MIME type: image/heic
main brand: heic
compatible brands: heic, mif1

image: 512x384 (id=1), primary
  colorspace: YCbCr, 4:2:0
  bit depth: 8
  thumbnail: 128x96
  color profile: no
  alpha channel: yes 
  depth channel: no
metadata:
  XMP: 356 bytes
transformations:
  none
region annotations:
  none
properties:

image: 1024x768 (id=3)
  colorspace: YCbCr, 4:2:0
  bit depth: 8
  thumbnail: 128x96
  color profile: no
  alpha channel: no 
  depth channel: no
metadata:
  XMP: 356 bytes
transformations:
  none
region annotations:
  none
properties:

and ImageMagick is more on-board with this:

$ identify -verbose ./file_with_shared_everything.heic 
Image:
  Filename: ./file_with_shared_everything.heic
  Format: HEIC (Apple High efficiency Image Format)
  Mime type: image/x-heic
  Class: DirectClass
  Geometry: 512x384+0+0
  Units: Undefined
  Colorspace: YCbCr
  Type: TrueColor
  Base type: Undefined
  Endianness: Undefined
  Depth: 8-bit
  Channel depth:
    red: 8-bit
    green: 8-bit
    blue: 8-bit
  Channel statistics:
    Pixels: 196608
    Red:
      min: 0  (0)
      max: 255 (1)
      mean: 107.035 (0.419743)
      standard deviation: 55.2343 (0.216605)
      kurtosis: -0.385301
      skewness: 0.357236
      entropy: 0.969307
    Green:
      min: 33  (0.129412)
      max: 154 (0.603922)
      mean: 115.458 (0.452778)
      standard deviation: 12.9742 (0.0508791)
      kurtosis: 1.90072
      skewness: -1.07821
      entropy: 0.822435
    Blue:
      min: 103  (0.403922)
      max: 169 (0.662745)
      mean: 125.525 (0.492253)
      standard deviation: 5.94608 (0.023318)
      kurtosis: 2.07753
      skewness: 0.88771
      entropy: 0.748516
  Image statistics:
    Overall:
      min: 0  (0)
      max: 255 (1)
      mean: 116.006 (0.454925)
      standard deviation: 24.7182 (0.0969341)
      kurtosis: 3.09398
      skewness: -0.200445
      entropy: 0.846753
  Rendering intent: Perceptual
  Gamma: 0.454545
  Chromaticity:
    red primary: (0.64,0.33)
    green primary: (0.3,0.6)
    blue primary: (0.15,0.06)
    white point: (0.3127,0.329)
  Background color: ycbcr(255,255,255)
  Border color: ycbcr(223,223,223)
  Matte color: ycbcr(189,189,189)
  Transparent color: ycbcr(0,0,0)
  Interlace: None
  Intensity: Undefined
  Compose: Over
  Page geometry: 512x384+0+0
  Dispose: Undefined
  Iterations: 0
  Compression: Undefined
  Orientation: TopLeft
  Properties:
    date:create: 2024-03-23T09:25:46+00:00
    date:modify: 2024-03-20T11:58:28+00:00
    signature: 92c65b443b025008d972d4a002b234cc750bcc5535bfc2f73c99def327e5c49a
  Artifacts:
    filename: ./file_with_shared_everything.heic
    verbose: true
  Tainted: False
  Filesize: 0B
  Number pixels: 196608
  Pixels per second: 5.75156MB
  User time: 0.040u
  Elapsed time: 0:01.034
  Version: ImageMagick 6.9.11-60 Q16 x86_64 2021-01-25 https://imagemagick.org

The alpha still isn't properly assigned to the second image. Need to go again on the scaling.

[bradh]

Have updated the implementation.

./examples/heif-info ../file_with_shared_everything.heic
MIME type: image/heic
main brand: heic
compatible brands: heic, mif1

image: 512x384 (id=1), primary
  colorspace: YCbCr, 4:2:0
  bit depth: 8
  thumbnail: 128x96
  color profile: no
  alpha channel: yes 
  depth channel: no
metadata:
  XMP: 356 bytes
transformations:
  none
region annotations:
  none
properties:

image: 1024x768 (id=3)
  colorspace: YCbCr, 4:2:0
  bit depth: 8
  thumbnail: 128x96
  color profile: no
  alpha channel: yes 
  depth channel: no
metadata:
  XMP: 356 bytes
transformations:
  none
region annotations:
  none
properties: