More "native order" decoders?

[Zahariel]

TL;DR: can we add some more "native byte order" raw unpacking modes other than I:32N, and/or more "predictable byte order" raw packing modes (which I;32S isn't)?

I'm having a problem specifically coming from pycairo. Specifically, pycairo's FORMAT_ARGB32 format, which is "documented" (I use the term somewhat loosely) as storing each pixel in ARGB, with premultiplied alpha, in "native order". On a little-endian machine (i.e. almost everywhere), this means that the data comes out of surface.get_data() in what Pillow would call BGRa. But on a big-endian machine, if you can find such a creature, it comes out as aRGB. I would like to be able to convert this, portably, to a more predictable format, for example to display in pygame (please just ignore my Rube Goldberg display system here).

I eventually constructed the formulation:

buf = cairo_surface.get_data() # either aRGB or BGRa
pil_image = Image.frombuffer("I", size, bytes(buf), "raw", "I;32B", 0, 1)
fixed_buf = pil_image.tobytes("raw", "I;32S") # implied "native"; definitely aRGB
pygame_image = pygame.image.frombuffer(fixed_buf2, size, "ARGB")
pygame_screen.blit(pygame_image, (0,0), special_flags=pygame.BLEND_PREMULTIPLIED)

Which gets the job done, technically, but really isn't clear at all, especially since I really wanted to read I;32N and output I;32B; but no such packing mode exists, so I swapped them. But what I really, actually wanted, was a notional aRGB;N unpacking mode that means "the 32-bit values are aRGB, but they're in machine-native order, so it's probably actually BGRa; you figure it out". In fact, even unpacking aRGB as RGBA (which is what pygame really wants, although modern versions are willing to put up with premultiplied alpha) doesn't exist; only BGRa -> RGBA works, which I could do with a 2-step process. It looks quite silly, but it does work.

buf = cairo_surface.get_data() # either aRGB or BGRa
pil_image = Image.frombuffer("I", size, bytes(buf), "raw", "I;32", 0, 1) # now using little-endian
fixed_buf = pil_image.tobytes("raw", "I;32S") # implied "native"; definitely BGRa

pil_image2 = Image.frombuffer("RGBA", size, fixed_buf, "raw", "BGRa", 0, 1)
fixed_buf2 = pil_image2.tobytes("raw", "RGBA")

pygame_image = pygame.image.frombuffer(fixed_buf2, size, "RGBA")
pygame_screen.blit(pygame_image, (0,0))

(What I really, truly, actually wanted was for cairo's get_data() to do something more predictable, but that ship has most likely sailed.)

So my questions here are:

1. is there a better way for me to do this? Have I just hacked myself into a ridiculous corner and I should really do this totally differently?
2. would it be plausible to add my notional aRGB;N and similar machine-order-based raw unpacking modes?
3. additionally, you may have noted my use of bytes(buf) creating a Pillow image from a pycairo surface. surface.get_data() returns a memoryview. I would think this is bytes-like enough for Pillow's use? But it doesn't work.

[radarhere]

1. is there a better way for me to do this? Have I just hacked myself into a ridiculous corner and I should really do this totally differently?

Why isn't https://pycairo.readthedocs.io/en/latest/integration.html#pillow-pil-cairo sufficient for your purposes?

Or, if you're only using Pillow as a stepping stone, https://pycairo.readthedocs.io/en/latest/tutorial/pygame.html#pygame-imagesurface seems like it would solve your problem.

[Zahariel]

Your second suggestion is what we had in the first place, and it doesn't actually work (or at least it didn't when I asked this question a couple years ago), no matter how much the documentation says it should; the red and blue channels come out swapped. That's what I was trying to fix.

Your first suggestion is going in the wrong direction. Cairo's import utilities are correctly implemented and predictable; it's the export from cairo that doesn't work correctly.

[radarhere]

Direct to Pygame: This might be in progress with pygame/pygame#3342, coming from pygame/pygame#2972 and pygobject/pycairo#247

[radarhere]

The pygame PR has been merged, so directly converting from pycairo to pygame should now be possible with pygame main.

[Zahariel]

Nice! I'm not running pygame:main, but hopefully 2.1.3 will be released sometime soon.

[radarhere]

pygame 2.1.3 has now been released.

[radarhere]

3. additionally, you may have noted my use of bytes(buf) creating a Pillow image from a pycairo surface. surface.get_data() returns a memoryview. I would think this is bytes-like enough for Pillow's use? But it doesn't work.

With PR #6974, this should work with Pillow 9.5.0.

[radarhere]

2. would it be plausible to add my notional aRGB;N and similar machine-order-based raw unpacking modes?

The closest we have at the moment is "RGBa;16N".

However, looking at the code, it becomes clear that the channel order isn't swapped depending on the endianness, simply the values of each channel individually.

Pillow/src/libImaging/Unpack.c

Lines 1619 to 1634 in e7fa309

	#ifdef WORDS_BIGENDIAN
	{"RGB", "RGB;16N", 48, unpackRGB16B},
	{"RGBA", "RGBa;16N", 64, unpackRGBa16B},
	{"RGBA", "RGBA;16N", 64, unpackRGBA16B},
	{"RGBX", "RGBX;16N", 64, unpackRGBA16B},
	{"RGB", "R;16N", 16, band016B},
	{"RGB", "G;16N", 16, band116B},
	{"RGB", "B;16N", 16, band216B},
	{"RGBA", "R;16N", 16, band016B},
	{"RGBA", "G;16N", 16, band116B},
	{"RGBA", "B;16N", 16, band216B},
	{"RGBA", "A;16N", 16, band316B},
	#else
	{"RGB", "RGB;16N", 48, unpackRGB16L},
	{"RGBA", "RGBa;16N", 64, unpackRGBa16L},

Pillow/src/libImaging/Unpack.c

Lines 842 to 888 in e7fa309

	static void
	unpackRGBa16L(UINT8 *_out, const UINT8 *in, int pixels) {
	int i;
	/* premultiplied 16-bit RGBA, little-endian */
	for (i = 0; i < pixels; i++) {
	int a = in[7];
	UINT32 iv;
	if (!a) {
	iv = 0;
	} else if (a == 255) {
	iv = MAKE_UINT32(in[1], in[3], in[5], a);
	} else {
	iv = MAKE_UINT32(
	CLIP8(in[1] * 255 / a),
	CLIP8(in[3] * 255 / a),
	CLIP8(in[5] * 255 / a),
	a);
	}
	memcpy(_out, &iv, sizeof(iv));
	in += 8;
	_out += 4;
	}
	}
	static void
	unpackRGBa16B(UINT8 *_out, const UINT8 *in, int pixels) {
	int i;
	/* premultiplied 16-bit RGBA, big-endian */
	for (i = 0; i < pixels; i++) {
	int a = in[6];
	UINT32 iv;
	if (!a) {
	iv = 0;
	} else if (a == 255) {
	iv = MAKE_UINT32(in[0], in[2], in[4], a);
	} else {
	iv = MAKE_UINT32(
	CLIP8(in[0] * 255 / a),
	CLIP8(in[2] * 255 / a),
	CLIP8(in[4] * 255 / a),
	a);
	}
	memcpy(_out, &iv, sizeof(iv));
	in += 8;
	_out += 4;
	}
	}

So I'd be reluctant to introduce confusion into the process by introducing this new kind of unpacker.

I think that answers all of your questions. Let us know if not.