How and where does Pillow render successive gif frames on top of one another?

[RLaursen]

When handling a gif file, Pillow does something a little bit weird. Instead of retaining transparency of frames so they may be displayed on top of one another, it stacks the next frame on the previous one and then flattens them. I assume this is because Pillow is designed to support conversion to other formats, so it attempts to ensure gifs will be rendered as they would be displayed without relying on them being re-encoded as gifs.

If the two frames have different color tables and transparency, this may lead to errors.

I have been trying to figure out where this behavior actually occurs. I cannot find it. Where exactly is each frame flattened onto the previous frames in the gif? It apparently happens after GifImageFile._seek because disposal methods 2+ may cover previous frames. I have stepped through the process of loading/saving a gif a few times with a debugger, read much of the C involved, and and failed to find where this behavior occurs.

Example of the behavior:

Here is a gif with disposal methods of 2 for every frame and a background index the same as its transparency index, it has consistent transparency:

Here it is after Pillow has loaded and saved it:

# code used to save this gif
    f = Image.open('disposaltest.gif')
    f.save('r.gif', save_all=True)

Pillow stacks the image on top of the previous frame, which may have a different local color table. In the first frames, the color at 0x00 is different.... Which produces this error.

Again, where in the code is this stacking happening? I want to try to fix it, if only for myself.

(In addition, Pillow also alters the gif's transparent color index, which leads to another error, but this is getting long already...)

Here is the relevant data from that gif, I wrote a handy utility to make figuring this bug out easier:

FRAME 0 
Local table 0x00: e6e2e1
{'Extension Introducer': '0x21', 'Graphic Control Label': '0xf9', 'Byte Size': '0x04', 'Packed Field': {'Reserved for Future Use': '000', 'Disposal Method': '000', 'User Input Flag': '0', 'Transparent Color Flag': '1'}, 'Delay Time': 11, 'Transparent Color Index': '0x00', 'Block Terminator': '0x00'}
{'Image Seperator': '0x2c', 'Image Left': 1, 'Image Top': 62, 'Image Width': 380, 'Image Height': 407, 'Packed Field': {'Local Color Table': '1', 'Interlace Flag': '0', 'Sort Flag': '0', 'Reserved For Future Use': '00', 'Size of Local Color Table': '111'}}
{'Canvas Width': 385, 'Canvas Height': 480, 'Packed Field': {'Global Color Flag': '1', 'Color Resolution': '111', 'Sort Flag': '0', 'Size of Global Color Table': '111'}, 'Background Color Index': '0x00', 'Pixel Aspect Ratio': '0x00'}
FRAME 1 
Local table 0x00: 000000
{'Extension Introducer': '0x21', 'Graphic Control Label': '0xf9', 'Byte Size': '0x04', 'Packed Field': {'Reserved for Future Use': '000', 'Disposal Method': '000', 'User Input Flag': '0', 'Transparent Color Flag': '1'}, 'Delay Time': 11, 'Transparent Color Index': '0x00', 'Block Terminator': '0x00'}
{'Image Seperator': '0x2c', 'Image Left': 41, 'Image Top': 59, 'Image Width': 312, 'Image Height': 406, 'Packed Field': {'Local Color Table': '1', 'Interlace Flag': '0', 'Sort Flag': '0', 'Reserved For Future Use': '00', 'Size of Local Color Table': '111'}}
{'Canvas Width': 385, 'Canvas Height': 480, 'Packed Field': {'Global Color Flag': '1', 'Color Resolution': '111', 'Sort Flag': '0', 'Size of Global Color Table': '111'}, 'Background Color Index': '0x00', 'Pixel Aspect Ratio': '0x00'}
FRAME 2 
Local table 0x00: 000000
{'Extension Introducer': '0x21', 'Graphic Control Label': '0xf9', 'Byte Size': '0x04', 'Packed Field': {'Reserved for Future Use': '000', 'Disposal Method': '000', 'User Input Flag': '0', 'Transparent Color Flag': '1'}, 'Delay Time': 11, 'Transparent Color Index': '0x00', 'Block Terminator': '0x00'}
{'Image Seperator': '0x2c', 'Image Left': 74, 'Image Top': 50, 'Image Width': 251, 'Image Height': 409, 'Packed Field': {'Local Color Table': '1', 'Interlace Flag': '0', 'Sort Flag': '0', 'Reserved For Future Use': '00', 'Size of Local Color Table': '111'}}
{'Canvas Width': 385, 'Canvas Height': 480, 'Packed Field': {'Global Color Flag': '1', 'Color Resolution': '111', 'Sort Flag': '0', 'Size of Global Color Table': '111'}, 'Background Color Index': '0x00', 'Pixel Aspect Ratio': '0x00'}
FRAME 3 
Local table 0x00: 000000
{'Extension Introducer': '0x21', 'Graphic Control Label': '0xf9', 'Byte Size': '0x04', 'Packed Field': {'Reserved for Future Use': '000', 'Disposal Method': '000', 'User Input Flag': '0', 'Transparent Color Flag': '1'}, 'Delay Time': 11, 'Transparent Color Index': '0x00', 'Block Terminator': '0x00'}
{'Image Seperator': '0x2c', 'Image Left': 55, 'Image Top': 51, 'Image Width': 281, 'Image Height': 407, 'Packed Field': {'Local Color Table': '1', 'Interlace Flag': '0', 'Sort Flag': '0', 'Reserved For Future Use': '00', 'Size of Local Color Table': '111'}}
{'Canvas Width': 385, 'Canvas Height': 480, 'Packed Field': {'Global Color Flag': '1', 'Color Resolution': '111', 'Sort Flag': '0', 'Size of Global Color Table': '111'}, 'Background Color Index': '0x00', 'Pixel Aspect Ratio': '0x00'}
FRAME 4 
Local table 0x00: 000000
{'Extension Introducer': '0x21', 'Graphic Control Label': '0xf9', 'Byte Size': '0x04', 'Packed Field': {'Reserved for Future Use': '000', 'Disposal Method': '000', 'User Input Flag': '0', 'Transparent Color Flag': '1'}, 'Delay Time': 11, 'Transparent Color Index': '0x00', 'Block Terminator': '0x00'}
{'Image Seperator': '0x2c', 'Image Left': 3, 'Image Top': 67, 'Image Width': 380, 'Image Height': 399, 'Packed Field': {'Local Color Table': '1', 'Interlace Flag': '0', 'Sort Flag': '0', 'Reserved For Future Use': '00', 'Size of Local Color Table': '111'}}
{'Canvas Width': 385, 'Canvas Height': 480, 'Packed Field': {'Global Color Flag': '1', 'Color Resolution': '111', 'Sort Flag': '0', 'Size of Global Color Table': '111'}, 'Background Color Index': '0x00', 'Pixel Aspect Ratio': '0x00'}
FRAME 5 
Local table 0x00: 000000
{'Extension Introducer': '0x21', 'Graphic Control Label': '0xf9', 'Byte Size': '0x04', 'Packed Field': {'Reserved for Future Use': '000', 'Disposal Method': '000', 'User Input Flag': '0', 'Transparent Color Flag': '1'}, 'Delay Time': 11, 'Transparent Color Index': '0x00', 'Block Terminator': '0x00'}
{'Image Seperator': '0x2c', 'Image Left': 41, 'Image Top': 59, 'Image Width': 317, 'Image Height': 407, 'Packed Field': {'Local Color Table': '1', 'Interlace Flag': '0', 'Sort Flag': '0', 'Reserved For Future Use': '00', 'Size of Local Color Table': '111'}}
{'Canvas Width': 385, 'Canvas Height': 480, 'Packed Field': {'Global Color Flag': '1', 'Color Resolution': '111', 'Sort Flag': '0', 'Size of Global Color Table': '111'}, 'Background Color Index': '0x00', 'Pixel Aspect Ratio': '0x00'}
FRAME 6 
Local table 0x00: 000000
{'Extension Introducer': '0x21', 'Graphic Control Label': '0xf9', 'Byte Size': '0x04', 'Packed Field': {'Reserved for Future Use': '000', 'Disposal Method': '000', 'User Input Flag': '0', 'Transparent Color Flag': '1'}, 'Delay Time': 11, 'Transparent Color Index': '0x00', 'Block Terminator': '0x00'}
{'Image Seperator': '0x2c', 'Image Left': 83, 'Image Top': 48, 'Image Width': 259, 'Image Height': 411, 'Packed Field': {'Local Color Table': '1', 'Interlace Flag': '0', 'Sort Flag': '0', 'Reserved For Future Use': '00', 'Size of Local Color Table': '111'}}
{'Canvas Width': 385, 'Canvas Height': 480, 'Packed Field': {'Global Color Flag': '1', 'Color Resolution': '111', 'Sort Flag': '0', 'Size of Global Color Table': '111'}, 'Background Color Index': '0x00', 'Pixel Aspect Ratio': '0x00'}

Process finished with exit code 0

[RLaursen]

So I've figured out that the decoder is the culprit here, commit #5333 reversed the previous issue decoding gifs had. Previously, a subsequent gif frame would use the background's palette for the foreground, now subsequent frames' backgrounds use the current frame's palette. By flattening the GIF in the decoder (by leaving image8 as is where transparency exists), the wrong colors are being used for the background.

[RLaursen]

I believe the best compromise for this would be to make any pixels in the background image of a color not in the foreground image's color table transparent.

This means that the gif will display correctly, as the color not in the foreground color table would show through the transparency.

This could lead to the extraction of gif frames leaving users with transparency they don't expect, but isn't that better than making everything in the background colors they don't expect?

I think reverting #5333 is necessary.
edit: I have begun writing a fix and there is a workaround which maintains the usefulness of behavior introduced in #5333 and avoids it when it's not useful