FPE in sixel_encoder_do_resize, encoder.c:636

[waugustus]

Description

There is a floating point exception error in sixel_encoder_do_resize, encoder.c:636 in img2sixel 1.8.6. Remote attackers could leverage this vulnerability to cause a denial-of-service via a crafted JPEG file.

Version

$ img2sixel -V
img2sixel 1.8.6

configured with:
  libcurl: yes
  libpng: yes
  libjpeg: yes
  gdk-pixbuf2: no
  GD: no

Reproduction

# img2sixel -h 128 poc /tmp/foo
ASAN:DEADLYSIGNAL
=================================================================
==363283==ERROR: AddressSanitizer: FPE on unknown address 0x55890ff6ea81 (pc 0x55890ff6ea81 bp 0x7ffc4b1bb790 sp 0x7ffc4b1bb760 T0)
    #0 0x55890ff6ea80 in sixel_encoder_do_resize /root/cov_test/libsixel/src/encoder.c:636
    #1 0x55890ff6fdf0 in sixel_encoder_encode_frame /root/cov_test/libsixel/src/encoder.c:968
    #2 0x55890ff743a9 in load_image_callback /root/cov_test/libsixel/src/encoder.c:1679
    #3 0x55890ffca596 in load_gif /root/cov_test/libsixel/src/fromgif.c:671
    #4 0x55890ffc3f92 in load_with_builtin /root/cov_test/libsixel/src/loader.c:908
    #5 0x55890ffc4936 in sixel_helper_load_image_file /root/cov_test/libsixel/src/loader.c:1418
    #6 0x55890ff7480c in sixel_encoder_encode /root/cov_test/libsixel/src/encoder.c:1743
    #7 0x55890ff6aa0e in main /root/cov_test/libsixel/converters/img2sixel.c:457
    #8 0x7f02f1eebc86 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21c86)
    #9 0x55890ff68359 in _start (/root/cov_test/libsixel/build_asan/bin/img2sixel+0x39359)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: FPE /root/cov_test/libsixel/src/encoder.c:636 in sixel_encoder_do_resize
==363283==ABORTING

poc.zip

Platform

# uname -a
Linux 4a409ce47130 5.4.0-70-generic #78~18.04.1-Ubuntu SMP Sat Mar 20 14:10:07 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux