Brotli v1.1.0 tests fail with pypy3

[anthrotype]

(moved from google/brotli-wheels#19)

I tried to update the submodule to brotli v1.1.0 in google/brotli-wheels@c7db335 (google/brotli-wheels#18), but encountered an issue with pypy. The wheels get built but then when the tests are run, we get these sort of errors:

test__test_compress_file_quality_11_lgwin_10_random_org_10k (tests.bro_test.TestBroCompress) ... RPython traceback:
  File "pypy_module_cpyext_1.c", line 6714, in wrapper_second_level__star_2_51
  File "implement.c", line 26795, in dispatcher_41
Fatal RPython error: AssertionError

e.g. see CI log https://github.com/google/brotli-wheels/actions/runs/6085235897/job/16508840821#step:5:944

I don't know how to fix that, so for the time being I will simply disable building wheels for pypy...
If anybody would like to help fix this, they'll be welcome to send a PR.

Also note that pypy users may benefit from using the alternative CFFI-based bindings from https://github.com/python-hyper/brotlicffi, whose API is mostly a drop-in replacement for our official brotli bindings.

[mgorny]

git bisect points to c8df4b3.

commit c8df4b3049ff1283fc4525defbcd003188f88963
Author: Ma Lin <animalize@users.noreply.github.com>
Date:   Thu Dec 29 21:07:16 2022 +0800

    Python: use a new output buffer code (#902)
    
    Currently, the output buffer is a std::vector<uint8_t>.
    When the buffer grows, resizing will cause unnecessary memcpy().
    
    This change uses a list of bytes object to represent output buffer, can avoid the extra overhead of resizing.
    In addition, C++ code can be removed, it's a pure C extension.

 python/_brotli.c  | 985 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 python/_brotli.cc | 755 -----------------------------------------
 setup.py          |  56 ++--
 3 files changed, 1009 insertions(+), 787 deletions(-)
 create mode 100644 python/_brotli.c
 delete mode 100644 python/_brotli.cc

I'm going to report this to PyPy upstream as well, as that RPython assertion looks suspicious.

Curious enough, with tip of master I'm getting a segfault rather than the assertion. I'm going to run a second bisect for that.

[mgorny]

The segfault is introduced in acc2656.

commit acc265655d6dde4ff1bce65308e9e152fbf3381a
Author:     Thomas Fischbacher <tfish@google.com>
AuthorDate: 2023-07-19 14:43:51 +0200
Commit:     Copybara-Service <copybara-worker@google.com>
CommitDate: 2023-07-19 14:44:36 +0200

    Small Python modernization of Brotli code.
    
    PiperOrigin-RevId: 549289787

[mgorny]

CC @animalize, @fischbacher

[mgorny]

Here's what I managed to gdb out of the segfault:

Core was generated by `/tmp/brotli/.venv/bin/python /tmp/brotli/python/bro.py -f -q 11 --lgwin 10 -i /'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __memcpy_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:540
540	../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S: No such file or directory.
(gdb) bt
#0  __memcpy_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:540
#1  0x00007f73df9816d6 in BlocksOutputBuffer_Finish (buffer=0x7fffe614e9d0, avail_out=32768) at python/_brotli.c:187
#2  0x00007f73df981bce in compress_stream (enc=0x15de790, op=BROTLI_OPERATION_PROCESS, input=0x15bc1dc "XXXXXXXXXXYYYYYYYYYY", 
    input_length=20) at python/_brotli.c:320
#3  0x00007f73df981ea9 in brotli_Compressor_process (self=0x15bc140, args=0x15bc170) at python/_brotli.c:447
#4  0x00007f73e3044f71 in pypy_g_generic_cpy_call__StdObjSpaceConst_funcPtr_SomeI_13 ()
   from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#5  0x00007f73e304566d in pypy_g_W_PyCFunctionObject_call_varargs ()
   from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#6  0x00007f73e27bbd1d in pypy_g.call_4 () from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#7  0x00007f73e29a7244 in pypy_g_BuiltinCodePassThroughArguments1_funcrun_obj ()
   from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#8  0x00007f73e27314ad in pypy_g_dispatcher_funcrun_obj () from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#9  0x00007f73e2a3e7e5 in pypy_g_call_valuestack__AccessDirect_None ()
   from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#10 0x00007f73e337240d in pypy_g_CALL_METHOD__AccessDirect_star_1 ()
   from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#11 0x00007f73e2a436ed in pypy_g_dispatch_bytecode__AccessDirect_None ()
   from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#12 0x00007f73e2a0b872 in pypy_g_handle_bytecode__AccessDirect_None ()
   from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#13 0x00007f73e31423d0 in pypy_g_portal_11 () from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#14 0x00007f73e35259eb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter ()
   from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#15 0x00007f73e29ce86a in pypy_g_execute_frame () from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#16 0x00007f73e390eb26 in __vmprof_eval_vmprof () from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#17 0x00007f73e337231b in pypy_g_CALL_METHOD_KW__AccessDirect_star_1 ()
   from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#18 0x00007f73e2a42ffd in pypy_g_dispatch_bytecode__AccessDirect_None ()
   from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#19 0x00007f73e2a0b872 in pypy_g_handle_bytecode__AccessDirect_None ()
   from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#20 0x00007f73e31423d0 in pypy_g_portal_11 () from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#21 0x00007f73e35259eb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter ()
   from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#22 0x00007f73e29ce86a in pypy_g_execute_frame () from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#23 0x00007f73e390eb26 in __vmprof_eval_vmprof () from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#24 0x00007f73e2a3fa7f in pypy_g_CALL_FUNCTION__AccessDirect_None ()
   from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#25 0x00007f73e2a4386f in pypy_g_dispatch_bytecode__AccessDirect_None ()
   from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#26 0x00007f73e2a0b872 in pypy_g_handle_bytecode__AccessDirect_None ()
   from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#27 0x00007f73e31423d0 in pypy_g_portal_11 () from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#28 0x00007f73e35259eb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter ()
   from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#29 0x00007f73e29ce86a in pypy_g_execute_frame () from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#30 0x00007f73e390eb26 in __vmprof_eval_vmprof () from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#31 0x00007f73e2c19474 in pypy_g_exec_ () from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#32 0x00007f73e275a0da in pypy_g_BuiltinActivation_UwS_ObjSpace_W_Root_W_Root_W_R_2 ()
   from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#33 0x00007f73e29a6f3b in pypy_g_BuiltinCode_funcrun_obj () from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#34 0x00007f73e2a3fa7f in pypy_g_CALL_FUNCTION__AccessDirect_None ()
   from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#35 0x00007f73e2a4386f in pypy_g_dispatch_bytecode__AccessDirect_None ()
   from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#36 0x00007f73e2a0b872 in pypy_g_handle_bytecode__AccessDirect_None ()
   from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#37 0x00007f73e31423d0 in pypy_g_portal_11 () from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#38 0x00007f73e35259eb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter ()
   from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#39 0x00007f73e29ce86a in pypy_g_execute_frame () from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#40 0x00007f73e390eb26 in __vmprof_eval_vmprof () from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#41 0x00007f73e2a3ecfd in pypy_g_CALL_FUNCTION_EX__AccessDirect_None ()
   from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#42 0x00007f73e2a43c8d in pypy_g_dispatch_bytecode__AccessDirect_None ()
   from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#43 0x00007f73e2a0b872 in pypy_g_handle_bytecode__AccessDirect_None ()
   from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#44 0x00007f73e31423d0 in pypy_g_portal_11 () from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#45 0x00007f73e35259eb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter ()
   from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#46 0x00007f73e29ce86a in pypy_g_execute_frame () from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#47 0x00007f73e390eb26 in __vmprof_eval_vmprof () from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#48 0x00007f73e2a3ecfd in pypy_g_CALL_FUNCTION_EX__AccessDirect_None ()
   from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#49 0x00007f73e2a43c8d in pypy_g_dispatch_bytecode__AccessDirect_None ()
   from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#50 0x00007f73e2a0b872 in pypy_g_handle_bytecode__AccessDirect_None ()
   from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#51 0x00007f73e31423d0 in pypy_g_portal_11 () from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#52 0x00007f73e35259eb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter ()
   from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#53 0x00007f73e29ce86a in pypy_g_execute_frame () from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#54 0x00007f73e390eb26 in __vmprof_eval_vmprof () from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#55 0x00007f73e2a3ecfd in pypy_g_CALL_FUNCTION_EX__AccessDirect_None ()
   from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#56 0x00007f73e2a43c8d in pypy_g_dispatch_bytecode__AccessDirect_None ()
   from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#57 0x00007f73e2a0b872 in pypy_g_handle_bytecode__AccessDirect_None ()
   from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#58 0x00007f73e31423d0 in pypy_g_portal_11 () from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#59 0x00007f73e35259eb in pypy_g_ll_portal_runner__Unsigned_Bool_pypy_interpreter ()
   from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#60 0x00007f73e29ce86a in pypy_g_execute_frame () from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#61 0x00007f73e390eb26 in __vmprof_eval_vmprof () from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#62 0x00007f73e29afba5 in pypy_g_call_function.star_3 () from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#63 0x00007f73e2983ac9 in pypy_g_entry_point () from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#64 0x00007f73e3a211af in pypy_main_startup () from /tmp/pypy-c-jit-107713-1775742c19f7-linux64/bin/libpypy3.10-c.so
#65 0x00007f73e1423f0a in __libc_start_call_main (main=main@entry=0x401060 <main>, argc=argc@entry=11, argv=argv@entry=0x7fffe614fcb8)
    at ../sysdeps/nptl/libc_start_call_main.h:58
#66 0x00007f73e1423fc5 in __libc_start_main_impl (main=0x401060 <main>, argc=11, argv=0x7fffe614fcb8, init=<optimized out>, 
    fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffe614fca8) at ../csu/libc-start.c:360
#67 0x000000000040108e in _start ()
(gdb) up
#1  0x00007f73df9816d6 in BlocksOutputBuffer_Finish (buffer=0x7fffe614e9d0, avail_out=32768) at python/_brotli.c:187
187	            memcpy(posi, PyBytes_AS_STRING(block), Py_SIZE(block));
(gdb) p posi
$1 = 0x15e855c ""
(gdb) p block
$2 = (PyObject *) 0x15e04f0
(gdb) up
#2  0x00007f73df981bce in compress_stream (enc=0x15de790, op=BROTLI_OPERATION_PROCESS, input=0x15bc1dc "XXXXXXXXXXYYYYYYYYYY", 
    input_length=20) at python/_brotli.c:320
320	  ret = BlocksOutputBuffer_Finish(&buffer, available_out);
(gdb) p buffer
$3 = {list = 0x138b680, allocated = 32768}
(gdb) p available_out 
$4 = 32768

I'm going to report the assertion prior to that commit first, and see what comes out of that.

[mgorny]

I've filed to PyPy upstream at https://foss.heptapod.net/pypy/pypy/-/issues/4000

[wRAR]

We get a suspicious crash in a Scrapy test (which passes garbage into the brotli decompressor among other things), the Python traceback is unhelpful as it's truncated, but the gdb one points to brotli:

#4  0x00007ffff697ee41 in pypy_debug_catch_fatal_exception () from ~/.pyenv/versions/pypy3.9-7.3.11/bin/libpypy3.9-c.so
#5  0x00007ffff5e69fea in pypy_g_wrapper_second_level.star_2_51 () from ~/.pyenv/versions/pypy3.9-7.3.11/bin/libpypy3.9-c.so
#6  0x00007ffff03357d7 in BlocksOutputBuffer_Finish () from ~/scrapy/.tox/py/lib/pypy3.9/site-packages/_brotli.pypy39-pp73-x86_64-linux-gnu.so
#7  0x00007ffff0335a70 in brotli_Decompressor_process () from ~/scrapy/.tox/py/lib/pypy3.9/site-packages/_brotli.pypy39-pp73-x86_64-linux-gnu.so
#8  0x00007ffff5f586f1 in pypy_g_generic_cpy_call__StdObjSpaceConst_funcPtr_SomeI_13 () from ~/.pyenv/versions/pypy3.9-7.3.11/bin/libpypy3.9-c.so

It doesn't look like this bug needs more info but if does please let me know!

[ghost]

#1  0x00007f73df9816d6 in BlocksOutputBuffer_Finish (buffer=0x7fffe614e9d0, avail_out=32768) at python/_brotli.c:187
187	            memcpy(posi, PyBytes_AS_STRING(block), Py_SIZE(block));

If change these invokings, does it fix the problem?

• change PyBytes_AS_STRING to PyBytes_AsString
• change Py_SIZE to PyBytes_Size

[mgorny]

Unfortunately, the segfault seems to be a heisenbug and I can't reproduce it right now. As for the assertion from pypy3, these changes don't help, though it seems to happen elsewhere. I've pasted the log for that to the PyPy bug; it's failing on:

#6  0x00007f567d30771e in BlocksOutputBuffer_Finish (buffer=0x7ffd6c889ff0, avail_out=32768) at python/_brotli.c:186
186	            block = PyList_GET_ITEM(buffer->list, i);

[mattip]

I can reliably reproduce this failure when running the command line from the segfaulting test

git clone https://github.com/google/brotli/
cd brotli
<activate pypy3.9 virtual env>
  CFLAGS="-O0 -g" pyp3.9 -m pip install .
gdb --args pypy3.9 python/bro.py -f -q 11 --lgwin 10 -i tests/testdata/10x10y -o /tmp/10x10y.bro

The cause of the segfault is in this stanza buffer->allocated - avail_out is zero, but the list of length 65 has data to be copied. I am not sure where the calculation of buffer->allocated is going wrong

    // Final bytes object
    result = PyBytes_FromStringAndSize(NULL, buffer->allocated - avail_out);
    if (result == NULL) {
        PyErr_SetString(PyExc_MemoryError, unable_allocate_msg);
        return NULL;
    }

    // Memory copy
    if (list_len > 0) {
        char *posi = PyBytes_AS_STRING(result);

        // Blocks except the last one
        Py_ssize_t i = 0;
        for (; i < list_len-1; i++) {
            block = PyList_GET_ITEM(buffer->list, i);
            memcpy(posi, PyBytes_AS_STRING(block), Py_SIZE(block));
            posi += Py_SIZE(block);
        }
        // The last block
        block = PyList_GET_ITEM(buffer->list, i);
        memcpy(posi, PyBytes_AS_STRING(block), Py_SIZE(block) - avail_out);
    } else {
        assert(Py_SIZE(result) == 0);
    }

[muxator]

Here's an example that reliably crashes on brotli v1.1.0 and does not crash on brotli v1.0.9.

I am on Fedora 39. Pypy 3.10 is taken from the system's packages:

sudo dnf install -y pypy3.10 pypy3.10-devel
pypy3.10 -m ensurepip
pypy3.10 -m venv /tmp/venv

Get brotli v1.1.0 from PyPI and compile it. If in this command you replace the version with 1.0.9, there will be no crashes at the subsequent pass:

/tmp/venv/bin/pip install brotli==1.1.0

Here's a minimal program that causes the crash:

/tmp/venv/bin/pypy3.10 -c "import brotli; brotli.compress(b'')"

Crashes with:

RPython traceback:
  File "pypy_module_cpyext_1.c", line 7139, in wrapper_second_level__star_2_51
  File "implement.c", line 30138, in dispatcher_Py_21
Fatal RPython error: AssertionError
Aborted (core dumped)

View the core dump (I am assuming there where no other crashes in the system after this one):

DUMP_PID=$(coredumpctl -1 --json=pretty | jq '.[0].pid')
coredumpctl gdb "${DUMP_PID}"

Which seems to show that the crash is in BlocksOutputBuffer_Finish() (maybe caused elsewhere):

[...]
Core was generated by `/tmp/venv/bin/pypy3.10 -c import brotli; brotli.compress(b'')'.
Program terminated with signal SIGABRT, Aborted.
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
44            return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;                                                                                                                                                     
Missing separate debuginfos, use: dnf debuginfo-install pypy3.10-7.3.13-1.3.10.fc39.x86_64
(gdb) bt
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007f8e612ae8a3 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#2  0x00007f8e6125c8ee in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007f8e612448ff in __GI_abort () at abort.c:79
#4  0x00007f8e6364a581 in ?? () from /lib64/libpypy3.10-c.so.0.1
#5  0x00007f8e62ce251d in ?? () from /lib64/libpypy3.10-c.so.0.1
#6  0x00007f8e534347d7 in BlocksOutputBuffer_Finish () from /tmp/venv/lib64/pypy3.10/site-packages/_brotli.pypy310-pp73-x86_64-linux-gnu.so
#7  0x00007f8e53434cc8 in compress_stream () from /tmp/venv/lib64/pypy3.10/site-packages/_brotli.pypy310-pp73-x86_64-linux-gnu.so
#8  0x00007f8e53434e40 in brotli_Compressor_process () from /tmp/venv/lib64/pypy3.10/site-packages/_brotli.pypy310-pp73-x86_64-linux-gnu.so
[...]

Something similar happens when calling brotli.decompress().