Add Reproducible Error Injection [3.0]

[bernd-edlinger]

This adds reproducible memory error and test-data
error injection, to the fuzzy-test framework.

This feature can be enabled with ./config -DERROR_INJECT and additionally to enable call stacks -DERROR_CALLSTACK

If enable-asan is used, the callstack is printed by the sanitizer, otherwise please set a breakpoint
at the function "break_here", which is executed each time a memory allocation error is injected.

If called with the environment variable ERROR_INJECT defined to the empty string the initialization value is printed, and can be used to reproduce the failure later, by passing the value to the ERROR_INJECT variable.

There is a search script that can be used to look
for errors, and print the command to reproduce the bug:

./testrun.sh

This runs in endless mode until an error is found.

Checklist

• documentation is added or updated
• tests are added or updated

[bernd-edlinger]

There is already one Error Detected here: https://github.com/openssl/openssl/actions/runs/5829582130

2023-08-11T06:32:23.3615084Z ERROR_INJECT=1691772012 ../util/shlib_wrap.sh ./asn1-test ./corpora/asn1/0110b728e90d45b06d479874ca848e815a6fa4fb
2023-08-11T06:32:23.3616264Z log file: asn1-0110b728e90d45b06d479874ca848e815a6fa4fb-21104-test.out
2023-08-11T06:32:23.3626907Z ERROR_INJECT=1691772012
2023-08-11T06:32:23.3627587Z     #0 0x7f81fe80af18 in __sanitizer_print_stack_trace ../../../../src/libsanitizer/asan/asan_stack.cpp:86
2023-08-11T06:32:23.3628332Z     #1 0x5558b143c774 in my_malloc fuzz/test-corpus.c:117
2023-08-11T06:32:23.3628851Z     #2 0x7f81fd34ec85 in CRYPTO_zalloc crypto/mem.c:197
2023-08-11T06:32:23.3629968Z     #3 0x7f81fd377cf4 in CRYPTO_THREAD_lock_new crypto/threads_pthread.c:50
2023-08-11T06:32:23.3630723Z     #4 0x7f81fd3415e4 in context_init crypto/context.c:86
2023-08-11T06:32:23.3631124Z     #5 0x7f81fd341863 in default_context_do_init crypto/context.c:155
2023-08-11T06:32:23.3631537Z     #6 0x7f81fd341863 in default_context_do_init_ossl_ crypto/context.c:152
2023-08-11T06:32:23.3632127Z     #7 0x7f81fbe1ff67  (/lib/x86_64-linux-gnu/libc.so.6+0x99f67)
2023-08-11T06:32:23.3632523Z     #8 0x7f81fd377dee in CRYPTO_THREAD_run_once crypto/threads_pthread.c:156
2023-08-11T06:32:23.3633032Z     #9 0x7f81fd341cc6 in get_thread_default_context crypto/context.c:166
2023-08-11T06:32:23.3633396Z     #10 0x7f81fd341cc6 in get_default_context crypto/context.c:174
2023-08-11T06:32:23.3633775Z     #11 0x7f81fd341cc6 in ossl_lib_ctx_get_concrete crypto/context.c:295
2023-08-11T06:32:23.3634182Z     #12 0x7f81fd341cc6 in ossl_lib_ctx_get_concrete crypto/context.c:291
2023-08-11T06:32:23.3634560Z     #13 0x7f81fd342110 in ossl_lib_ctx_get_data crypto/context.c:373
2023-08-11T06:32:23.3634926Z     #14 0x7f81fd36dc28 in get_provider_store crypto/provider_core.c:339
2023-08-11T06:32:23.3635334Z     #15 0x7f81fd36dc28 in ossl_provider_info_add_to_store crypto/provider_core.c:365
2023-08-11T06:32:23.3635750Z     #16 0x7f81fd367899 in OSSL_PROVIDER_add_builtin crypto/provider.c:131
2023-08-11T06:32:23.3636572Z     #17 0x5558b143bee5 in FuzzerSetRand fuzz/fuzz_rand.c:159
2023-08-11T06:32:23.3636962Z     #18 0x5558b1439b1c in FuzzerInitialize fuzz/asn1.c:287
2023-08-11T06:32:23.3637539Z     #19 0x5558b1439493 in main fuzz/test-corpus.c:197
2023-08-11T06:32:23.3637960Z     #20 0x7f81fbdafd8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)
2023-08-11T06:32:23.3638462Z     #21 0x7f81fbdafe3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f)
2023-08-11T06:32:23.3639000Z     #22 0x5558b1439a44 in _start (/home/runner/work/openssl/openssl/fuzz/asn1-test+0x9a44)
2023-08-11T06:32:23.3639250Z 
2023-08-11T06:32:23.3639503Z crypto/threads_pthread.c:97:9: runtime error: null pointer passed as argument 1, which is declared to never be null
2023-08-11T06:32:23.3639955Z     #0 0x7f81fcf0f3a0 in CRYPTO_THREAD_read_lock crypto/threads_pthread.c:97
2023-08-11T06:32:23.3640349Z     #1 0x7f81fd34213d in ossl_lib_ctx_get_data crypto/context.c:377
2023-08-11T06:32:23.3640988Z     #2 0x7f81fd36dc28 in get_provider_store crypto/provider_core.c:339
2023-08-11T06:32:23.3641400Z     #3 0x7f81fd36dc28 in ossl_provider_info_add_to_store crypto/provider_core.c:365
2023-08-11T06:32:23.3641793Z     #4 0x7f81fd367899 in OSSL_PROVIDER_add_builtin crypto/provider.c:131
2023-08-11T06:32:23.3642165Z     #5 0x5558b143bee5 in FuzzerSetRand fuzz/fuzz_rand.c:159
2023-08-11T06:32:23.3642512Z     #6 0x5558b1439b1c in FuzzerInitialize fuzz/asn1.c:287
2023-08-11T06:32:23.3643003Z     #7 0x5558b1439493 in main fuzz/test-corpus.c:197
2023-08-11T06:32:23.3643438Z     #8 0x7f81fbdafd8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)
2023-08-11T06:32:23.3643934Z     #9 0x7f81fbdafe3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f)
2023-08-11T06:32:23.3644452Z     #10 0x5558b1439a44 in _start (/home/runner/work/openssl/openssl/fuzz/asn1-test+0x9a44)
2023-08-11T06:32:23.3644693Z 
2023-08-11T06:32:28.3671969Z ##[error]Process completed with exit code 1.

[bernd-edlinger]

The latest CI test run https://github.com/openssl/openssl/actions/runs/5902344013
found a memory leak in the 3.0 branch:

2023-08-18T12:10:35.4246986Z ERROR_INJECT=1692111091 ../util/shlib_wrap.sh ./asn1-test ./corpora/asn1/0157dd1386a0e3b3972074e4b26ce92c9918e4c2
2023-08-18T12:10:35.4248170Z log file: asn1-0157dd1386a0e3b3972074e4b26ce92c9918e4c2-21162-test.out
2023-08-18T12:10:35.4258787Z ERROR_INJECT=1692111091
2023-08-18T12:10:35.4259557Z     #0 0x7f0830cddf18 in __sanitizer_print_stack_trace ../../../../src/libsanitizer/asan/asan_stack.cpp:86
2023-08-18T12:10:35.4261794Z     #1 0x563a84caf6f4 in my_realloc fuzz/test-corpus.c:132
2023-08-18T12:10:35.4285051Z     #2 0x7f082f995c2a in sk_reserve crypto/stack/stack.c:215
2023-08-18T12:10:35.4287600Z     #3 0x7f082f995c2a in OPENSSL_sk_insert crypto/stack/stack.c:271
2023-08-18T12:10:35.4288741Z     #4 0x7f082f81ddba in CRYPTO_set_ex_data crypto/ex_data.c:477
2023-08-18T12:10:35.4289183Z     #5 0x7f082f8144ad in ossl_lib_ctx_generic_new crypto/context.c:333
2023-08-18T12:10:35.4289761Z     #6 0x7f082f81dc2b in ossl_crypto_alloc_ex_data_intern crypto/ex_data.c:456
2023-08-18T12:10:35.4290252Z     #7 0x7f082f815675 in ossl_lib_ctx_get_data crypto/context.c:428
2023-08-18T12:10:35.4290671Z     #8 0x7f082f79cc25 in get_evp_method_store crypto/evp/evp_fetch.c:82
2023-08-18T12:10:35.4291165Z     #9 0x7f082f79cc25 in evp_method_store_cache_flush crypto/evp/evp_fetch.c:458
2023-08-18T12:10:35.4291737Z     #10 0x7f082f84171e in provider_flush_store_cache crypto/provider_core.c:1186
2023-08-18T12:10:35.4292239Z     #11 0x7f082f84171e in ossl_provider_activate crypto/provider_core.c:1258
2023-08-18T12:10:35.4292672Z     #12 0x7f082f84171e in ossl_provider_activate crypto/provider_core.c:1243
2023-08-18T12:10:35.4293150Z     #13 0x7f082f83a4c6 in OSSL_PROVIDER_try_load crypto/provider.c:31
2023-08-18T12:10:35.4294230Z     #14 0x563a84caef1e in FuzzerSetRand fuzz/fuzz_rand.c:161
2023-08-18T12:10:35.4294719Z     #15 0x563a84cacb1c in FuzzerInitialize fuzz/asn1.c:287
2023-08-18T12:10:35.4295336Z     #16 0x563a84cac493 in main fuzz/test-corpus.c:197
2023-08-18T12:10:35.4295872Z     #17 0x7f082e282d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)
2023-08-18T12:10:35.4296883Z     #18 0x7f082e282e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f)
2023-08-18T12:10:35.4297520Z     #19 0x563a84caca44 in _start (/home/runner/work/openssl/openssl/fuzz/asn1-test+0x9a44)
2023-08-18T12:10:35.4297872Z 
2023-08-18T12:10:35.4298075Z # ./corpora/asn1/0157dd1386a0e3b3972074e4b26ce92c9918e4c2
2023-08-18T12:10:35.4298315Z 
2023-08-18T12:10:35.4298481Z =================================================================
2023-08-18T12:10:35.4298838Z ==21289==ERROR: LeakSanitizer: detected memory leaks
2023-08-18T12:10:35.4299078Z 
2023-08-18T12:10:35.4299277Z Direct leak of 48 byte(s) in 1 object(s) allocated from:
2023-08-18T12:10:35.4300305Z     #0 0x7f0830cd1887 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
2023-08-18T12:10:35.4300803Z     #1 0x7f082f821c85 in CRYPTO_zalloc crypto/mem.c:197
2023-08-18T12:10:35.4301229Z     #2 0x7f082f8e4203 in ossl_method_store_new crypto/property/property.c:252
2023-08-18T12:10:35.4301719Z     #3 0x7f082f814456 in ossl_lib_ctx_generic_new crypto/context.c:324
2023-08-18T12:10:35.4302197Z     #4 0x7f082f81dc2b in ossl_crypto_alloc_ex_data_intern crypto/ex_data.c:456
2023-08-18T12:10:35.4302663Z     #5 0x7f082f815675 in ossl_lib_ctx_get_data crypto/context.c:428
2023-08-18T12:10:35.4303138Z     #6 0x7f082f79cc25 in get_evp_method_store crypto/evp/evp_fetch.c:82
2023-08-18T12:10:35.4303618Z     #7 0x7f082f79cc25 in evp_method_store_cache_flush crypto/evp/evp_fetch.c:458
2023-08-18T12:10:35.4304117Z     #8 0x7f082f84171e in provider_flush_store_cache crypto/provider_core.c:1186
2023-08-18T12:10:35.4304558Z     #9 0x7f082f84171e in ossl_provider_activate crypto/provider_core.c:1258
2023-08-18T12:10:35.4305086Z     #10 0x7f082f84171e in ossl_provider_activate crypto/provider_core.c:1243
2023-08-18T12:10:35.4305581Z     #11 0x7f082f83a4c6 in OSSL_PROVIDER_try_load crypto/provider.c:31
2023-08-18T12:10:35.4305978Z     #12 0x563a84caef1e in FuzzerSetRand fuzz/fuzz_rand.c:161
2023-08-18T12:10:35.4306425Z     #13 0x563a84cacb1c in FuzzerInitialize fuzz/asn1.c:287
2023-08-18T12:10:35.4307019Z     #14 0x563a84cac493 in main fuzz/test-corpus.c:197
2023-08-18T12:10:35.4307576Z     #15 0x7f082e282d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)
2023-08-18T12:10:35.4307768Z 
2023-08-18T12:10:35.4307974Z Indirect leak of 56 byte(s) in 1 object(s) allocated from:
2023-08-18T12:10:35.4308518Z     #0 0x7f0830cd1887 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
2023-08-18T12:10:35.4309002Z     #1 0x7f082f821c85 in CRYPTO_zalloc crypto/mem.c:197
2023-08-18T12:10:35.4309454Z     #2 0x7f082f84acf4 in CRYPTO_THREAD_lock_new crypto/threads_pthread.c:50
2023-08-18T12:10:35.4309977Z     #3 0x7f082f8e4263 in ossl_method_store_new crypto/property/property.c:256
2023-08-18T12:10:35.4310452Z     #4 0x7f082f814456 in ossl_lib_ctx_generic_new crypto/context.c:324
2023-08-18T12:10:35.4310922Z     #5 0x7f082f81dc2b in ossl_crypto_alloc_ex_data_intern crypto/ex_data.c:456
2023-08-18T12:10:35.4311337Z     #6 0x7f082f815675 in ossl_lib_ctx_get_data crypto/context.c:428
2023-08-18T12:10:35.4311853Z     #7 0x7f082f79cc25 in get_evp_method_store crypto/evp/evp_fetch.c:82
2023-08-18T12:10:35.4312336Z     #8 0x7f082f79cc25 in evp_method_store_cache_flush crypto/evp/evp_fetch.c:458
2023-08-18T12:10:35.4312774Z     #9 0x7f082f84171e in provider_flush_store_cache crypto/provider_core.c:1186
2023-08-18T12:10:35.4313259Z     #10 0x7f082f84171e in ossl_provider_activate crypto/provider_core.c:1258
2023-08-18T12:10:35.4313907Z     #11 0x7f082f84171e in ossl_provider_activate crypto/provider_core.c:1243
2023-08-18T12:10:35.4314333Z     #12 0x7f082f83a4c6 in OSSL_PROVIDER_try_load crypto/provider.c:31
2023-08-18T12:10:35.4314785Z     #13 0x563a84caef1e in FuzzerSetRand fuzz/fuzz_rand.c:161
2023-08-18T12:10:35.4315224Z     #14 0x563a84cacb1c in FuzzerInitialize fuzz/asn1.c:287
2023-08-18T12:10:35.4315769Z     #15 0x563a84cac493 in main fuzz/test-corpus.c:197
2023-08-18T12:10:35.4316482Z     #16 0x7f082e282d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)
2023-08-18T12:10:35.4316747Z 
2023-08-18T12:10:35.4316953Z Indirect leak of 56 byte(s) in 1 object(s) allocated from:
2023-08-18T12:10:35.4317451Z     #0 0x7f0830cd1887 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
2023-08-18T12:10:35.4317874Z     #1 0x7f082f821c85 in CRYPTO_zalloc crypto/mem.c:197
2023-08-18T12:10:35.4318349Z     #2 0x7f082f84acf4 in CRYPTO_THREAD_lock_new crypto/threads_pthread.c:50
2023-08-18T12:10:35.4318898Z     #3 0x7f082f8e4291 in ossl_method_store_new crypto/property/property.c:257
2023-08-18T12:10:35.4319376Z     #4 0x7f082f814456 in ossl_lib_ctx_generic_new crypto/context.c:324
2023-08-18T12:10:35.4319981Z     #5 0x7f082f81dc2b in ossl_crypto_alloc_ex_data_intern crypto/ex_data.c:456
2023-08-18T12:10:35.4320446Z     #6 0x7f082f815675 in ossl_lib_ctx_get_data crypto/context.c:428
2023-08-18T12:10:35.4320957Z     #7 0x7f082f79cc25 in get_evp_method_store crypto/evp/evp_fetch.c:82
2023-08-18T12:10:35.4321397Z     #8 0x7f082f79cc25 in evp_method_store_cache_flush crypto/evp/evp_fetch.c:458
2023-08-18T12:10:35.4321894Z     #9 0x7f082f84171e in provider_flush_store_cache crypto/provider_core.c:1186
2023-08-18T12:10:35.4322401Z     #10 0x7f082f84171e in ossl_provider_activate crypto/provider_core.c:1258
2023-08-18T12:10:35.4322882Z     #11 0x7f082f84171e in ossl_provider_activate crypto/provider_core.c:1243
2023-08-18T12:10:35.4323338Z     #12 0x7f082f83a4c6 in OSSL_PROVIDER_try_load crypto/provider.c:31
2023-08-18T12:10:35.4323795Z     #13 0x563a84caef1e in FuzzerSetRand fuzz/fuzz_rand.c:161
2023-08-18T12:10:35.4325953Z     #14 0x563a84cacb1c in FuzzerInitialize fuzz/asn1.c:287
2023-08-18T12:10:35.4326564Z     #15 0x563a84cac493 in main fuzz/test-corpus.c:197
2023-08-18T12:10:35.4327103Z     #16 0x7f082e282d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)
2023-08-18T12:10:35.4327443Z 
2023-08-18T12:10:35.4327657Z Indirect leak of 32 byte(s) in 1 object(s) allocated from:
2023-08-18T12:10:35.4328110Z     #0 0x7f0830cd1887 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
2023-08-18T12:10:35.4328616Z     #1 0x7f082f821c85 in CRYPTO_zalloc crypto/mem.c:197
2023-08-18T12:10:35.4329150Z     #2 0x7f082f8e4231 in ossl_sa_ALGORITHM_new crypto/property/property.c:96
2023-08-18T12:10:35.4329694Z     #3 0x7f082f8e4231 in ossl_method_store_new crypto/property/property.c:255
2023-08-18T12:10:35.4330121Z     #4 0x7f082f814456 in ossl_lib_ctx_generic_new crypto/context.c:324
2023-08-18T12:10:35.4330604Z     #5 0x7f082f81dc2b in ossl_crypto_alloc_ex_data_intern crypto/ex_data.c:456
2023-08-18T12:10:35.4331070Z     #6 0x7f082f815675 in ossl_lib_ctx_get_data crypto/context.c:428
2023-08-18T12:10:35.4331482Z     #7 0x7f082f79cc25 in get_evp_method_store crypto/evp/evp_fetch.c:82
2023-08-18T12:10:35.4332019Z     #8 0x7f082f79cc25 in evp_method_store_cache_flush crypto/evp/evp_fetch.c:458
2023-08-18T12:10:35.4332533Z     #9 0x7f082f84171e in provider_flush_store_cache crypto/provider_core.c:1186
2023-08-18T12:10:35.4333030Z     #10 0x7f082f84171e in ossl_provider_activate crypto/provider_core.c:1258
2023-08-18T12:10:35.4333456Z     #11 0x7f082f84171e in ossl_provider_activate crypto/provider_core.c:1243
2023-08-18T12:10:35.4333975Z     #12 0x7f082f83a4c6 in OSSL_PROVIDER_try_load crypto/provider.c:31
2023-08-18T12:10:35.4334602Z     #13 0x563a84caef1e in FuzzerSetRand fuzz/fuzz_rand.c:161
2023-08-18T12:10:35.4335005Z     #14 0x563a84cacb1c in FuzzerInitialize fuzz/asn1.c:287
2023-08-18T12:10:35.4335554Z     #15 0x563a84cac493 in main fuzz/test-corpus.c:197
2023-08-18T12:10:35.4336082Z     #16 0x7f082e282d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)
2023-08-18T12:10:35.4336397Z 
2023-08-18T12:10:35.4336623Z SUMMARY: AddressSanitizer: 192 byte(s) leaked in 4 allocation(s).
2023-08-18T12:10:40.4292902Z ##[error]Process completed with exit code 1.