Redis 6.2.13 crashed by signal: 4, si_code: 2 #4402
Comments
|
my cpu info is below. could anyone help me urgently. I'm suffering this for 1 week. $ cat /proc/cpuinfo processor : 1 processor : 2 processor : 3 |
|
hi @zahidekilic |
|
here is my host machine's CPU flags. There is no avx flag. $ cat /proc/cpuinfo | grep sse |
|
Unfortunately, it appears that changing permissions may not resolve the issue in this case The problem is likely related to the binary compilation for the Docker image. This is a bug that should be addressed. One workaround could be running the Docker image on a more modern machine, and the second would be compiling from source. Sorry for the inconvenience. |
|
This issue is stale because it has been open for 60 days with no activity. |
I'm new to redis and redisearch ecosystem. I'm running redis on a docker container with the command below:
docker run -it --sysctl net.core.somaxconn=4096 --network=mynet -p 6379:6379 --name myredis redislabs/redisearch:latest
from python I'm pushing a vector database to redis. There is no problem with it. But while I was searching the newly pushed vectors redis crashes with the logs below. I'm a on a production host which is AlmaLinux release 8.9 (Midnight Oncilla).
here is the crash log. Thanks for responses
1:M 29 Jan 2024 09:13:51.920 # Redis 6.2.13 crashed by signal: 4, si_code: 2
1:M 29 Jan 2024 09:13:51.920 # Crashed running the instruction at: 0x7f19d4570829
------ STACK TRACE ------
EIP:
/usr/lib/redis/modules/redisearch.so(_Z35FP32_InnerProductSIMD16Ext_SSE_implPKvS0_m+0x9)[0x7f19d4570829]
Backtrace:
/lib/x86_64-linux-gnu/libpthread.so.0(+0x13140)[0x7f19d525b140]
redis-server *:6379(invalidFunctionWasCalled+0x0)[0x560db9e4b7a0]
/usr/lib/redis/modules/redisearch.so(_Z30FP32_InnerProductSIMD16Ext_SSEPKvS0_m+0x9)[0x7f19d45708a9]
/usr/lib/redis/modules/redisearch.so(_ZNK15BruteForceIndexIffE18computeBlockScoresEP11VectorBlockPKvPvP22VecSimQueryResult_Code+0x1ce)[0x7f19d4544dbe]
/usr/lib/redis/modules/redisearch.so(_ZN15BruteForceIndexIffE9topKQueryEPKvmP17VecSimQueryParams+0x146)[0x7f19d4545496]
/usr/lib/redis/modules/redisearch.so(VecSimIndex_TopKQuery+0x14)[0x7f19d453b9f4]
/usr/lib/redis/modules/redisearch.so(+0x141908)[0x7f19d44ba908]
/usr/lib/redis/modules/redisearch.so(+0x16c38c)[0x7f19d44e538c]
/usr/lib/redis/modules/redisearch.so(+0x16c6a5)[0x7f19d44e56a5]
/usr/lib/redis/modules/redisearch.so(+0x16cc30)[0x7f19d44e5c30]
/usr/lib/redis/modules/redisearch.so(+0x16c718)[0x7f19d44e5718]
/usr/lib/redis/modules/redisearch.so(sendChunk+0x92)[0x7f19d4489642]
/usr/lib/redis/modules/redisearch.so(AREQ_Execute+0x19)[0x7f19d4489d39]
/usr/lib/redis/modules/redisearch.so(RSSearchCommand+0xd8)[0x7f19d448a4d8]
redis-server *:6379(RedisModuleCommandDispatcher+0x53)[0x560db9e860e3]
redis-server *:6379(call+0xdb)[0x560db9dfa37b]
redis-server *:6379(processCommand+0x5a3)[0x560db9dfbff3]
redis-server *:6379(processInputBuffer+0xf8)[0x560db9e0f9e8]
redis-server *:6379(+0xfcc78)[0x560db9ea7c78]
redis-server *:6379(aeProcessEvents+0x292)[0x560db9df2e82]
redis-server *:6379(aeMain+0x1d)[0x560db9df30ed]
redis-server *:6379(main+0x316)[0x560db9def306]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea)[0x7f19d5097d0a]
redis-server *:6379(_start+0x2a)[0x560db9def7da]
------ REGISTERS ------
1:M 29 Jan 2024 09:13:51.923 #
RAX:00007f19d0e55a48 RBX:0000000000000000
RCX:00007f19d10767a4 RDX:0000000000000180
RDI:00007f19d0e55448 RSI:00007ffe067fa910
RBP:00007f19d4d0d548 RSP:00007ffe067fa888
R8 :0000000000000208 R9 :00007f19d4c033f8
R10:00007f19d506e778 R11:0000000000000012
R12:00007ffe067faf60 R13:00007f19d4dba1c8
R14:00007f19d4c585b8 R15:00007ffe067fa910
RIP:00007f19d4570829 EFL:0000000000010287
CSGSFS:002b000000000033
1:M 29 Jan 2024 09:13:51.923 # (00007ffe067fa897) -> 0000000000000000
1:M 29 Jan 2024 09:13:51.923 # (00007ffe067fa896) -> 00007f19d4c58480
1:M 29 Jan 2024 09:13:51.923 # (00007ffe067fa895) -> 00007f19d4d0d548
1:M 29 Jan 2024 09:13:51.923 # (00007ffe067fa894) -> 00007f19d4c2b1b8
1:M 29 Jan 2024 09:13:51.923 # (00007ffe067fa893) -> 00007ffe067fafd0
1:M 29 Jan 2024 09:13:51.923 # (00007ffe067fa892) -> 00007f19d4d608a0
1:M 29 Jan 2024 09:13:51.923 # (00007ffe067fa891) -> 0000560dbad977a0
1:M 29 Jan 2024 09:13:51.923 # (00007ffe067fa890) -> 00007f19d47ed0f0
1:M 29 Jan 2024 09:13:51.923 # (00007ffe067fa88f) -> 00007f19d5257c00
1:M 29 Jan 2024 09:13:51.923 # (00007ffe067fa88e) -> 0000560dbad977a8
1:M 29 Jan 2024 09:13:51.923 # (00007ffe067fa88d) -> 00007f19d10767a4
1:M 29 Jan 2024 09:13:51.923 # (00007ffe067fa88c) -> 00007ffe067faf58
1:M 29 Jan 2024 09:13:51.923 # (00007ffe067fa88b) -> 000000000000000b
1:M 29 Jan 2024 09:13:51.923 # (00007ffe067fa88a) -> 00007f19d4544dbe
1:M 29 Jan 2024 09:13:51.923 # (00007ffe067fa889) -> 0000000000000000
1:M 29 Jan 2024 09:13:51.923 # (00007ffe067fa888) -> 00007f19d45708a9
------ INFO OUTPUT ------
Server
redis_version:6.2.13
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:6a48f422b940fd75
redis_mode:standalone
os:Linux 4.18.0-513.11.1.el8_9.x86_64 x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:10.2.1
process_id:1
process_supervised:no
run_id:542e1a5f0d8bc746b7c8e1ac7c02ccff787506d1
tcp_port:6379
server_time_usec:1706519631919864
uptime_in_seconds:68
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:12020815
executable:/data/redis-server
config_file:
io_threads_active:0
Clients
connected_clients:1
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:24
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
Memory
used_memory:3457888
used_memory_human:3.30M
used_memory_rss:17670144
used_memory_rss_human:16.85M
used_memory_peak:3881016
used_memory_peak_human:3.70M
used_memory_peak_perc:89.10%
used_memory_overhead:928816
used_memory_startup:907744
used_memory_dataset:2529072
used_memory_dataset_perc:99.17%
allocator_allocated:3896768
allocator_active:4296704
allocator_resident:6868992
total_system_memory:8058515456
total_system_memory_human:7.51G
used_memory_lua:30720
used_memory_lua_human:30.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.10
allocator_frag_bytes:399936
allocator_rss_ratio:1.60
allocator_rss_bytes:2572288
rss_overhead_ratio:2.57
rss_overhead_bytes:10801152
mem_fragmentation_ratio:5.19
mem_fragmentation_bytes:14265080
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:20504
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
Persistence
loading:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:34
rdb_bgsave_in_progress:0
rdb_last_save_time:1706519563
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
Stats
total_connections_received:12
total_commands_processed:73
instantaneous_ops_per_sec:0
total_net_input_bytes:45782
total_net_output_bytes:20398
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:1
evicted_keys:0
keyspace_hits:11
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:25
dump_payload_sanitizations:0
total_reads_processed:84
total_writes_processed:72
io_threaded_reads_processed:0
io_threaded_writes_processed:0
Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:b8e2b256d794b6a429191fb5fd9bcf5b446ffe9e
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
CPU
used_cpu_sys:0.112307
used_cpu_user:0.123880
used_cpu_sys_children:0.004418
used_cpu_user_children:0.003021
used_cpu_sys_main_thread:0.108948
used_cpu_user_main_thread:0.122417
Modules
module:name=ReJSON,ver=999999,api=1,filters=0,usedby=[search],using=[],options=[handle-io-errors]
module:name=search,ver=20613,api=1,filters=0,usedby=[],using=[ReJSON],options=[handle-io-errors]
Commandstats
cmdstat_module:calls=12,usec=54,usec_per_call=4.50,rejected_calls=0,failed_calls=0
cmdstat_FT.CREATE:calls=1,usec=2043,usec_per_call=2043.00,rejected_calls=0,failed_calls=0
cmdstat_info:calls=13,usec=184,usec_per_call=14.15,rejected_calls=0,failed_calls=0
cmdstat_hset:calls=11,usec=7925,usec_per_call=720.45,rejected_calls=0,failed_calls=0
cmdstat_FT.INFO:calls=12,usec=1319,usec_per_call=109.92,rejected_calls=0,failed_calls=1
cmdstat_client:calls=24,usec=238,usec_per_call=9.92,rejected_calls=0,failed_calls=24
Errorstats
errorstat_ERR:count=24
errorstat_Unknown:count=1
Cluster
cluster_enabled:0
Keyspace
db0:keys=11,expires=0,avg_ttl=0
------ CLIENT LIST OUTPUT ------
id=20 addr=172.19.0.3:45868 laddr=172.19.0.2:6379 fd=8 name= age=9 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=1802 qbuf-free=39152 argv-mem=1678 obl=0 oll=0 omem=0 tot-mem=63278 events=r cmd=FT.SEARCH user=default redir=-1
------ CURRENT CLIENT INFO ------
id=20 addr=172.19.0.3:45868 laddr=172.19.0.2:6379 fd=8 name= age=9 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=1802 qbuf-free=39152 argv-mem=1678 obl=0 oll=0 omem=0 tot-mem=63278 events=r cmd=FT.SEARCH user=default redir=-1
argv[0]: 'FT.SEARCH'
argv[1]: 'myasistan'
argv[2]: '()=>[KNN 5 @content_vector $vector AS distance]'
argv[3]: 'RETURN'
argv[4]: '2'
argv[5]: 'content'
argv[6]: 'source'
argv[7]: 'SORTBY'
argv[8]: 'distance'
argv[9]: 'ASC'
argv[10]: 'DIALECT'
argv[11]: '2'
argv[12]: 'LIMIT'
argv[13]: '0'
argv[14]: '5'
argv[15]: 'params'
argv[16]: '2'
argv[17]: 'vector'
argv[18]: 'W]j<▒"▒▒6▒▒▒?▒QD▒▒}▒= ▒▒▒▒▒|=▒▒▒;▒D=gV▒▒dɼ?▒
▒▒W;▒▒▒<▒▒▒װZ=▒▒7▒½▒輕▒▒;▒▒▒<Q3F▒▒ұ▒▒C▒▒▒&▒<▒▒;▒%▒▒=M▒:▒▒װ<}=▒ּ/?X▒w~▒==3<▒t▒▒▒▒7▒P==J=f▒▒J▒y▒s▒▒▒(▒▒<▒▒I=▒▒▒▒[▒▒▒▒▒<輒7
▒L▒▒▒j▒,▒▒▒?▒▒<3W▒▒▒x▒G▒<d0=▒@o▒▒@=▒▒=▒▒z=▒Nc▒▒▒V<=▒▒<▒▒▒;▒L▒^
=▒▒j▒▒a=▒Qi=▒▒;Ek▒▒▒▒▒▒[P▒_{▒▒2=▒▒<▒▒▒▒▒<▒;▒3▒m-J▒▒▒= ▒.▒▒▒▒:▒ ▒W;y▒6;哼▒z▒<o▒/=H
▒=▒▒▒▒bX▒▒▒▒=▒%▒▒}▒▒N0=)▒N▒▒▒▒Uf]<▒ͥ▒▒Q=+ =▒<▒<▒
▒<w▒=▒▒<▒▒=▒X<'▒2▒O▒=▒▒n<tּ▒(J=▒&▒
^{▒;=^'
------ MODULES INFO OUTPUT ------
ReJSON_trace
ReJSON_trace: 0: redis_module::base_info_func
at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/redis-module-2.0.4/src/lib.rs:64:33
1: rejson::__info_func
at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/redis-module-2.0.4/src/macros.rs:166:13
2: modulesCollectInfo
at /usr/src/redis/src/module.c:7100:9
3: logModulesInfo
at /usr/src/redis/src/debug.c:1624:22
4: printCrashReport
at /usr/src/redis/src/debug.c:1896:5
sigsegvHandler
at /usr/src/redis/src/debug.c:1878:5
5:
6: _Z35FP32_InnerProductSIMD16Ext_SSE_implPKvS0_m
at /build/deps/VectorSimilarity/src/VecSim/spaces/IP/IP_SSE_FP32.cpp:21:19
7: _Z30FP32_InnerProductSIMD16Ext_SSEPKvS0_m
at /build/deps/VectorSimilarity/src/VecSim/spaces/IP/IP_SSE_FP32.cpp:55:54
8: _ZNK15BruteForceIndexIffE18computeBlockScoresEP11VectorBlockPKvPvP22VecSimQueryResult_Code
at /build/deps/VectorSimilarity/src/VecSim/algorithms/brute_force/brute_force.h:224:27
9: _ZN15BruteForceIndexIffE9topKQueryEPKvmP17VecSimQueryParams
at /build/deps/VectorSimilarity/src/VecSim/algorithms/brute_force/brute_force.h:258:14
10: VecSimIndex_TopKQuery
at /build/deps/VectorSimilarity/src/VecSim/vec_sim.cpp:225:57
11: prepareResults
at /build/src/hybrid_reader.c:224:11
HR_ReadKnnUnsorted
at /build/src/hybrid_reader.c:315:5
12: rpidxNext
at /build/src/result_processor.c:94:10
13: rpMetricsNext
at /build/src/result_processor.c:271:8
14: rpsortNext_innerLoop
at /build/src/result_processor.c:400:12
rpsortNext_Accum
at /build/src/result_processor.c:496:16
15: rploaderNext
at /build/src/result_processor.c:674:12
16: sendChunk
at /build/src/aggregate/aggregate_exec.c:254:8
17: AREQ_Execute
at /build/src/aggregate/aggregate_exec.c:329:3
18: execCommandCommon
at /build/src/aggregate/aggregate_exec.c:455:5
RSSearchCommand
at /build/src/aggregate/aggregate_exec.c:470:10
19: RedisModuleCommandDispatcher
at /usr/src/redis/src/module.c:695:5
20: call
at /usr/src/redis/src/server.c:3750:5
21: processCommand
at /usr/src/redis/src/server.c:4297:9
22: processCommandAndResetClient
at /usr/src/redis/src/networking.c:2105:9
processInputBuffer
at /usr/src/redis/src/networking.c:2206:17
23: callHandler
at /usr/src/redis/src/connhelpers.h:79:18
connSocketEventHandler
at /usr/src/redis/src/connection.c:295:14
24: aeProcessEvents
at /usr/src/redis/src/ae.c:427:17
25: aeMain
at /usr/src/redis/src/ae.c:487:9
26: main
at /usr/src/redis/src/server.c:6474:5
27: __libc_start_main
28: _start
search_version
search_version:2.6.13
search_redis_version:6.2.13 - oss
search_index
search_number_of_indexes:1
search_fields_statistics
search_fields_text:Text=2
search_fields_vector:Vector=1,Flat=1
search_dialect_statistics
search_dialect_1:0
search_dialect_2:1
search_dialect_3:0
search_runtime_configurations
search_concurrent_mode:OFF
search_enableGC:ON
search_minimal_term_prefix:2
search_maximal_prefix_expansions:200
search_query_timeout_ms:500
search_timeout_policy:return
search_cursor_read_size:1000
search_cursor_max_idle_time:300000
search_max_doc_table_size:1000000
search_max_search_results:1000000
search_max_aggregate_results:-1
search_search_pool_size:20
search_index_pool_size:8
search_gc_scan_size:100
search_min_phonetic_term_length:3
------ FAST MEMORY TEST ------
1:M 29 Jan 2024 09:13:52.060 # Bio thread for job type #0 terminated
1:M 29 Jan 2024 09:13:52.060 # Bio thread for job type #1 terminated
1:M 29 Jan 2024 09:13:52.060 # Bio thread for job type #2 terminated
*** Preparing to test memory region 560db9fbf000 (2281472 bytes)
*** Preparing to test memory region 560dbad7f000 (270336 bytes)
*** Preparing to test memory region 7f19cb5dd000 (7340032 bytes)
*** Preparing to test memory region 7f19cd438000 (6291456 bytes)
*** Preparing to test memory region 7f19ce608000 (13107200 bytes)
*** Preparing to test memory region 7f19d07e1000 (3145728 bytes)
*** Preparing to test memory region 7f19d0e55000 (2621440 bytes)
*** Preparing to test memory region 7f19d10d6000 (8388608 bytes)
*** Preparing to test memory region 7f19d18d7000 (8388608 bytes)
*** Preparing to test memory region 7f19d20d8000 (8388608 bytes)
*** Preparing to test memory region 7f19d28d9000 (8388608 bytes)
*** Preparing to test memory region 7f19d3378000 (8388608 bytes)
*** Preparing to test memory region 7f19d3b79000 (8388608 bytes)
*** Preparing to test memory region 7f19d47fa000 (24576 bytes)
*** Preparing to test memory region 7f19d4800000 (8388608 bytes)
*** Preparing to test memory region 7f19d506e000 (24576 bytes)
*** Preparing to test memory region 7f19d5244000 (16384 bytes)
*** Preparing to test memory region 7f19d5266000 (16384 bytes)
*** Preparing to test memory region 7f19d555a000 (16384 bytes)
*** Preparing to test memory region 7f19d573b000 (8192 bytes)
*** Preparing to test memory region 7f19d576b000 (4096 bytes)
.O.O.xterm-256colorxterm-256colorO.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
------ DUMPING CODE AROUND EIP ------
Symbol: _Z35FP32_InnerProductSIMD16Ext_SSE_implPKvS0_m (base: 0x7f19d4570820)
Module: /usr/lib/redis/modules/redisearch.so (base 0x7f19d4379000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x7f19d4570820 -D -b binary -m i386:x86-64 /tmp/dump.bin
1:M 29 Jan 2024 09:13:52.502 # dump of function (hexdump of 137 bytes):
488d04974839c7736fc5f057c90f1f00c5f8101fc5e059064883c7404883c640c5f81067d0c5f8106fe0c5f81056f0c5f858c1c5d8594ed0c5f858c1c5d0594ee0c5f858c1c5f8104ff0c5f059cac5f058c84839f877b9c5f0c6c155c5fa58c1c5f015d1c5fa58d2c5f0c6c1ffc5ea58c0c3660f1f440000c5f857c0c30f1f004883ec08e877ffffff
Function at 0x7f19d4570820 is _Z35FP32_InnerProductSIMD16Ext_SSE_implPKvS0_m
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
If a Redis module was involved, please open in the module's repo instead.
Suspect RAM error? Use redis-server --test-memory to verify it.
Some other issues could be detected by redis-server --check-system
The text was updated successfully, but these errors were encountered: