[BUG] After creating index Redis Stack 7.2.3 crashed

[ghczko]

Hello,

I am facing a crashing Redis stack when creating index for searching in JSON. In Redis stack I have around 100k JSON records.
When I create a index with NodeJS library the Redis Stack crash

13509:M 28 Mar 2024 12:50:43.973 * <module> Scanning index pool:index in background
thread '<unnamed>' panicked at json_path/src/json_node.rs:238:24:
called `Option::unwrap()` on a `None` value
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
fatal runtime error: failed to initiate panic, error 5
=== REDIS BUG REPORT START: Cut & paste starting from here ===
13509:M 28 Mar 2024 12:50:44.169 # Redis 7.2.3 crashed by signal: 6, si_code: 0
13509:M 28 Mar 2024 12:50:44.169 # Crashed running the instruction at: 0x1872060dc

------ STACK TRACE ------
EIP:
0   libsystem_kernel.dylib              0x00000001872060dc __pthread_kill + 8

Backtrace:
0   libsystem_platform.dylib            0x000000018726da24 _sigtramp + 56
1   libsystem_pthread.dylib             0x000000018723dcc0 pthread_kill + 288
2   libsystem_c.dylib                   0x0000000187149a40 abort + 180
3   rejson.so                           0x0000000101f62f9c _ZN108_$LT$std..sys..unix..locks..pthread_mutex..AllocatedMutex$u20$as$u20$std..sys_common..lazy_box..LazyInit$GT$4init17hc1d68b7de7fd100eE + 0
4   rejson.so                           0x0000000101f616cc _ZN3std12backtrace_rs9symbolize6Symbol4name17hc8326d09aeb77feaE + 0
5   rejson.so                           0x0000000101f61610 _ZN3std9panicking20rust_panic_with_hook17h84c8637cb6e56008E + 596
6   rejson.so                           0x0000000101f61368 _ZN3std9panicking19begin_panic_handler28_$u7b$$u7b$closure$u7d$$u7d$17h25482adda06c7b7fE + 144
7   rejson.so                           0x0000000101f60518 _ZN3std10sys_common9backtrace15output_filename17h60eb3d725954b704E + 0
8   rejson.so                           0x0000000101f61108 _ZN90_$LT$std..panicking..begin_panic_handler..PanicPayload$u20$as$u20$core..panic..BoxMeUp$GT$8take_box17hdcefff1827414461E + 0
9   rejson.so                           0x0000000101f8cca8 _ZN4core9panicking18panic_nounwind_fmt17ha4f6bbf8fe9fb5aeE + 0
10  rejson.so                           0x0000000101f8cd18 _ZN4core9panicking14panic_nounwind17ha756411eedbc9594E + 0
11  rejson.so                           0x0000000101e72bb4 _ZN9json_path9json_node87_$LT$impl$u20$json_path..select_value..SelectValue$u20$for$u20$ijson..value..IValue$GT$10get_double17h5b3fea498d2eb9bfE + 0
12  rejson.so                           0x0000000101e3ee98 JSONAPI_getInt + 72
13  redisearch.so                       0x000000010191d004 JSON_StoreInDocField + 208
14  redisearch.so                       0x000000010191d4b4 JSON_LoadDocumentField + 424
15  redisearch.so                       0x00000001019046e0 Document_LoadSchemaFieldJson + 304
16  redisearch.so                       0x0000000101947ed8 IndexSpec_UpdateDoc + 392
17  redisearch.so                       0x0000000101949f9c Indexes_ScanProc + 308
18  redis-server                        0x0000000100633998 moduleScanCallback + 296
19  redis-server                        0x00000001005350b4 dictScan + 328
20  redis-server                        0x0000000100633840 RM_Scan + 88
21  redisearch.so                       0x0000000101945ee8 Indexes_ScanAndReindexTask + 276
22  redisearch.so                       0x00000001018de34c thread_do + 752
23  libsystem_pthread.dylib             0x000000018723e034 _pthread_start + 136
24  libsystem_pthread.dylib             0x0000000187238e3c thread_start + 8

------ REGISTERS ------
13509:M 28 Mar 2024 12:50:44.176 #
x0:0000000000000000 x1:0000000000000000 x2:0000000000000000 x3:0000000000000000
x4:0000000000000000 x5:0000000000000001 x6:0000000000000005 x7:0000000000000150
x8:27bfbc5ce9aa4c4a x9:27bfbc5d99d73c4a x10:0000000000110000 x11:0000000102067970
x12:0000000000000000 x13:0000000000000000 x14:0000000000000000 x15:0000000187243cc4
x16:0000000000000148 x17:00000001e6a8f340 x18:0000000000000000 x19:0000000000000006
x20:00000001707d7000 x21:0000000000001303 x22:00000001707d70e0 x23:000000010206c340
x24:0000000187246620 x25:0000000101fed590 x26:0000000000000007 x27:0000000101c03dd0
x28:000000000000000a fp:00000001707d6770 lr:000000018723dcc0
sp:00000001707d6750 pc:00000001872060dc cpsr:40001000

13509:M 28 Mar 2024 12:50:44.176 # (00000001707d675f) -> 0000000101f616cc
13509:M 28 Mar 2024 12:50:44.176 # (00000001707d675e) -> 00000001707d6820
13509:M 28 Mar 2024 12:50:44.176 # (00000001707d675d) -> 315b800101f62f9c
13509:M 28 Mar 2024 12:50:44.176 # (00000001707d675c) -> 00000001707d67c0
13509:M 28 Mar 2024 12:50:44.176 # (00000001707d675b) -> 0000000000000001
13509:M 28 Mar 2024 12:50:44.176 # (00000001707d675a) -> 00000001020678c0
13509:M 28 Mar 2024 12:50:44.176 # (00000001707d6759) -> 00000000fffff9df
13509:M 28 Mar 2024 12:50:44.176 # (00000001707d6758) -> 0000000000000000
13509:M 28 Mar 2024 12:50:44.176 # (00000001707d6757) -> ffffffff01f5e65c
13509:M 28 Mar 2024 12:50:44.176 # (00000001707d6756) -> 00000001707d67c0
13509:M 28 Mar 2024 12:50:44.176 # (00000001707d6755) -> ce2b000187149a40
13509:M 28 Mar 2024 12:50:44.176 # (00000001707d6754) -> 00000001707d67b0
13509:M 28 Mar 2024 12:50:44.176 # (00000001707d6753) -> 00000001707d7000
13509:M 28 Mar 2024 12:50:44.176 # (00000001707d6752) -> 00000001707d6798
13509:M 28 Mar 2024 12:50:44.176 # (00000001707d6751) -> 00000001707d68f0
13509:M 28 Mar 2024 12:50:44.176 # (00000001707d6750) -> 000000010206c180

------ INFO OUTPUT ------
# Server
redis_version:7.2.3
redis_git_sha1:7f4bae81
redis_git_dirty:0
redis_build_id:a97542f11b1bfce4
redis_mode:standalone
os:Darwin 23.2.0 arm64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:kqueue
atomicvar_api:c11-builtin
gcc_version:4.2.1
process_id:13509
process_supervised:no
run_id:2110b56b87d13088e006830fcefdc9df242de31b
tcp_port:6379
server_time_usec:1711626644167971
uptime_in_seconds:37
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:350612
executable:/opt/homebrew/Caskroom/redis-stack-server/7.2.0-v6/bin/redis-server
config_file:/opt/homebrew/Caskroom/redis-stack-server/7.2.0-v6/etc/redis-stack.conf
io_threads_active:0
listener0:name=tcp,bind=*,bind=-::*,port=6379

# Clients
connected_clients:2
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:16
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
total_blocking_keys:0
total_blocking_keys_on_nokey:0

# Memory
used_memory:89174336
used_memory_human:85.04M
used_memory_rss:93552640
used_memory_rss_human:89.22M
used_memory_peak:89174336
used_memory_peak_human:85.04M
used_memory_peak_perc:102.29%
used_memory_overhead:7427472
used_memory_startup:1435440
used_memory_dataset:81746864
used_memory_dataset_perc:93.17%
allocator_allocated:87174880
allocator_active:93520896
allocator_resident:93520896
total_system_memory:17179869184
total_system_memory_human:16.00G
used_memory_lua:31744
used_memory_vm_eval:31744
used_memory_lua_human:31.00K
used_memory_scripts_eval:0
number_of_cached_scripts:0
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:32768
used_memory_vm_total:64512
used_memory_vm_total_human:63.00K
used_memory_functions:216
used_memory_scripts:216
used_memory_scripts_human:216B
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.07
allocator_frag_bytes:6346016
allocator_rss_ratio:1.00
allocator_rss_bytes:0
rss_overhead_ratio:1.00
rss_overhead_bytes:31744
mem_fragmentation_ratio:1.07
mem_fragmentation_bytes:6377760
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_total_replication_buffers:0
mem_clients_slaves:0
mem_clients_normal:3648
mem_cluster_links:0
mem_aof_buffer:0
mem_allocator:libc
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

# Persistence
loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:4
rdb_bgsave_in_progress:0
rdb_last_save_time:1711626607
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_saves:0
rdb_last_cow_size:0
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:123487
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:0
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:3
total_commands_processed:22
instantaneous_ops_per_sec:6
total_net_input_bytes:2334
total_net_output_bytes:177296
total_net_repl_input_bytes:0
total_net_repl_output_bytes:0
instantaneous_input_kbps:1.07
instantaneous_output_kbps:0.05
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
evicted_clients:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:254007
keyspace_misses:2
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:3
dump_payload_sanitizations:0
total_reads_processed:13
total_writes_processed:15
io_threaded_reads_processed:0
io_threaded_writes_processed:0
reply_buffer_shrinks:3
reply_buffer_expands:0
eventloop_cycles:370
eventloop_duration_sum:60962
eventloop_duration_cmd_sum:17170
instantaneous_eventloop_cycles_per_sec:13
instantaneous_eventloop_duration_usec:118
acl_access_denied_auth:0
acl_access_denied_cmd:0
acl_access_denied_key:0
acl_access_denied_channel:0

# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:91623a63f06557c641a812556aac70046b3eeaf5
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:0.096157
used_cpu_user:0.474580
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000

# Modules
module:name=search,ver=20809,api=1,filters=0,usedby=[],using=[ReJSON],options=[handle-io-errors]
module:name=bf,ver=20608,api=1,filters=0,usedby=[],using=[],options=[]
module:name=RedisCompat,ver=1,api=1,filters=0,usedby=[],using=[],options=[]
module:name=ReJSON,ver=20607,api=1,filters=0,usedby=[search],using=[],options=[handle-io-errors]
module:name=redisgears_2,ver=20014,api=1,filters=0,usedby=[],using=[],options=[]
module:name=timeseries,ver=11009,api=1,filters=0,usedby=[],using=[],options=[]

# Commandstats
cmdstat_config|get:calls=1,usec=6,usec_per_call=6.00,rejected_calls=0,failed_calls=0
cmdstat_quit:calls=1,usec=0,usec_per_call=0.00,rejected_calls=0,failed_calls=0
cmdstat_client|setinfo:calls=2,usec=1,usec_per_call=0.50,rejected_calls=0,failed_calls=0
cmdstat_client|setname:calls=1,usec=2,usec_per_call=2.00,rejected_calls=0,failed_calls=0
cmdstat_dbsize:calls=1,usec=1339,usec_per_call=1339.00,rejected_calls=0,failed_calls=0
cmdstat_FT.SEARCH:calls=0,usec=0,usec_per_call=0.00,rejected_calls=1,failed_calls=0
cmdstat_info:calls=5,usec=1196,usec_per_call=239.20,rejected_calls=0,failed_calls=0
cmdstat_FT.CREATE:calls=2,usec=2596,usec_per_call=1298.00,rejected_calls=0,failed_calls=0
cmdstat_scan:calls=1,usec=10625,usec_per_call=10625.00,rejected_calls=0,failed_calls=0
cmdstat_set:calls=2,usec=86,usec_per_call=43.00,rejected_calls=0,failed_calls=0
cmdstat_unlink:calls=2,usec=9,usec_per_call=4.50,rejected_calls=0,failed_calls=0
cmdstat_get:calls=2,usec=1326,usec_per_call=663.00,rejected_calls=0,failed_calls=0
cmdstat_FT.DROPINDEX:calls=2,usec=69,usec_per_call=34.50,rejected_calls=0,failed_calls=2

# Errorstats
errorstat_LOADING:count=1
errorstat_Unknown:count=2

# Latencystats
latency_percentiles_usec_config|get:p50=6.015,p99=6.015,p99.9=6.015
latency_percentiles_usec_quit:p50=0.001,p99=0.001,p99.9=0.001
latency_percentiles_usec_client|setinfo:p50=0.001,p99=1.003,p99.9=1.003
latency_percentiles_usec_client|setname:p50=2.007,p99=2.007,p99.9=2.007
latency_percentiles_usec_dbsize:p50=1343.487,p99=1343.487,p99.9=1343.487
latency_percentiles_usec_info:p50=8.031,p99=1122.303,p99.9=1122.303
latency_percentiles_usec_FT.CREATE:p50=290.815,p99=2310.143,p99.9=2310.143
latency_percentiles_usec_scan:p50=10682.367,p99=10682.367,p99.9=10682.367
latency_percentiles_usec_set:p50=17.023,p99=69.119,p99.9=69.119
latency_percentiles_usec_unlink:p50=1.003,p99=8.031,p99.9=8.031
latency_percentiles_usec_get:p50=9.023,p99=1318.911,p99.9=1318.911
latency_percentiles_usec_FT.DROPINDEX:p50=5.023,p99=64.255,p99.9=64.255

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=123489,expires=0,avg_ttl=0

------ CLIENT LIST OUTPUT ------
id=14 addr=[::1]:49949 laddr=[::1]:6379 fd=12 name= age=36 idle=36 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1824 events=r cmd=FT.SEARCH user=default redir=-1 resp=2 lib-name= lib-ver=
id=15 addr=127.0.0.1:50001 laddr=127.0.0.1:6379 fd=10 name=redisinsight-browser-93b67723 age=25 idle=25 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1824 events=r cmd=scan user=default redir=-1 resp=2 lib-name= lib-ver=

------ MODULES INFO OUTPUT ------
# search_version
search_version:2.8.9
search_redis_version:7.2.3 - oss

# search_index
search_number_of_indexes:2

# search_fields_statistics
search_fields_text:Text=3
search_fields_numeric:Numeric=2,Sortable=2
search_fields_tag:Tag=12

# search_dialect_statistics
search_dialect_1:0
search_dialect_2:0
search_dialect_3:0
search_dialect_4:0

# search_runtime_configurations
search_concurrent_mode:OFF
search_enableGC:ON
search_minimal_term_prefix:2
search_maximal_prefix_expansions:200
search_query_timeout_ms:500
search_timeout_policy:return
search_cursor_read_size:1000
search_cursor_max_idle_time:300000
search_max_doc_table_size:1000000
search_max_search_results:10000
search_max_aggregate_results:10000
search_search_pool_size:20
search_index_pool_size:8
search_gc_scan_size:100
search_min_phonetic_term_length:3

# ReJSON_trace
ReJSON_backtrace:   0: backtrace::capture::Backtrace::new
   1: redis_module::basic_info_command_handler
   2: rejson::__info_func
   3: _modulesCollectInfo
   4: _printCrashReport
   5: _sigsegvHandler
   6: __platform_memmove
   7: __pthread_atfork_prepare_handlers
   8: <unknown>
   9: std::sys::unix::abort_internal
  10: _rust_panic
  11: std::panicking::rust_panic_with_hook
  12: std::panicking::begin_panic_handler::{{closure}}
  13: std::sys_common::backtrace::__rust_end_short_backtrace
  14: _rust_begin_unwind
  15: core::panicking::panic_fmt
  16: core::panicking::panic
  17: json_path::json_node::<impl json_path::select_value::SelectValue for ijson::value::IValue>::get_long
  18: _JSONAPI_getInt
  19: _JSON_StoreInDocField
  20: _JSON_LoadDocumentField
  21: _Document_LoadSchemaFieldJson
  22: _IndexSpec_UpdateDoc
  23: _Indexes_ScanProc
  24: _moduleScanCallback
  25: _dictScan
  26: _RM_Scan
  27: _Indexes_ScanAndReindexTask
  28: _thread_do
  29: __pthread_joiner_wake


# redisgears_2_trace
redisgears_2_backtrace:   0: backtrace::capture::Backtrace::new
   1: redis_module::basic_info_command_handler
   2: redisgears::gears_module::__info_func
   3: _modulesCollectInfo
   4: _printCrashReport
   5: _sigsegvHandler
   6: __platform_memmove
   7: __pthread_atfork_prepare_handlers
   8: <unknown>
   9: std::sys::unix::abort_internal
  10: _rust_panic
  11: std::panicking::rust_panic_with_hook
  12: std::panicking::begin_panic_handler::{{closure}}
  13: std::sys_common::backtrace::__rust_end_short_backtrace
  14: _rust_begin_unwind
  15: core::panicking::panic_fmt
  16: core::panicking::panic
  17: json_path::json_node::<impl json_path::select_value::SelectValue for ijson::value::IValue>::get_long
  18: _JSONAPI_getInt
  19: _JSON_StoreInDocField
  20: _JSON_LoadDocumentField
  21: _Document_LoadSchemaFieldJson
  22: _IndexSpec_UpdateDoc
  23: _Indexes_ScanProc
  24: _moduleScanCallback
  25: _dictScan
  26: _RM_Scan
  27: _Indexes_ScanAndReindexTask
  28: _thread_do
  29: __pthread_joiner_wake


# redisgears_2_UninitialisedBackends
redisgears_2_backend_name:js

------ CONFIG DEBUG OUTPUT ------
replica-read-only yes
io-threads-do-reads no
slave-read-only yes
proto-max-bulk-len 512mb
lazyfree-lazy-server-del no
lazyfree-lazy-expire no
lazyfree-lazy-user-del no
client-query-buffer-limit 1gb
lazyfree-lazy-eviction no
repl-diskless-load disabled
sanitize-dump-payload no
io-threads 1
repl-diskless-sync yes
lazyfree-lazy-user-flush no
list-compress-depth 0
activedefrag no

------ DUMPING CODE AROUND EIP ------
Symbol: __pthread_kill (base: 0x1872060d4)
Module: /usr/lib/system/libsystem_kernel.dylib (base 0x1871fc000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x1872060d4 -D -b binary -m i386:x86-64 /tmp/dump.bin
------
13509:M 28 Mar 2024 12:50:44.200 # dump of function (hexdump of 136 bytes):
102980d2011000d4030100547f2303d5fd7bbfa9fd03009151e0ff97bf030091fd7bc1a8ff0f5fd6c0035fd6700a80d2011000d4030100547f2303d5fd7bbfa9fd03009146e0ff97bf030091fd7bc1a8ff0f5fd6c0035fd6703080d2011000d4030100547f2303d5fd7bbfa9fd0300913be0ff97bf030091fd7bc1a8ff0f5fd6c0035fd664010034

=== REDIS BUG REPORT END. Make sure to include from START to END. ===

[meiravgri]

Hi @ghczko
we suspect it's related to the datatype.
Can you please provide an example of a json record and the command that caused the crash?
Thanks

[ephraimfeldblum]

If there are numbers in the document within the range (INT64_MAX, UINT64_MAX], this issue is known and should be fixed in the next stack release.

[ghczko]

I confirm that the issue is related with range (INT64_MAX, UINT64_MAX). INT64 is limitation in my use case, is it planned to change it?

[mezga0153]

it seems i'm also hitting this bug, i can reproduce it with:

FLUSHALL
JSON.SET test:entry $ '{"bigNumber":9223372036854775808}'
FT.CREATE test:index ON JSON PREFIX 1 test: SCHEMA '$["bigNumber"]' AS bigNumber NUMERIC NOINDEX