Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Redis 6.2.6 with TS 1.6.17 crashes with signal: 11 and si_code: 1 #1577

Open
namirsab opened this issue Jan 30, 2024 · 3 comments
Open

Redis 6.2.6 with TS 1.6.17 crashes with signal: 11 and si_code: 1 #1577

namirsab opened this issue Jan 30, 2024 · 3 comments

Comments

@namirsab
Copy link

Hey Redis TS team.

We have a problem with our Redis TS cluster: it crashes with no apparent reason (and only one node crashes).
We are running Redis 6.2.6 and RedisTS 1.6.17

We have 15 pods running on Kubernetes. For some reason, the pod with index 0 always crashes with the following report.

Can you see what is the module doing when it crashes? This could give us a hint on what's the problem.

=== REDIS BUG REPORT START: Cut & paste starting from here ===
1:M 30 Jan 2024 07:38:22.205 # Redis 6.2.6 crashed by signal: 11, si_code: 1
1:M 30 Jan 2024 07:38:22.205 # Accessing address: 0x14
1:M 30 Jan 2024 07:38:22.205 # Crashed running the instruction at: 0x7f326e85868d

------ STACK TRACE ------
EIP:
/usr/lib/redis/modules/redistimeseries.so(+0x4e68d)[0x7f326e85868d]

Backtrace:
/lib/x86_64-linux-gnu/libpthread.so.0(+0x14140)[0x7f326ea5d140]
/usr/lib/redis/modules/redistimeseries.so(+0x4e68d)[0x7f326e85868d]
/usr/lib/redis/modules/redistimeseries.so(+0x2365f)[0x7f326e82d65f]
/usr/lib/redis/modules/redistimeseries.so(+0x3252f)[0x7f326e83c52f]
/usr/lib/redis/modules/redistimeseries.so(+0x3280f)[0x7f326e83c80f]
/usr/lib/redis/modules/redistimeseries.so(+0x32d9e)[0x7f326e83cd9e]
/usr/lib/redis/modules/redistimeseries.so(event_base_loop+0x2c8)[0x7f326e83d4f9]
/usr/lib/redis/modules/redistimeseries.so(+0x2618d)[0x7f326e83018d]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7)[0x7f326ea51ea7]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f326e981def]

------ REGISTERS ------
1:M 30 Jan 2024 07:38:22.205 # 
RAX:00007f326e858670 RBX:0000000000000000
RCX:0000000000000000 RDX:0000000000000000
RDI:00007f27140021e0 RSI:00007f326d3154ac
RBP:00007f27140021e0 RSP:00007f326d3154a0
R8 :0000000000000000 R9 :00007f2723231708
R10:00007f271400b090 R11:0000000000000000
R12:00007fffabaa15be R13:00007fffabaa15bf
R14:00007f326d315780 R15:0000000000802000
RIP:00007f326e85868d EFL:0000000000010206
CSGSFS:002b000000000033
1:M 30 Jan 2024 07:38:22.205 # (00007f326d3154af) -> 000055d536f6c620
1:M 30 Jan 2024 07:38:22.205 # (00007f326d3154ae) -> 00007f2714004620
1:M 30 Jan 2024 07:38:22.205 # (00007f326d3154ad) -> 00007f326e83fc91
1:M 30 Jan 2024 07:38:22.205 # (00007f326d3154ac) -> 00007f326d315540
1:M 30 Jan 2024 07:38:22.205 # (00007f326d3154ab) -> 000055d536f6c620
1:M 30 Jan 2024 07:38:22.205 # (00007f326d3154aa) -> 00007f27140023f0
1:M 30 Jan 2024 07:38:22.205 # (00007f326d3154a9) -> 00007f326e83c52f
1:M 30 Jan 2024 07:38:22.205 # (00007f326d3154a8) -> 00007f326d315560
1:M 30 Jan 2024 07:38:22.205 # (00007f326d3154a7) -> 00007f326e87c8f0
1:M 30 Jan 2024 07:38:22.205 # (00007f326d3154a6) -> 0000000000000008
1:M 30 Jan 2024 07:38:22.205 # (00007f326d3154a5) -> 00007f326e82d65f
1:M 30 Jan 2024 07:38:22.205 # (00007f326d3154a4) -> 00007f27140023b0
1:M 30 Jan 2024 07:38:22.205 # (00007f326d3154a3) -> 0000000000000004
1:M 30 Jan 2024 07:38:22.205 # (00007f326d3154a2) -> 000055d536f6c620
1:M 30 Jan 2024 07:38:22.205 # (00007f326d3154a1) -> 000000006e84ff7e
1:M 30 Jan 2024 07:38:22.205 # (00007f326d3154a0) -> 00007f27140023b0

------ INFO OUTPUT ------
# Server
redis_version:6.2.6
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:9c335ca9779faba5
redis_mode:cluster
os:Linux 5.4.0-139-generic x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:10.2.1
process_id:1
process_supervised:no
run_id:3d5342ea021f7f25d90b26e1a496438d93fb36fd
tcp_port:6380
server_time_usec:1706600302204185
uptime_in_seconds:507
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:12101486
executable:/data/redis-server
config_file:/redis/config/redis.conf
io_threads_active:0

# Clients
connected_clients:23
cluster_connections:28
maxclients:10000
client_recent_max_input_buffer:40
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0

# Memory
used_memory:40573839496
used_memory_human:37.79G
used_memory_rss:41652232192
used_memory_rss_human:38.79G
used_memory_peak:40574381064
used_memory_peak_human:37.79G
used_memory_peak_perc:100.00%
used_memory_overhead:7480456
used_memory_startup:1576640
used_memory_dataset:40566359040
used_memory_dataset_perc:99.99%
allocator_allocated:40573916560
allocator_active:40574431232
allocator_resident:41648312320
total_system_memory:270158614528
total_system_memory_human:251.60G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.00
allocator_frag_bytes:514672
allocator_rss_ratio:1.03
allocator_rss_bytes:1073881088
rss_overhead_ratio:1.00
rss_overhead_bytes:3919872
mem_fragmentation_ratio:1.03
mem_fragmentation_bytes:1078399560
mem_not_counted_for_evict:762
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:471520
mem_aof_buffer:768
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

# Persistence
loading:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:10712368
rdb_bgsave_in_progress:0
rdb_last_save_time:1706599795
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:1
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
aof_current_size:34703786180
aof_base_size:34700606036
aof_pending_rewrite:0
aof_buffer_length:0
aof_rewrite_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:0

# Stats
total_connections_received:2769
total_commands_processed:2959254
instantaneous_ops_per_sec:173
total_net_input_bytes:138682615
total_net_output_bytes:85493746
instantaneous_input_kbps:446.09
instantaneous_output_kbps:226.83
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:1
evicted_keys:0
keyspace_hits:109500
keyspace_misses:12
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:1290358
dump_payload_sanitizations:0
total_reads_processed:44133
total_writes_processed:38661
io_threaded_reads_processed:0
io_threaded_writes_processed:0

# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:63171f280833940da5c40761e85222ca7864a644
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:21.546695
used_cpu_user:315.634839
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
used_cpu_sys_main_thread:0.004425
used_cpu_user_main_thread:0.000000

# Modules
module:name=timeseries,ver=10613,api=1,filters=0,usedby=[],using=[],options=[handle-io-errors]

# Commandstats
cmdstat_ts.madd:calls=0,usec=0,usec_per_call=0.00,rejected_calls=3,failed_calls=0
cmdstat_ts.range:calls=21,usec=659,usec_per_call=31.38,rejected_calls=1165,failed_calls=12
cmdstat_cluster:calls=2546,usec=462960,usec_per_call=181.84,rejected_calls=189,failed_calls=0
cmdstat_timeseries.REFRESHCLUSTER:calls=1,usec=38,usec_per_call=38.00,rejected_calls=1,failed_calls=0
cmdstat_info:calls=4927,usec=122945,usec_per_call=24.95,rejected_calls=0,failed_calls=0
cmdstat_ts.add:calls=30206,usec=1223215,usec_per_call=40.50,rejected_calls=1281946,failed_calls=0
cmdstat_timeseries.HELLO:calls=2687,usec=79424,usec_per_call=29.56,rejected_calls=4355,failed_calls=2687
cmdstat_config:calls=5,usec=88,usec_per_call=17.60,rejected_calls=0,failed_calls=0
cmdstat_client:calls=2489,usec=5232,usec_per_call=2.10,rejected_calls=0,failed_calls=0
cmdstat_ping:calls=6,usec=6,usec_per_call=1.00,rejected_calls=0,failed_calls=0

# Errorstats
errorstat_CLUSTERDOWN:count=26767
errorstat_ERR:count=12
errorstat_ERRCLUSTER:count=2687
errorstat_LOADING:count=4545
errorstat_MOVED:count=1256347

# Cluster
cluster_enabled:1

# Keyspace
db0:keys=109573,expires=0,avg_ttl=0

------ CLIENT LIST OUTPUT ------
id=20 addr=100.64.3.104:42002 laddr=100.64.4.228:6380 fd=22 name= age=506 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=timeseries.HELLO user=default redir=-1
id=21 addr=100.64.6.22:42348 laddr=100.64.4.228:6380 fd=23 name= age=506 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=timeseries.HELLO user=default redir=-1
id=22 addr=100.64.0.52:34184 laddr=100.64.4.228:6380 fd=24 name= age=506 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=timeseries.HELLO user=default redir=-1
id=23 addr=100.64.33.211:59036 laddr=100.64.4.228:6380 fd=25 name= age=506 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=timeseries.HELLO user=default redir=-1
id=24 addr=100.64.33.210:59270 laddr=100.64.4.228:6380 fd=26 name= age=506 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=timeseries.HELLO user=default redir=-1
id=1675 addr=100.64.32.228:54002 laddr=100.64.4.228:6380 fd=63 name= age=79 idle=79 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1
id=25 addr=100.64.34.154:46332 laddr=100.64.4.228:6380 fd=27 name= age=506 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=timeseries.HELLO user=default redir=-1
id=26 addr=100.64.2.165:60704 laddr=100.64.4.228:6380 fd=28 name= age=506 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=timeseries.HELLO user=default redir=-1
id=27 addr=100.64.2.164:40026 laddr=100.64.4.228:6380 fd=29 name= age=506 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=timeseries.HELLO user=default redir=-1
id=28 addr=100.64.34.155:42562 laddr=100.64.4.228:6380 fd=30 name= age=506 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=timeseries.HELLO user=default redir=-1
id=29 addr=100.64.5.163:44372 laddr=100.64.4.228:6380 fd=31 name= age=506 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61448 events=r cmd=timeseries.HELLO user=default redir=-1
id=30 addr=100.64.6.21:57270 laddr=100.64.4.228:6380 fd=32 name= age=506 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61448 events=r cmd=timeseries.HELLO user=default redir=-1
id=33 addr=100.64.25.222:52200 laddr=100.64.4.228:6380 fd=33 name= age=506 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20520 events=r cmd=ts.add user=default redir=-1
id=44 addr=100.64.20.171:53832 laddr=100.64.4.228:6380 fd=36 name= age=497 idle=191 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20520 events=r cmd=ts.madd user=default redir=-1
id=2773 addr=100.64.14.237:55000 laddr=100.64.4.228:6380 fd=64 name=ioredis-cluster(refresher) age=1 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=cluster user=default redir=-1
id=204 addr=100.64.7.151:40900 laddr=100.64.4.228:6380 fd=34 name= age=219 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=timeseries.HELLO user=default redir=-1
id=7 addr=100.64.22.199:34056 laddr=100.64.4.228:6380 fd=18 name= age=507 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20520 events=r cmd=ts.add user=default redir=-1
id=1402 addr=100.64.27.169:45926 laddr=100.64.4.228:6380 fd=62 name= age=101 idle=101 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1
id=1750 addr=100.64.14.209:56000 laddr=100.64.4.228:6380 fd=65 name= age=71 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=ping user=default redir=-1
id=2097 addr=100.64.13.246:41608 laddr=100.64.4.228:6380 fd=67 name= age=47 idle=47 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=cluster user=default redir=-1
id=11 addr=100.64.14.237:42468 laddr=100.64.4.228:6380 fd=17 name= age=506 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61472 events=r cmd=ts.add user=default redir=-1
id=18 addr=100.64.4.229:51948 laddr=100.64.4.228:6380 fd=20 name= age=506 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61448 events=r cmd=timeseries.HELLO user=default redir=-1
id=19 addr=100.64.7.150:40158 laddr=100.64.4.228:6380 fd=21 name= age=506 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61448 events=r cmd=timeseries.HELLO user=default redir=-1

------ MODULES INFO OUTPUT ------

------ FAST MEMORY TEST ------
1:M 30 Jan 2024 07:38:22.206 # main thread terminated
1:M 30 Jan 2024 07:38:22.206 # Bio thread for job type #0 terminated
1:M 30 Jan 2024 07:38:22.206 # Bio thread for job type #1 terminated
1:M 30 Jan 2024 07:38:22.206 # Bio thread for job type #2 terminated

Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

------ DUMPING CODE AROUND EIP ------
Symbol: (null) (base: (nil))
Module: /usr/lib/redis/modules/redistimeseries.so (base 0x7f326e80a000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=(nil) -D -b binary -m i386:x86-64 /tmp/dump.bin
------

=== REDIS BUG REPORT END. Make sure to include from START to END. ===

Thanks in advance!
@namirsab namirsab changed the title Redis crashes Redis 6.2.6 with TS 1.6.17 crashes with signal: 11 and si_code: 1 Jan 30, 2024
@LiorKogan
Copy link
Member

LiorKogan commented Jan 30, 2024 •
edited

Sorry, but we are no longer maintaining 1.6

@namirsab
Copy link
Author

Ah okay. But can we upgrade without problem to the currently maintained version with the same aof files?

@LiorKogan
Copy link
Member

LiorKogan commented Jan 30, 2024 •
edited

I suggest to test it. It should work, but we cannot guarantee it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@namirsab @LiorKogan