High System Load and abrtd Activity When Accessing BE Compaction Status on StarRocks 3.2.6

[timlincool]

Steps to reproduce the behavior (Required)

1. Navigate to the compaction status page via http://<ip>/api/compaction/show?tablet_id=
2. Monitor system load and system logs as the load increases.

Expected behavior (Required)

The compaction status page should load without causing high system load or failures, displaying the current compaction status of the specified tablet.

Real behavior (Required)

Unable to view the compaction status page. The system's load spikes to 100%, and the abrtd service starts collecting crash reports, leading to significant performance degradation. Here are the relevant logs and system outputs:

• Systemd status of abrtd:

abrtd.service - ABRT Automated Bug Reporting Tool
Loaded: loaded (/usr/lib/systemd/system/abrtd.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2024-04-19 10:01:29 CST; 2 weeks 6 days ago
Main PID: 864 (abrtd)
CGroup: /system.slice/abrtd.service
└─864 /usr/sbin/abrtd -d -s

Apr 19 10:01:29 host32 systemd[1]: Started ABRT Automated Bug Reporting Tool.
Apr 19 10:01:31 host32 abrtd[864]: Init complete, entering main loop
May 09 10:14:15 host32 abrt-server[18280]: Executable '/root/StarRocks-3.2.6/be/lib/starrocks_be' doesn't belong to any package and ProcessUnpackaged is set to 'no'
May 09 10:14:15 host32 abrt-server[18280]: 'post-create' on '/var/spool/abrt/ccpp-2024-05-09-10:13:32-18302' exited with 1
May 09 10:14:15 host32 abrt-server[18280]: Deleting problem directory '/var/spool/abrt/ccpp-2024-05-09-10:13:32-18302'
May 09 10:36:03 host32 abrt-server[22381]: Executable '/root/StarRocks-3.2.4/be/lib/starrocks_be' doesn't belong to any package and ProcessUnpackaged is set to 'no'
May 09 10:36:03 host32 abrt-server[22381]: 'post-create' on '/var/spool/abrt/ccpp-2024-05-09-10:30:36-18737' exited with 1
May 09 10:36:03 host32 abrt-server[22381]: Deleting problem directory '/var/spool/abrt/ccpp-2024-05-09-10:30:36-18737'

• Error message from dmesg:
  [1728712.551536] http_server[18769]: segfault at 48 ip 0000000005122e60 sp 00002b2af486f0e0 error 4 in starrocks_be[400000+a1b2000]

• Error message from be.out:
  *** Aborted at 1715220810 (unix time) try "date -d @1715220810" if you are using GNU date ***
  PC: @ 0x5122e60 starrocks::TabletUpdates::get_compaction_status()
  *** SIGSEGV (@0x48) received by PID 18302 (TID 0x2b2af4876700) from PID 72; stack trace: ***
  @ 0x67749a2 google::(anonymous namespace)::FailureSignalHandler()
  @ 0x2b2aa7fb2630 (unknown)
  @ 0x5122e60 starrocks::TabletUpdates::get_compaction_status()
  @ 0x33b7f0e starrocks::CompactionAction::_handle_show_compaction()
  @ 0x33bcdd5 starrocks::CompactionAction::handle()
  @ 0x6806de7 evhttp_handle_request
  @ 0x6807a93 bufferevent_readcb
  @ 0x67f41d2 event_process_active_single_queue
  @ 0x67f490f event_base_loop
  @ 0x3386aa4 _ZZN9starrocks12EvHttpServer5startEvENKUlvE_clEv
  @ 0x8bb3010 execute_native_thread_routine
  @ 0x2b2aa7faaea5 start_thread
  @ 0x2b2aa8be5b0d __clone
  @ 0x0 (unknown)

StarRocks version (Required)

• Version 3.2.6-2585333 , confirmed by executing SQL select current_version().

Additional context

The issue appears to resolve itself temporarily, as subsequent attempts to access the page successfully display the compaction status without issues, though it's unclear if this is related to the error logs found in be.out, which include messages about segmentation faults during compaction status retrieval.