Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Opt performance of async log #2602

Merged
merged 2 commits into from
May 27, 2024
Merged

Opt performance of async log #2602

merged 2 commits into from
May 27, 2024

Conversation

chenBright
Copy link
Contributor

What problem does this PR solve?

Issue Number:

Problem Summary:

 thread_num=1 log_type=sync log_size=100 count=462130 duration=2s qps=231065 average_time=4349.05us
 thread_num=8 log_type=sync log_size=100 count=271989 duration=2s qps=135994 average_time=59007.1us
 thread_num=1 log_type=async log_size=100 count=1449393 duration=2s qps=724696 average_time=1383.17us
 thread_num=8 log_type=async log_size=100 count=952119 duration=2s qps=476060 average_time=17001.6us

目前异步日志(完全由异步线程落盘日志)的性能仅仅是同步日志的3倍左右,没达到每秒百万的级别。

通过cpu profiler分析发现,瓶颈在PrintLogPrefix,特别是localtime_r或者localtime_s(有锁)。

What is changed and the side effects?

Changed:

PrintLogPrefix逻辑后移到异步线程运行,加快LOG(xxx)的速度。

优化后,异步日志性能得到了很大的提升,性能是同步日志的十几倍以上。

 thread_num=1 log_type=sync log_size=100 count=612247 duration=2s qps=306123 average_time=3287.74us
 thread_num=8 log_type=sync log_size=100 count=453624 duration=2s qps=226812 average_time=35352.2us
 thread_num=1 log_type=async log_size=100 count=9856504 duration=2s qps=4928252 average_time=203.911us
 thread_num=8 log_type=async log_size=100 count=14547148 duration=2s qps=7273574 average_time=1104.67us

Side effects:

  • Performance effects(性能影响):

  • Breaking backward compatibility(向后兼容性):


Check List:

  • Please make sure your changes are compilable(请确保你的更改可以通过编译).
  • When providing us with a new feature, it is best to add related tests(如果你向我们增加一个新的功能, 请添加相关测试).
  • Please follow Contributor Covenant Code of Conduct.(请遵循贡献者准则).

@wwbmmm
Copy link
Contributor

wwbmmm commented Apr 22, 2024

LGTM

@Superskyyy
Copy link
Member

Hi, let's merge this?

@chenBright
Copy link
Contributor Author

@wwbmmm 这个PR应该可以合了

@wwbmmm wwbmmm merged commit bdc141c into apache:master May 27, 2024
20 checks passed
@chenBright chenBright deleted the opt_async_log branch May 27, 2024 02:12
@wasphin
Copy link
Member

wasphin commented May 27, 2024

这里对日志时间的影响是可接收的吗?

@chenBright
Copy link
Contributor Author

日志时间应该是跟原来一样的,只影响落盘时间。

brpc/src/butil/logging.cc

Lines 1308 to 1314 in bdc141c

info.severity = severity;
info.timestamp = GetTimestamp();
info.file = file;
info.func = func;
info.line = line;
info.content = content.as_string();
info.raw = true;

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants