Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ObSQL Audit 无锁队列扩缩容 #1556

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Conversation

fhkong
Copy link
Contributor

@fhkong fhkong commented Sep 7, 2023

Task Description

ObMySQLRequestManager中使用 一个无锁的 ObRaQueue 记录每一个 audit 记录的指针。ObRaQueue 是无锁实现,但内存是静态申请的,导致在小租户下内存占用较大 (80M) 。目前仅在meta租户和mini mode下使用较小的queue size初始化以减少内存占用,但不能处理租户内存扩缩容场景,希望将此queue做为动态大小。

Solution Description

设计目标:

  1. 向上提供随机读取的接口;
  2. 支持内存动态扩缩容;
  3. 无锁实现高并发线程安全;

具体细节:
image
使用一个两层队列的结构,通过构造多个小的静态队列来组成一个大的动态队列。

  1. 节点内存:一级队列存放二级队列的指针和该二级队列上的引用计数,占用16B;二级队列存在每个Audit记录的指针,占用8B;
  2. 扩缩容:每次操作的粒度为一个小队列的大小(Evict Batch)。Batch设置的越小,队列的内存可控粒度越小,但malloc/free操作越频繁;

Passed Regressions

oceanbase/unittest/observer/test_ob_ra_queue.cpp
oceanbase/unittest/observer/test_ob_tl_queue.cpp
oceanbase/unittest/observer/test_ob_mysql_request_manager.cpp

Upgrade Compatibility

Other Information

Release Note

@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


fhkong seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants