Allocate Material Above Nodes #1437

zhangyuqin1998 · 2023-06-16T05:57:00Z

Task Description

In the volcano model, the pipeline's working mode makes it difficult for us to analyze the actual execution performance of a certain operator. To clarify the data flow of the operator on the timeline for the purpose of analyzing its actual execution performance, material operators can be inserted before and after the operator to divide the data flow.
close #1454

Solution Description

Using hint to insert material operator, using /*+ blocking('all') */ to insert material operator above all operators, using /*+ blocking('dfo') */ to insert material operator above exchange in operators, using /*+ blocking(4, 5, 6) */ to insert material insert material operators before specific operators.

To achieve this function, we added a TraverseOp named ALLOC_OP in ObLogPlan::plan_traverse_loop . While passing the logical plan tree, we will do some prepare job and decide whether to insert a material operator.

Besides, we can not insert material ops in some situations, such as right child of nested loop join and sub plan filter, and it is difficult to insert material ops for remote plan. A whitelist mechanism based on pattern matching is established to recognize these situations

Passed Regressions

N/A

Upgrade Compatibility

N/A

Other Information

before inserting material

after inserting material

we can see that the pipeline is divided after inserting material operators

Release Note

N/A

src/sql/code_generator/ob_static_engine_cg.cpp

raywill

请分别按照 review 修改。特别注意：

一些成员，不是那么明显的，添加适当注释
错误码覆盖问题
push_back 等函数调用的错误码返回值检查问题。如果确认不需要检查，那么就应该返回void，或者在调用处这么写： (void) func();

raywill · 2023-07-03T02:15:23Z

src/sql/ob_sql_define.h

@@ -165,7 +165,7 @@ enum ObMatchAgainstMode {

 #define IS_OUTER_OR_CONNECT_BY_JOIN(join_type) (IS_OUTER_JOIN(join_type) || CONNECT_BY_JOIN == join_type)

-#define IS_DUMMY_PHY_OPERATOR(op_type) ((op_type == PHY_MONITORING_DUMP))
+#define IS_DUMMY_PHY_OPERATOR(op_type) ((op_type == PHY_MONITORING_DUMP || op_type == PHY_MATERIAL))


这个地方会不会有点危险？ PHY_MATERIAL 并不总是 DUMMY 算子，有些时候是实际有用的常规算子。

int ObOperator::get_real_child(ObOperator *&child, const int32_t child_idx) 请确认下这里是否安全。

这个地方会不会有点危险？ PHY_MATERIAL 并不总是 DUMMY 算子，有些时候是实际有用的常规算子。

int ObOperator::get_real_child(ObOperator *&child, const int32_t child_idx) 请确认下这里是否安全。

确实是这样，这里我先改回去了

raywill · 2023-07-03T02:17:51Z

src/sql/optimizer/ob_log_plan.cpp

@@ -11096,7 +11095,8 @@ int ObLogPlan::generate_plan()
  } else if (OB_FAIL(plan_traverse_loop(RUNTIME_FILTER,
                                        ALLOC_GI,
                                        PX_PIPE_BLOCKING,
-                                        ALLOC_MONITORING_DUMP,
+                                        OPERATOR_NUMBERING,


做两次 OPERATOR_NUMBERING？为什么？必要性是？原来为什么不需要？

做两次 OPERATOR_NUMBERING？为什么？必要性是？原来为什么不需要？

因为白名单和枚举的插入级别需要根据原始op id判断是否插入算子，所以需要先OPERATOR_NUMBERING分配原始的op id。插入物化算子后，再做一次OPERATOR_NUMBERING生成新的op id。
我后面在代码里加下注释解释这个事情～

raywill · 2023-07-03T02:20:31Z

src/sql/resolver/dml/ob_dml_resolver.cpp

+      } else if (NULL != node->str_value_){
+        ObString alloc_level;
+        alloc_level.assign_ptr(node->str_value_, static_cast<int32_t>(node->str_len_));
+        if (0 == alloc_level.case_compare("all") || 0 == alloc_level.case_compare("ALL")) {


为什么不用 case 不敏感的 compare 函数？

为什么不用 case 不敏感的 compare 函数？

这里是我搞错了，case_compare本身就是大小写不敏感，已经修改～

raywill · 2023-07-03T02:21:43Z

src/sql/resolver/dml/ob_dml_resolver.cpp

      }
    }
+    if (OB_FAIL(alloc_op_hints.push_back(alloc_op_hint))) {


新手写 OB 代码最常犯的一个错误：错误码覆盖。假如上面的分支抛出了一个错误（L13599)，这里 OB_FAIL 宏会覆盖这个错误。

新手写 OB 代码最常犯的一个错误：错误码覆盖。假如上面的分支抛出了一个错误（L13599)，这里 OB_FAIL 宏会覆盖这个错误。

好滴，我把其他地方也检查一遍～

src/sql/resolver/dml/ob_hint.cpp

src/sql/optimizer/ob_logical_operator.cpp

raywill · 2023-07-03T03:36:47Z

src/sql/optimizer/ob_logical_operator.cpp

+    // disable all nodes behind right child
+    OZ(get_child(second_child)->recursively_disable_alloc_op_above(ctx));
+  }
+  if (get_plan()->get_optimizer_context().get_exec_ctx()->get_sql_ctx()->is_remote_sql_


这个和上面是 else if 关系还是，还是分段判断关系？这个 if 里面，有可能覆盖上面的错误码。

错误码覆盖，是非常大的坑。可能会导致一些线上非常难排查的问题。

这个和上面是 else if 关系还是，还是分段判断关系？这个 if 里面，有可能覆盖上面的错误码。

错误码覆盖，是非常大的坑。可能会导致一些线上非常难排查的问题。

是分段关系，我处理一下错误码的问题

raywill · 2023-07-04T05:50:13Z

src/sql/resolver/dml/ob_hint.cpp

+          LOG_WARN("failed to print blocking hint", K(ret));
+        }
+      }
+      if (OB_FAIL(BUF_PRINTF(")"))) {


错误码覆盖

错误码覆盖

done

src/sql/optimizer/ob_logical_operator.cpp

raywill · 2023-07-04T05:55:24Z

src/sql/optimizer/ob_logical_operator.cpp

+  if (expr->get_expr_type() == T_FUN_SYS_ROWNUM) {
+    found = true;
+  } else {
+    for (auto i = 0; !found && i < expr->get_param_count(); i++) {


这里，不同循环次数之间也可能发生错误码覆盖？

raywill · 2023-07-04T05:55:57Z

src/sql/optimizer/ob_logical_operator.cpp

+    LOG_WARN("fail to find rownum expr", K(ret));
+  } else {
+    if (found) {
+      for (int64_t i = 0; i < cur_path.count(); ++i) {


错误码覆盖？

raywill · 2023-07-04T05:59:46Z

src/sql/optimizer/ob_logical_operator.cpp

+              NESTED_LOOP_JOIN == static_cast<ObLogJoin*>(this)->get_join_algo())) {
+      OB_ASSERT(get_num_of_child() > 1 && nullptr != get_child(second_child));
+      // disable all nodes behind right child
+      if (OB_FAIL(get_child(second_child)->recursively_disable_alloc_op_above(ctx))) {


覆盖了 5453 行的错误码。

raywill · 2023-07-04T05:59:47Z

src/sql/optimizer/ob_logical_operator.cpp

@@ -4406,6 +4443,41 @@ void ObLogicalOperator::set_parent(ObLogicalOperator *parent)
  parent_ = parent;
 }

+int ObLogicalOperator::allocate_material_node_above()
+{
+  int ret = OB_SUCCESS;


加注释，说明下列情况是做什么，处理什么场景

CLAassistant · 2024-05-10T08:07:14Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

zhangyuqin1998 seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

zhangyuqin1998 commented Jun 16, 2023

View reviewed changes

src/sql/code_generator/ob_static_engine_cg.cpp Show resolved Hide resolved

raywill reviewed Jul 3, 2023

View reviewed changes

src/sql/code_generator/ob_static_engine_cg.cpp Show resolved Hide resolved

raywill suggested changes Jul 3, 2023

View reviewed changes

zhangyuqin1998 force-pushed the material branch from ccc2839 to c629920 Compare July 4, 2023 03:32

raywill reviewed Jul 4, 2023

View reviewed changes

zhangyuqin1998 force-pushed the material branch 2 times, most recently from 4c960ef to 6fbf625 Compare July 4, 2023 09:07

raywill approved these changes Jul 4, 2023

View reviewed changes

zhangyuqin1998 force-pushed the material branch 4 times, most recently from bf2866d to 76e6612 Compare July 11, 2023 05:46

zhangyuqin1998 added 2 commits July 11, 2023 13:51

add blocking hint to monitoring

1b12c6f

add test case for blocking hint

2f77458

zhangyuqin1998 force-pushed the material branch from 76e6612 to 2f77458 Compare July 11, 2023 05:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allocate Material Above Nodes #1437

Allocate Material Above Nodes #1437

zhangyuqin1998 commented Jun 16, 2023 •

edited

raywill left a comment

raywill Jul 3, 2023

zhangyuqin1998 Jul 4, 2023

raywill Jul 3, 2023

zhangyuqin1998 Jul 3, 2023

raywill Jul 3, 2023

zhangyuqin1998 Jul 3, 2023

raywill Jul 3, 2023

zhangyuqin1998 Jul 3, 2023

raywill Jul 3, 2023

zhangyuqin1998 Jul 3, 2023

raywill Jul 4, 2023

zhangyuqin1998 Jul 4, 2023

raywill Jul 4, 2023

zhangyuqin1998 Jul 4, 2023

raywill Jul 4, 2023

zhangyuqin1998 Jul 4, 2023

raywill Jul 4, 2023

zhangyuqin1998 Jul 4, 2023

raywill Jul 4, 2023

zhangyuqin1998 Jul 4, 2023

CLAassistant commented May 10, 2024

Allocate Material Above Nodes #1437

Are you sure you want to change the base?

Allocate Material Above Nodes #1437

Conversation

zhangyuqin1998 commented Jun 16, 2023 • edited

Task Description

Solution Description

Passed Regressions

Upgrade Compatibility

Other Information

Release Note

raywill left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CLAassistant commented May 10, 2024

zhangyuqin1998 commented Jun 16, 2023 •

edited