DAOS-623 rebuild: uniform identifier in logs part 1 #14383

kccain · 2024-05-15T23:14:03Z

To the extent possible in the rebuild code execution flow, when rebuild emits log messages, include a uniform rebuild operation identifier in those messages. This covers activities across all pool storage engines (including the pool service leader), system and per-target threads/xstreams, and dynamically spawned user-level threads.

The motivation is to enable some amount of automated searching through logfiles for all (or specific) rebuilds that occurred during execution, and speed up DAOS engineer analysis/interpretation of the logs.

The baseline format (defined in the DF_RB macro) is:
"rb=" DF_UUID "/%u/%u/%s"
and corresponds to:
<pool_uuid>/<rebuild_ver>/<rebuild_gen>/

A verbose format (defined in the DF_RBF macro) adds the following (for <leader_rank>/)
" ld=%u/" DF_U64

Various DP_RB_* and DP_RBF_* macros are defined to specify the arguments to go with the DF_RB and DF_RBF formats, given some common rebuild implementation structures such as:
struct rebuild_global_pool_tracker
struct rebuild_tgt_pool_tracker
struct rebuild_scan_in (REBUILD_OBJECTS_SCAN RPC input)
struct migrate_query_arg

This initial patch covers the pool service leader execution in functions (and those that they invoke) such as:
rebuild_ults()
rebuild_task_ult()
rebuild_leader_start()
rebuild_leader_status_check()
rebuild_leader_status_notify().

And this patch covers "scan side" execution in all pool storage engines (including the leader), in functions such as:
rebuild_tgt_scan_handler()
rebuild_tgt_status_check_ult()
ds_migrate_query_status()
migrate_check_one()
dss_rebuild_check_one()
rebuild_scan_leader()
rebuild_scanner()
rebuild_objects_send_ult()
rebuild_scan_done()

Features: rebuild
Allow-unstable-test: true
Skip-list: test_ec_degrade:DAOS-15843

Before requesting gatekeeper:

Two review approvals and any prior change requests have been resolved.
Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
Commit messages follows the guidelines outlined here.
Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

github-actions · 2024-05-15T23:14:24Z

Ticket title is 'Generic ticket for minor code cleanup and improvement'
Status is 'Resolved'
Labels: 'request_for_2.6,request_for_2.6.1'
https://daosio.atlassian.net/browse/DAOS-623

kccain · 2024-05-15T23:32:54Z

@liuxuezhao @gnailzenh here is a draft patch for what I discussed today. Can you take a look and provide some early comments on the approach?

To the extent possible in the rebuild code execution flow, when rebuild emits log messages, include a uniform rebuild operation identifier in those messages. This covers activities across all pool storage engines (including the pool service leader), system and per-target threads/xstreams, and dynamically spawned user-level threads. The motivation is to enable some amount of automated searching through logfiles for all (or specific) rebuilds that occurred during execution, and speed up DAOS engineer analysis/interpretation of the logs. The baseline format (defined in the DF_RB macro) is: "rb=" DF_UUID "/%u/" DF_U64 "/%u/%u/%s" and corresponds to: <pool_uuid>/<rebuild_ver>/<ps_term>/<rebuild_gen>/<ps_rank>/<opc_str> A verbose format (defined in the DF_RBF macro) adds the following (for <engine_rank>:<tgt_idx>) r:t=%u:%d" Various DP_RB_* and DP_RBF_* macros are defined to specify the arguments to go with the DF_RB and DF_RBF formats, given some common rebuild implementation structures such as: struct rebuild_global_pool_tracker struct rebuild_tgt_pool_tracker struct rebuild_scan_in (REBUILD_OBJECTS_SCAN RPC input) struct migrate_query_arg This initial patch covers the pool service leader execution in functions (and those that they invoke) such as: rebuild_ults() rebuild_task_ult() rebuild_leader_start() rebuild_leader_status_check() rebuild_leader_status_notify(). And this patch covers "scan side" execution in all pool storage engines (including the leader), in functions such as: rebuild_tgt_scan_handler() rebuild_tgt_status_check_ult() ds_migrate_query_status() migrate_check_one() dss_rebuild_check_one() rebuild_scan_leader() rebuild_scanner() rebuild_objects_send_ult() rebuild_scan_done() Features: rebuild Allow-unstable-test: true Skip-list: test_ec_degrade:DAOS-15843 Signed-off-by: Kenneth Cain <kenneth.c.cain@intel.com>

src/include/daos_srv/daos_engine.h

src/include/daos_srv/rebuild.h

src/include/daos_srv/daos_engine.h

Features: rebuild Allow-unstable-test: true Skip-list: test_ec_degrade:DAOS-15843 Signed-off-by: Kenneth Cain <kenneth.c.cain@intel.com>

kccain

addressed feedback in the latest push

src/include/daos_srv/daos_engine.h

src/include/daos_srv/rebuild.h

src/include/daos_srv/daos_engine.h

src/include/daos_srv/rebuild.h

Features: rebuild Allow-unstable-test: true Skip-list: test_ec_degrade:DAOS-15843 Signed-off-by: Kenneth Cain <kenneth.c.cain@intel.com>

daosbuild1 · 2024-05-29T04:26:50Z

Test stage Functional Hardware Large completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14383/10/execution/node/651/log

daosbuild1 · 2024-05-29T05:24:39Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14383/10/execution/node/667/log

daosbuild1 · 2024-05-29T05:32:03Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14383/10/execution/node/505/log

kccain · 2024-05-29T22:29:48Z

all per-PR tests passed. Some full_regression (weekly regresison tests) failed and are known issues (listed at the end)

But this one from Functional HW Large testing that includes an engine assertion seems new.] I don't see a direct link between the changes in this PR and the aggregation engine assertion and stacktrace. @liuxuezhao do you think this patch could have caused it? Or possibly it is a rare issue not yet seen in the testing?

erasurecode/offline_rebuild_aggregation.py failure

ERROR: daos_engine:0 05/29-00:48:44.26 wolf-113 DAOS[185489/8/640] vos EMRG src/vos/vos_aggregate.c:640 recx2ext() Assertion 'recx->rx_nr > 0' failed

For now I've prepared a test-only PR 14474 just the same master commit base as this patch, to see if it reproduces independently.

Functional HW Large testing with known existing failures:

ior/hard_rebuild.py failure is an instance of DAOS-15863

Functional HW Large MD on SSD testing with known existing failures:

erasurecode/online_rebuild.py cases 2 and 4 failures are also an instance of DAOS-15863

Functional HW Medium MD on SSD testing with known existing failures:

deployment/server_rank_failure.py cases 1 and 2 failures are instances of DAOS-15809

And change one debug log to error in rebuild_objects_send_ult() Allow-unstable-test: true Signed-off-by: Kenneth Cain <kenneth.c.cain@intel.com>

liuxuezhao

LGTM, thanks @kccain

kccain · 2024-05-31T13:44:12Z

CI testing is blocked at the moment by https://daosio.atlassian.net/browse/SRE-2233
This PR has passed Features: rebuild testing in build 10 before, so I will move it from draft -> review-able status now and we can get a second reviewer in parallel to CI testing, once that's ready to resume.

src/tests/ftest/erasurecode/offline_rebuild_single.yaml

daosbuild1 · 2024-06-03T01:19:00Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14383/17/execution/node/1386/log

daosbuild1 · 2024-06-03T03:47:58Z

Test stage Functional Hardware Medium UCX Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14383/17/execution/node/1524/log

kccain · 2024-06-03T10:13:22Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14383/17/execution/node/1386/log

The single test failure seen in this testing (scrubber/target_auto_eviction.py) is an instance of existing known issue https://daosio.atlassian.net/browse/DAOS-14585

kccain · 2024-06-03T10:15:40Z

Test stage Functional Hardware Medium UCX Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14383/17/execution/node/1524/log

The single test failure seen in this testing (daos_test/suite.py:DaosCoreTest.test_daos_single_rdg_tx) is an instance of existing known issue https://daosio.atlassian.net/browse/DAOS-14982

kccain · 2024-06-03T10:17:23Z

Features testing done/successful in build 10.
PR testing done/successful in build 17

Patch requires one more peer review

kccain · 2024-06-04T16:34:30Z

While waiting for reviews, I am continuing to investigate if the weekly regression test failure (recx2ext() engine assertion) seen in build 10 is always associated with this patch, or if it can be seen in master. That work is being documented in https://daosio.atlassian.net/issues/DAOS-15941

For now I have only seen it with this patch.

kccain force-pushed the kccain/rebuild_uniform_logging branch from 7424223 to cb61d6d Compare May 15, 2024 23:27

daos-stack deleted a comment from daosbuild1 May 15, 2024

kccain requested a review from liuxuezhao May 15, 2024 23:33

This comment was marked as outdated.

Sign in to view

kccain force-pushed the kccain/rebuild_uniform_logging branch 2 times, most recently from d6b5852 to 64a4330 Compare May 16, 2024 14:56

kccain force-pushed the kccain/rebuild_uniform_logging branch from 64a4330 to 41005ed Compare May 17, 2024 13:24

liuxuezhao reviewed May 20, 2024

View reviewed changes

kccain added 2 commits May 21, 2024 10:05

Merge branch 'master' into kccain/rebuild_uniform_logging

867a0fa

Address feedback from Xuezhao

b16b7f9

Features: rebuild Allow-unstable-test: true Skip-list: test_ec_degrade:DAOS-15843 Signed-off-by: Kenneth Cain <kenneth.c.cain@intel.com>

kccain commented May 21, 2024

View reviewed changes

This comment was marked as outdated.

Sign in to view

Fix clang-format issues

449a2b6

Features: rebuild Allow-unstable-test: true Skip-list: test_ec_degrade:DAOS-15843 Signed-off-by: Kenneth Cain <kenneth.c.cain@intel.com>

kccain added 2 commits May 31, 2024 00:00

Merge branch 'master' into kccain/rebuild_uniform_logging

d1bdf6f

Restore logging info in rebuild_task_complete_schedule()

ede6688

And change one debug log to error in rebuild_objects_send_ult() Allow-unstable-test: true Signed-off-by: Kenneth Cain <kenneth.c.cain@intel.com>

This comment was marked as outdated.

Sign in to view

liuxuezhao approved these changes May 31, 2024

View reviewed changes

kccain marked this pull request as ready for review May 31, 2024 13:44

kccain requested review from a team as code owners May 31, 2024 13:44

daltonbohning reviewed May 31, 2024

View reviewed changes

src/tests/ftest/erasurecode/offline_rebuild_single.yaml Show resolved Hide resolved

kccain requested a review from wangshilong June 3, 2024 01:19

DAOS-623 rebuild: uniform identifier in logs part 1 #14383

Are you sure you want to change the base?

DAOS-623 rebuild: uniform identifier in logs part 1 #14383

Conversation

kccain commented May 15, 2024 • edited

Before requesting gatekeeper:

Gatekeeper:

github-actions bot commented May 15, 2024 • edited

kccain commented May 15, 2024

This comment was marked as outdated.

kccain left a comment

Choose a reason for hiding this comment

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

daosbuild1 commented May 29, 2024

daosbuild1 commented May 29, 2024

daosbuild1 commented May 29, 2024

kccain commented May 29, 2024 • edited

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

liuxuezhao left a comment

Choose a reason for hiding this comment

kccain commented May 31, 2024

daosbuild1 commented Jun 3, 2024

daosbuild1 commented Jun 3, 2024

kccain commented Jun 3, 2024 • edited

kccain commented Jun 3, 2024

kccain commented Jun 3, 2024

kccain commented Jun 4, 2024

kccain commented May 15, 2024 •

edited

github-actions bot commented May 15, 2024 •

edited

kccain commented May 29, 2024 •

edited

kccain commented Jun 3, 2024 •

edited