Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ARCTIC-910]Ignore committing error to HMS when committing Hive format table transaction #2482

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

hameizi
Copy link
Contributor

@hameizi hameizi commented Dec 28, 2023

…saction

Why are the changes needed?

Close #910 .

Brief change log

  • Ignore committing error to HMS when committing Hive format table transaction

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before making a pull request

Documentation

  • Does this pull request introduce a new feature? (no)
  • If yes, how is the feature documented? (not documented)

@github-actions github-actions bot added the module:mixed-hive Hive moduel for Mixed Format label Dec 28, 2023
Copy link

codecov bot commented Dec 28, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (5eb153f) 32.64% compared to head (92c2f16) 32.59%.

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #2482      +/-   ##
============================================
- Coverage     32.64%   32.59%   -0.05%     
+ Complexity     4473     4469       -4     
============================================
  Files           598      598              
  Lines         50276    50280       +4     
  Branches       6676     6676              
============================================
- Hits          16411    16389      -22     
- Misses        32550    32587      +37     
+ Partials       1315     1304      -11     
Flag Coverage Δ
core 30.75% <100.00%> (+<0.01%) ⬆️
trino 50.38% <ø> (-0.58%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@wangtaohz
Copy link
Contributor

Can you share the exception stack for the scenario addressed by this PR? I'm not sure when this issue occurs. @hameizi

@wangtaohz
Copy link
Contributor

I also noticed that it seems like the try-catch block was forgotten to be added here com.netease.arctic.hive.op.ReplaceHivePartitions#commit

  @Override
  public void commit() {
    if (!addFiles.isEmpty()) {
      List<DataFile> dataFiles =
          HiveCommitUtil.commitConsistentWriteFiles(this.addFiles, table.io(), table.spec());
      this.addFiles.clear();
      this.addFiles.addAll(dataFiles);
      this.addFiles.forEach(delegate::addFile);

      commitTimestamp = (int) (System.currentTimeMillis() / 1000);
      if (table.spec().isUnpartitioned()) {
        generateUnpartitionTableLocation();
      } else {
        applyHivePartitions();
      }

      delegate.commit();
      commitPartitionProperties();
      if (!insideTransaction) {
        transaction.commitTransaction();
      }

      // here the try-catch is missing
      if (table.spec().isUnpartitioned()) {
        commitUnPartitionedTable();
      } else {
        commitPartitionedTable();
      }
    }
  }

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
module:mixed-hive Hive moduel for Mixed Format
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug]: When optimize fails to commit and then deletes a file, the optimize task will fail when retrying
2 participants