Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[HUDI-7146] Implement secondary index write path #11146

Merged
merged 13 commits into from
May 30, 2024
Merged

[HUDI-7146] Implement secondary index write path #11146

merged 13 commits into from
May 30, 2024

Conversation

codope
Copy link
Member

@codope codope commented May 3, 2024 •
edited

Change Logs

Main changes in this PR are for secondary index write path:

  • New index type added in MetadataPartitionType
  • Initialization of the new index in HoodieBackedTableMetadataWriter
  • Util methods to support index creation and update in HoodieTableMetadataUtil
  • Changes to HoodieBackedTableMetadataWriter to handle update and deletes for secondary index.
  • New APIs in HoodieTableMetadata and their implementation in BaseTableMetadata and HoodieBackedTableMetadata to load secondary index.
  • Changes in HoodieMergedLogRecordScanner to merge secondary index payloads.

Another PR will follow this PR, wherein we will have the secondary index reader-side integration and more e2e tests.

Impact

Support secondary index creation and update.

Risk level (write none, low medium or high below)

medium

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".

  • The config description must be updated if new configs are added or the default value of the configs are changed
  • Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
    ticket number here and follow the instruction to make
    changes to the website.

Contributor's checklist

  • Read through contributor's guide
  • Change Logs and Impact were stated clearly
  • Adequate tests were added if applicable
  • CI passed

@github-actions github-actions bot added the size:XL PR with lines of changes > 1000 label May 3, 2024
@codope codope force-pushed the impl-sec-index-update-read branch 3 times, most recently from 9ac7e79 to 7f99a00 Compare May 6, 2024 15:08
@codope codope mentioned this pull request May 6, 2024
Closed
4 tasks
@codope codope changed the title [HUDI-7146][WIP] Implement secondary index update and delete [HUDI-7146] Implement secondary index write path May 6, 2024
@codope codope force-pushed the impl-sec-index-update-read branch from 7f99a00 to 1468a9a Compare May 6, 2024 15:48
@codope codope mentioned this pull request May 6, 2024
Open
4 tasks
@codope codope force-pushed the impl-sec-index-update-read branch from 1468a9a to f232b46 Compare May 7, 2024 05:14
@codope codope force-pushed the impl-sec-index-update-read branch from f232b46 to 8a6a98e Compare May 16, 2024 03:22
@codope codope mentioned this pull request May 17, 2024
Merged
4 tasks
@codope codope force-pushed the impl-sec-index-update-read branch from 8a6a98e to 420b192 Compare May 20, 2024 04:42
@github-actions github-actions bot added size:L PR with lines of changes in (300, 1000] and removed size:XL PR with lines of changes > 1000 labels May 20, 2024
@codope codope force-pushed the impl-sec-index-update-read branch from 9c3c82d to 3bfd7de Compare May 21, 2024 13:33
danny0405
danny0405 reviewed May 22, 2024
View reviewed changes
Copy link
Contributor

@danny0405 danny0405 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution, I have left some comments.

...di-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java Outdated Show resolved Hide resolved
...di-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java Outdated Show resolved Hide resolved
...di-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java Show resolved Hide resolved
...di-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java Outdated Show resolved Hide resolved
...di-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java Outdated Show resolved Hide resolved
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java Outdated Show resolved Hide resolved
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java Outdated Show resolved Hide resolved
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java Outdated Show resolved Hide resolved
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java Outdated Show resolved Hide resolved
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadata.java Outdated Show resolved Hide resolved
@codope codope force-pushed the impl-sec-index-update-read branch from 3bfd7de to 74d8ad0 Compare May 22, 2024 12:46
@github-actions github-actions bot added size:XL PR with lines of changes > 1000 and removed size:L PR with lines of changes in (300, 1000] labels May 22, 2024
@codope codope force-pushed the impl-sec-index-update-read branch 2 times, most recently from 0e0ec70 to 98d96be Compare May 27, 2024 16:16
danny0405
danny0405 reviewed May 28, 2024
View reviewed changes
Copy link
Contributor

@danny0405 danny0405 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the updates, I have left some comments.

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java Outdated Show resolved Hide resolved
...di-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java Outdated Show resolved Hide resolved
...park-client/src/main/java/org/apache/hudi/metadata/SparkHoodieBackedTableMetadataWriter.java Outdated Show resolved Hide resolved
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieIndexDefinition.java Show resolved Hide resolved
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieIndexesMetadata.java Outdated Show resolved Hide resolved
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java Outdated Show resolved Hide resolved
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java Outdated Show resolved Hide resolved
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java Outdated Show resolved Hide resolved
...di-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java Show resolved Hide resolved
...ark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/FunctionalIndexSupport.scala Outdated Show resolved Hide resolved
@codope codope force-pushed the impl-sec-index-update-read branch 4 times, most recently from 470bc5f to 3d9b55d Compare May 29, 2024 13:16
@codope codope force-pushed the impl-sec-index-update-read branch from 3d9b55d to e8a4507 Compare May 30, 2024 01:33
danny0405
danny0405 approved these changes May 30, 2024
View reviewed changes
Copy link
Contributor

@danny0405 danny0405 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, let's make all the CI pass before landing.

@hudi-bot
Copy link

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

@codope codope merged commit cd62c31 into apache:master May 30, 2024
46 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@danny0405 danny0405 danny0405 approved these changes

Assignees
No one assigned
Labels
index release-1.0.0-beta2 size:XL PR with lines of changes > 1000
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@codope @hudi-bot @danny0405