Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OH client side spec to accept replication config #43

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

rohitkum2506
Copy link
Collaborator

@rohitkum2506 rohitkum2506 commented Mar 6, 2024

Summary

Issue]
OH Client changes to accept Replication config as policy.
A table can be Altered to add replication config as:

spark.sql("ALTER TABLE SET POLICY (REPLICATION=({cluster:'', schedule:''},...))")

e.g:
spark.sql("ALTER TABLE db.testTable SET POLICY (REPLICATION=({cluster:'holdem', schedule:'0 0 0 9 0 ?'}))")
spark.sql("ALTER TABLE db.testTable SET POLICY (REPLICATION=({cluster:'holdem', schedule:'0 0 0 9 0 ?'}, {cluster:'war', schedule:'0 0 0 10 0 ?'} ))")

Note:
Each policy spec under {} defines one cluster config for replication and multiple such configs can be defined.
The schedule is free text string representing replication schedule which will be validated at service level.

Next steps:

Table service changes to recognize replication policy and store in table properties
Pipeline(Job scheduler) to setup replica table and trigger replication job

Changes

  • Client-facing API Changes
  • Internal API Changes
  • Bug Fixes
  • New Features
  • Performance Improvements
  • Code Style
  • Refactoring
  • Documentation
  • Tests

For all the boxes checked, please include additional details of the changes made in this pull request.

Testing

  • Manually Tested on local docker setup. Please include commands ran, and their output.
  • Added new tests for the changes made.
  • Updated existing tests to reflect the changes made.
  • No tests added or updated. Please explain why. If unsure, please feel free to ask for help.
  • Some other form of testing like staging or soak time in production. Please explain.

Steps:

  1. Unit tests
  2. Deployed on local docker and ran sql commands to validate alter table statements
  3. Verified that the updates policy from table OPS contains JSON for replication config.

Screenshot 2024-03-04 at 2 16 27 PM
Screenshot 2024-03-04 at 5 19 23 PM

For all the boxes checked, include a detailed description of the testing done for the changes made in this pull request.

Additional Information

  • Breaking Changes
  • Deprecations
  • Large PR broken into smaller PRs, and PR plan linked in the description.

For all the boxes checked, include additional details of the changes made in this pull request.

@rohitkum2506 rohitkum2506 marked this pull request as ready for review March 6, 2024 19:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant