New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[KYUUBI #6034] Kyuubi Server HA&ZK get server from serverHosts support more strategy #6213
base: master
Are you sure you want to change the base?
[KYUUBI #6034] Kyuubi Server HA&ZK get server from serverHosts support more strategy #6213
Conversation
LGTM in general, can we add some tests? |
i'm not sure did this improvement could execute unit test? But i will try |
function level unit test is OK |
...i-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/strategy/zk/PollingChooseStrategy.java
Outdated
Show resolved
Hide resolved
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #6213 +/- ##
============================================
- Coverage 61.10% 58.43% -2.68%
- Complexity 23 24 +1
============================================
Files 622 655 +33
Lines 37036 39673 +2637
Branches 5024 5456 +432
============================================
+ Hits 22631 23182 +551
- Misses 11967 14003 +2036
- Partials 2438 2488 +50 ☔ View full report in Codecov by Sentry. |
🔍 Description
Issue References 🔗
This pull request fixes #6034
Describe Your Solution 🔧
Currently, use beeline to connect kyuubiServer with HA mode, the strategy only support random, this will lead to a high load on the machine. So i make this pr to support choose strategy.
[description]
First, we need know, beeline connect kyuubiServer dependency on kyuubi-hive-jdbc, it is isolated from the kyuubi cluster, so the code only support random choose serverHost from zk node /${namespace}. Because kyuubi-hive-jdbc is a stateless module, only run once, cannot store var about get serverHost from zk node.
[Solution]
This pr, we could implement a interface named ChooseServerStrategy to choose serverHost. I implement two strategy
Types of changes 🔖
Test Plan 🧪
Test the Strategy in my test Cluster
Behavior Without This Pull Request ⚰️
Behavior With This Pull Request 🎉
[Use Case]
bin/beeline -u 'jdbc:hive2://xxx:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi;zooKeeperStrategy=poll?spark.yarn.queue=root.kylin;spark.app.name=testspark;spark.shuffle.useOldFetchProtocol=true' -n mfw_hadoop --verbose=true --showNestedErrs=true
bin/beeline -u 'jdbc:hive2://xxx:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi;zooKeeperStrategy=random?spark.yarn.queue=root.kylin;spark.app.name=testspark;spark.shuffle.useOldFetchProtocol=true' -n mfw_hadoop --verbose=true --showNestedErrs=true
orbin/beeline -u 'jdbc:hive2://xxx:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi?spark.yarn.queue=root.kylin;spark.app.name=testspark;spark.shuffle.useOldFetchProtocol=true' -n mfw_hadoop --verbose=true --showNestedErrs=true
bin/beeline -u 'jdbc:hive2://xxx:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi;zooKeeperStrategy=xxx.xxx.xxx.XxxChooseServerStrategy?spark.yarn.queue=root.kylin;spark.app.name=testspark;spark.shuffle.useOldFetchProtocol=true' -n mfw_hadoop --verbose=true --showNestedErrs=true
[Result: The Cluster have two Server (221,233)]
1.1. zkNode: counterValue
1.2. result:
random:
YourStrategy(the test case only get the first serverHost):
Related Unit Tests
There is no Unit Tests.
Checklist 📝
Be nice. Be informative.