Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TASK][EASY] Kyuubi Server HA&ZK get server from serverHosts support more strategy #6034

Open
3 of 4 tasks
davidyuan1223 opened this issue Jan 31, 2024 · 7 comments · May be fixed by #6213
Open
3 of 4 tasks

[TASK][EASY] Kyuubi Server HA&ZK get server from serverHosts support more strategy #6034

davidyuan1223 opened this issue Jan 31, 2024 · 7 comments · May be fixed by #6213

Comments

@davidyuan1223
Copy link
Contributor

Code of Conduct

Search before asking

  • I have searched in the issues and found no similar issues.

What would you like to be improved?

The current Kyuubi HA mode, which retrieves servers from ZooKeeper, only supports the random strategy. This may lead to an overload on certain nodes. Therefore, in order to address the overload issue, it is necessary to support more strategies.

How should we improve?

Update Kyuubi Hive JDBC to support ZooKeeperClientHelper to support more strategies, currently, there are two strategy:

  1. Random
  2. Polling

Are you willing to submit PR?

  • Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.
  • No. I cannot submit a PR at this time.
@davidyuan1223 davidyuan1223 changed the title [Improvement] Kyuubi Server HA&ZK get server from serverHost support more strategy [Improvement] Kyuubi Server HA&ZK get server from serverHosts support more strategy Jan 31, 2024
@davidyuan1223
Copy link
Contributor Author

@wForget @pan3793

@pan3793
Copy link
Member

pan3793 commented Jan 31, 2024

SGTM, and it's better to extract an Interface to allow user to implement their custom strategy

@davidyuan1223
Copy link
Contributor Author

SGTM, and it's better to extract an Interface to allow user to implement their custom strategy

hello, want ask a question, kyuubi-hive-jdbc is only a driver engine, which cannot read configuration from the kyuubiConf, so if we set a strategy configEntry in HA, the kyuubi-hive-jdbc also cannot read the config, the only way i think is to add in connection params, but if add in connection params, we cannot custom our strategy, what do you think?

@sunnyzhuzhu
Copy link

@davidyuan1223 hello, May I ask what strategy will you implement?

@davidyuan1223
Copy link
Contributor Author

@davidyuan1223 hello, May I ask what strategy will you implement?

sorry, forget response, curruntly, i implemented poll and random, because hive-jdbc module is a single module, we can not use kyuubi-ha module, so, if we want implemented more strategies, we only can add strategy in connection params, like '&zkStartegy=poll/random', if you have more useful starategy, you can give me some advice

@davidyuan1223
Copy link
Contributor Author

@davidyuan1223 hello, May I ask what strategy will you implement?

hello, this is the demo command
bin/beeline -u 'jdbc:hive2://xxx:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi;zooKeeperStrategy=poll?spark.app.name=testspark;spark.shuffle.useOldFetchProtocol=true' -n hadoop --verbose=true --showNestedErrs=true
currently it can use poll strategy to choose the right server, but there are some bugs, so i'm not commit a pr.

@davidyuan1223
Copy link
Contributor Author

@davidyuan1223 hello, May I ask what strategy will you implement?

hello, this is the demo command bin/beeline -u 'jdbc:hive2://xxx:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi;zooKeeperStrategy=poll?spark.app.name=testspark;spark.shuffle.useOldFetchProtocol=true' -n hadoop --verbose=true --showNestedErrs=true currently it can use poll strategy to choose the right server, but there are some bugs, so i'm not commit a pr.

i plan the user could implemented a interface named org.apache.kyuubi.jdbc.hive.strategy.ChooseServerStrategy, then use zooKeeperStrategy=xxx.xxx.xxx, so user can use themselves implement plan, of course, if you have more effective plan, you can offered me, and i will try to implement them

@cxzl25 cxzl25 changed the title [Improvement] Kyuubi Server HA&ZK get server from serverHosts support more strategy [TASK][EASY][Improvement] Kyuubi Server HA&ZK get server from serverHosts support more strategy Apr 15, 2024
@cxzl25 cxzl25 changed the title [TASK][EASY][Improvement] Kyuubi Server HA&ZK get server from serverHosts support more strategy [TASK][EASY] Kyuubi Server HA&ZK get server from serverHosts support more strategy Apr 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants