You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
According to [parisni in [HUDI-6150] Support bucketing for each hive client (https://github.com//pull/8657)
"So I assume hudi way of doing (which is not compliant with both hive and spark) cannot be used to improve query engines queries such join and filter. Then this leads all of below are wrong:
I think the support for 2 hudi tables' Spark sort-merge-join with bucket optimization is an important feature.
Currently if we join 2 hudi tables, the bucket index's bucket information is not usable by spark, so shuffle is always needs. As explained in 8657 - hashing- file naming- file numbering- file sorting are different.
Unfortunately, according to https://issues.apache.org/jira/browse/SPARK-19256, spark bucket is not compatible with hive bucket yet. So if we have to choose one between spark and hive, I think spark might be of higher priority.
This is really useful feature to have.
We want to use Hudi at work, but unfortunately we have couple of bucketed/sorted tables, and this is definitely a stopper for us to migrate to Hudi.
According to [parisni in [HUDI-6150] Support bucketing for each hive client (https://github.com//pull/8657)
"So I assume hudi way of doing (which is not compliant with both hive and spark) cannot be used to improve query engines queries such join and filter. Then this leads all of below are wrong:
the current config https://hudi.apache.org/docs/configurations/#hoodiedatasourcehive_syncbucket_sync
this current PR
the rfc statement about support of hive bucketing https://cwiki.apache.org/confluence/display/HUDI/RFC+-+29%3A+Hash+Index"
Do you have any update on this?
The text was updated successfully, but these errors were encountered: