[Bug] [mysql to hive 2.3.5] Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(Lorg/apache/hadoop/hive/conf/HiveConf;)V

[zhaoruosong]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

I updated version 2.3.5 of seatunnel. In this version of sink-hive doc, the description supports version 3.1.3 of hive. But after the experiment according to the description, the error was reported all the time.
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(Lorg/apache/hadoop/hive/conf/HiveConf;)V at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.<init>(HiveMetaStoreProxy.java:84) at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getInstance(HiveMetaStoreProxy.java:113) at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:73) at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.prepare(HiveSink.java:122) at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSink(JobConfigParser.java:165) at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSinks(JobConfigParser.java:135) at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:517) at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:200) at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getLogicalDag(ClientJobExecutionEnvironment.java:88) at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.execute(ClientJobExecutionEnvironment.java:156) at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:149) at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)

SeaTunnel Version

2.3.5

SeaTunnel Config

env {
  execution.parallelism = 1
  job.mode = "BATCH"
}
source{
    Jdbc {
        url = "jdbc:mysql://192.168.**.***:3306/test"
        driver = "com.mysql.cj.jdbc.Driver"
        connection_check_timeout_sec = 10000
        user = "****"
        password = "****"
        query = "select id,name from test_seatunnel"
    }
}
sink {
  Hive {
      table_name = "test.test_seatunnel"
      metastore_uri = "thrift://name192.168.***.**:9083"
  }

}

Running Command

seatunnel.sh --config /data/apache-seatunnel-2.3.5/job/test_seatunnel -m local

Error Exception

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(Lorg/apache/hadoop/hive/conf/HiveConf;)V
        at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.<init>(HiveMetaStoreProxy.java:84)
        at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getInstance(HiveMetaStoreProxy.java:113)
        at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:73)
        at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.prepare(HiveSink.java:122)
        at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSink(JobConfigParser.java:165)
        at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSinks(JobConfigParser.java:135)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:517)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:200)
        at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getLogicalDag(ClientJobExecutionEnvironment.java:88)
        at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.execute(ClientJobExecutionEnvironment.java:156)
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:149)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)

Zeta or Flink or Spark Version

zeta

Java or Scala Version

java1.8

Screenshots

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[liunaijie]

You need put seatunnel-hadoop3-3.1.4-uber.jar and hive-exec-3.1.3.jar and libfb303-0.9.3.jar in $SEATUNNEL_HOME/lib/ dir. Please check those jar are exist.

[NoPr]

Modify the pom file for connection live. Modify the hive. exec. version version to 3.1.3, recompile and replace the jar package under connections

[zhaoruosong]

You need put seatunnel-hadoop3-3.1.4-uber.jar and hive-exec-3.1.3.jar and libfb303-0.9.3.jar in $SEATUNNEL_HOME/lib/ dir. Please check those jar are exist.

There are all these.

[zhaoruosong]

Modify the pom file for connection live. Modify the hive. exec. version version to 3.1.3, recompile and replace the jar package under connections修改pom文件以进行实时连接。改造母舰。执行版本版本到3.1.3，重新编译并替换连接下的jar包

I'm sorry. I don't quite understand what you mean. Do you want to change the jar package version of hive.exec to 3.1.3? My hive.exec in lib is 3.1.3 now.

[NoPr]

修改 pom 文件以进行实时连接。修改蜂巢。执行。 version版本到3.1.3，重新编译并替换connections下的jar包修改pom文件以进行实时连接。改造母舰。执行版本版本到3.1.3，重新编译并替换连接下的jar包

对不起。我不太明白你的意思。是否要将hive.exec的jar包版本更改为3.1.3？我的 lib 中的 hive.exec 现在是 3.1.3。

修改源码connection-hive

[zhaoruosong]

修改源码connection-hive

@NoPr Thank you. After modifying the source code and recompiling, the problem is solved.