HiveUtils.fetchAll fetches one block too many

[fjakobs]

I noticed that HiveUtils.fetchAll calls driver.fetchResults twice for small operations that fit in a single response. Apparently checkIfOperationHasMoreRows returns true if the response contains any rows regardless of the hasMoreRows value provided by the server.

This might not be a big issue for fetching large datasets but for small operations on high latency connections this is pretty impactful.

[susodapop]

Thanks for calling attention to this.

Since this project started as a fork of Hive-driver, it integrates the official Apache Hive thrift definition. Databricks maintains backwards compatibility with this definition. But unfortunately the Hive thrift definition has a quirk where it always sends hasMoreRows==false regardless whether more rows are available at the server. Thus we fetch twice to guarantee nothing was left behind on the server.

In the coming quarter we'll transition this connector to use our own thrift definition which has been revised to include sensible hasMoreRows behaviour.

cc: @moderakh

[fjakobs]

Nice!