You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Fetching more than 100 rows seem to be rather slow compared to RJDBC (with the same jar files) or exporting data to CSV via impala-shell, so I suspect there might be some performance issues when converting the results returned by the JDBC driver to R. See some related benchmarks at http://datascience.la/r-and-impala-its-better-to-kiss-than-using-java
The text was updated successfully, but these errors were encountered:
Thanks for letting me know. Yes, I agree with you. I think a huge amount of time is spent converting the JDBC result set to R data types. I assume giving an option in the function to return the JDBC result as such and letting the user figure out how to use it will give similar data fetch times as RJDBC. In our use case we didn't really need to transfer huge data sets back to R which was probably why we overlooked this. I am a bit tied up right now. I will look into it and get back to you late next week.
Brilliant blog post by the way :)
Fetching more than 100 rows seem to be rather slow compared to
RJDBC
(with the samejar
files) or exporting data to CSV viaimpala-shell
, so I suspect there might be some performance issues when converting the results returned by the JDBC driver to R. See some related benchmarks at http://datascience.la/r-and-impala-its-better-to-kiss-than-using-javaThe text was updated successfully, but these errors were encountered: