Support for Snowflake Connector's query pushdown

[Ioankall]

Snowflake Connector supports Query Pushdown - a mechanism that turn normal spark transformations into a snowflake query plan. It's a very useful feature as you save all the networking cost of pulling data from huge tables to where your Spark Application is running.

I tried to run the Profiler on a Spark DataFrame that's been loaded from a Snowflake table with the Pushdown mechanism enabled. I got the following error:

23/07/14 11:59:50 INFO SnowflakeTelemetry$: Pushdown fails because of operation: stateful_approx_count_distinct @ AggregationStatement
message: pushdown failed for aggregate expression
isKnown: false

Is anyone in the community familiar with this mechanism/error?

[jhchee]

@Ioankall This sounds like a problem on Snowflake driver rather than deequ/Spark.