Recommended way to enable/disable Gluten

[acvictor]

What is the recommended way to enable/disable Gluten in a Spark session if we want the default as disabled and explicit config setting to enable?

Is it alright to set spark.plugins=io.glutenproject.GlutenPlugin, spark.gluten.enabled=false as part of Spark default confs and then use dynamically use spark.gluten.enabled=true in a session if we want native execution?

[acvictor]

cc @zhli1142015 @PHILO-HE @rui-mo

[zhli1142015]

cc @zhouyuan and @FelixYBW also.

[PHILO-HE]

@acvictor, I think so. We have to always configure spark.plugins=org.apache.gluten.GlutenPlugin to let initialization be done at driver/executor start time. And user can set spark.gluten.enabled to dynamically enable/disable Gluten for a session.

BTW, you can set a thread local property if you would like to control using Gluten for a single thread.
https://github.com/apache/incubator-gluten/blob/main/docs/Configuration.md?plain=1#L106