[TASK][MEDIUM] Resource isolation in Spark Scala mode

[pan3793]

What's the level of this task?

MEDIUM

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before creating

• I have searched in the task list and found no similar tasks.

Mentor

• I have sufficient expertise on this task, and I volunteer to be a mentor of this task to guide contributors through the task.

Skill requirements

• Knowledge of Spark/Scala REPL
• Knowledge of Classloader, JVM serialization

Background and Goals

The current Kyuubi Scala mode reuses the code of Spark Shell, while Spark Shell is designed for single session and client deploy mode, which has certain limitations for Kyuubis multi-tenant use cases.

Thanks to Spark Connect, a similar mechanism is required to support isolated sessions (i.e. isolated resources/classloader) so that each REPL user does not conflict with other users on the same cluster.

As of Spark 3.5, the core/sql already has infrastructure to support isolated resources/classloader, see details in

• SPARK-44078
• SPARK-44146
• SPARK-45856

Implementation steps

Investigate how could Kyuubi Spark Scala mode leverage those infrastructures to support session-level isolated resources/classloader in Spark 3.5+

Additional context

Introduction of 2024H1 Kyuubi Code Contribution Program

[beryllw]

Intersted in this, Please assign this task to me.

[pan3793]

@beryllw thanks, assgined

[zhaohehuhu]

Interesting task