Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[KYUUBI #6265] Resource isolation in Spark Scala mode #6331

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

beryllw
Copy link
Contributor

@beryllw beryllw commented Apr 22, 2024

🔍 Description

Issue References 🔗

This pull request fixes #6265

Describe Your Solution 🔧

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Types of changes 🔖

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪

Behavior Without This Pull Request ⚰️

Behavior With This Pull Request 🎉

Related Unit Tests


Checklist 📝

Be nice. Be informative.

@cxzl25 cxzl25 changed the title [KYUUBI #6265]Resource isolation in Spark Scala mode [KYUUBI #6265] Resource isolation in Spark Scala mode Apr 22, 2024
@codecov-commenter
Copy link

Codecov Report

Attention: Patch coverage is 82.14286% with 5 lines in your changes are missing coverage. Please review.

Project coverage is 58.45%. Comparing base (1591157) to head (e3c652e).
Report is 12 commits behind head on master.

Files Patch % Lines
...e/kyuubi/engine/spark/operation/ExecuteScala.scala 66.66% 3 Missing ⚠️
...a/org/apache/spark/sql/hive/HiveClientHelper.scala 85.71% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #6331      +/-   ##
============================================
- Coverage     58.47%   58.45%   -0.03%     
  Complexity       24       24              
============================================
  Files           653      655       +2     
  Lines         39769    39904     +135     
  Branches       5472     5484      +12     
============================================
+ Hits          23256    23326      +70     
- Misses        14030    14078      +48     
- Partials       2483     2500      +17     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@pan3793
Copy link
Member

pan3793 commented Apr 30, 2024

can you summarize your design/change on the PR description?

@beryllw
Copy link
Contributor Author

beryllw commented May 13, 2024

can you summarize your design/change on the PR description?

I've been quite busy recently,, sorry for not replying in time.

For now ExecuteScala use spark.sharedState.jarClassLoader to get the URLs for loading classes and resources, and then repl loaded.SharedState do not support isolated class loaded.

SparkSession use HiveSessionResourceLoader to support isolated class loaded.So we can get the URLs for loading classes and resources by HiveClientImpl.clientLoader.classLoader.getURLs.

Spark 3.5 use JobArtifactSet to support add jar/resources to different path.

We can use it to support add resources isolated.And then repl loaded isolated classes/resources from HiveSessionResourceLoader.

@beryllw
Copy link
Contributor Author

beryllw commented May 13, 2024

Only Spark 3.5 supports JobArtifactState. I need to use reflection to solve the compatibility problem, but I don’t know how to use reflection to call def withActiveJobArtifactState[T](state: JobArtifactState)(block: => T): T = {}.

@beryllw beryllw marked this pull request as draft May 13, 2024 06:28
@beryllw
Copy link
Contributor Author

beryllw commented May 20, 2024

@pan3793 cc, and need some help, thank you.

@pan3793 pan3793 requested a review from cxzl25 May 27, 2024 10:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[TASK][MEDIUM] Resource isolation in Spark Scala mode
3 participants