{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":114619105,"defaultBranch":"master","name":"kyuubi","ownerLogin":"apache","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2017-12-18T09:05:10.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/47359?v=4","public":true,"private":false,"isOrgOwned":true},"refInfo":{"name":"","listCacheKey":"v0:1715223039.0","currentOid":""},"activityList":{"items":[{"before":"6b3f3d25e85d402ec3c9ccae5c918dcb7abbf024","after":"3c3a8dd4e6a71ac124566228b88dec027f3ef1bb","ref":"refs/heads/branch-1.9","pushedAt":"2024-05-24T10:23:46.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"pan3793","name":"Cheng Pan","path":"/pan3793","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/26535726?s=80&v=4"},"commit":{"message":"[KYUUBI #6419] Bump Jetty 9.4.54.v20240208\n\n# :mag: Description\n\nRegular dependency upgrade.\n\n- https://github.com/jetty/jetty.project/releases/tag/jetty-9.4.53.v20231009\n- https://github.com/jetty/jetty.project/releases/tag/jetty-9.4.54.v20240208\n\n## Types of changes :bookmark:\n\n- [ ] Bugfix (non-breaking change which fixes an issue)\n- [ ] New feature (non-breaking change which adds functionality)\n- [ ] Breaking change (fix or feature that would cause existing functionality to change)\n\n## Test Plan πŸ§ͺ\n\nPass GHA\n\n---\n\n# Checklist πŸ“\n\n- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)\n\n**Be nice. Be informative.**\n\nCloses #6419 from pan3793/jetty-9.4.54.\n\nCloses #6419\n\na9530da3a [Cheng Pan] Bump Jetty 9.4.54.v20240208\n\nAuthored-by: Cheng Pan \nSigned-off-by: Cheng Pan \n(cherry picked from commit 6888874808723cb803b395f4b07526be9d8978ae)\nSigned-off-by: Cheng Pan ","shortMessageHtmlLink":"[KYUUBI #6419] Bump Jetty 9.4.54.v20240208"}},{"before":"d40f9155b0374e8144cbd6a3510a8ef323abb321","after":"6888874808723cb803b395f4b07526be9d8978ae","ref":"refs/heads/master","pushedAt":"2024-05-24T10:23:30.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"pan3793","name":"Cheng Pan","path":"/pan3793","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/26535726?s=80&v=4"},"commit":{"message":"[KYUUBI #6419] Bump Jetty 9.4.54.v20240208\n\n# :mag: Description\n\nRegular dependency upgrade.\n\n- https://github.com/jetty/jetty.project/releases/tag/jetty-9.4.53.v20231009\n- https://github.com/jetty/jetty.project/releases/tag/jetty-9.4.54.v20240208\n\n## Types of changes :bookmark:\n\n- [ ] Bugfix (non-breaking change which fixes an issue)\n- [ ] New feature (non-breaking change which adds functionality)\n- [ ] Breaking change (fix or feature that would cause existing functionality to change)\n\n## Test Plan πŸ§ͺ\n\nPass GHA\n\n---\n\n# Checklist πŸ“\n\n- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)\n\n**Be nice. Be informative.**\n\nCloses #6419 from pan3793/jetty-9.4.54.\n\nCloses #6419\n\na9530da3a [Cheng Pan] Bump Jetty 9.4.54.v20240208\n\nAuthored-by: Cheng Pan \nSigned-off-by: Cheng Pan ","shortMessageHtmlLink":"[KYUUBI #6419] Bump Jetty 9.4.54.v20240208"}},{"before":"549761720d7f2ecab43ee2798a00c09fcf4368c1","after":"6b3f3d25e85d402ec3c9ccae5c918dcb7abbf024","ref":"refs/heads/branch-1.9","pushedAt":"2024-05-24T10:21:51.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"pan3793","name":"Cheng Pan","path":"/pan3793","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/26535726?s=80&v=4"},"commit":{"message":"[KYUUBI #6418] Bump Jersey to 2.40\n\n# :mag: Description\n\nThis is a regular dependency upgrade.\n\nhttps://github.com/eclipse-ee4j/jersey/releases/tag/2.40\n\nThough 2.40 is not the latest version of the Jersey 2 serial, using a version adopted by Spark 3.5 is not harmful.\n\n## Types of changes :bookmark:\n\n- [ ] Bugfix (non-breaking change which fixes an issue)\n- [ ] New feature (non-breaking change which adds functionality)\n- [ ] Breaking change (fix or feature that would cause existing functionality to change)\n\n## Test Plan πŸ§ͺ\n\nPass GHA.\n\n---\n\n# Checklist πŸ“\n\n- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)\n\n**Be nice. Be informative.**\n\nCloses #6418 from pan3793/jersey-240.\n\nCloses #6418\n\ne925acdbd [Cheng Pan] update dependencyList\na853dc289 [Cheng Pan] Bump Jersey to 2.40\n\nAuthored-by: Cheng Pan \nSigned-off-by: Cheng Pan \n(cherry picked from commit d40f9155b0374e8144cbd6a3510a8ef323abb321)\nSigned-off-by: Cheng Pan ","shortMessageHtmlLink":"[KYUUBI #6418] Bump Jersey to 2.40"}},{"before":"4702f5a12bc4fb847e704c66c869341c41f9ce18","after":"d40f9155b0374e8144cbd6a3510a8ef323abb321","ref":"refs/heads/master","pushedAt":"2024-05-24T10:21:39.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"pan3793","name":"Cheng Pan","path":"/pan3793","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/26535726?s=80&v=4"},"commit":{"message":"[KYUUBI #6418] Bump Jersey to 2.40\n\n# :mag: Description\n\nThis is a regular dependency upgrade.\n\nhttps://github.com/eclipse-ee4j/jersey/releases/tag/2.40\n\nThough 2.40 is not the latest version of the Jersey 2 serial, using a version adopted by Spark 3.5 is not harmful.\n\n## Types of changes :bookmark:\n\n- [ ] Bugfix (non-breaking change which fixes an issue)\n- [ ] New feature (non-breaking change which adds functionality)\n- [ ] Breaking change (fix or feature that would cause existing functionality to change)\n\n## Test Plan πŸ§ͺ\n\nPass GHA.\n\n---\n\n# Checklist πŸ“\n\n- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)\n\n**Be nice. Be informative.**\n\nCloses #6418 from pan3793/jersey-240.\n\nCloses #6418\n\ne925acdbd [Cheng Pan] update dependencyList\na853dc289 [Cheng Pan] Bump Jersey to 2.40\n\nAuthored-by: Cheng Pan \nSigned-off-by: Cheng Pan ","shortMessageHtmlLink":"[KYUUBI #6418] Bump Jersey to 2.40"}},{"before":"89ea49f2035cc7d2978265bc2a5686a1816ea320","after":"549761720d7f2ecab43ee2798a00c09fcf4368c1","ref":"refs/heads/branch-1.9","pushedAt":"2024-05-24T10:20:42.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"pan3793","name":"Cheng Pan","path":"/pan3793","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/26535726?s=80&v=4"},"commit":{"message":"[KYUUBI #6416] Generate flattened POM\n\n# :mag: Description\n\nLeverage [Flatten Maven Plugin](https://www.mojohaus.org/flatten-maven-plugin/) to generate resolved `pom.xml`.\n\nThis recovers a dev case for Scala 2.13.\n\nWe usually perform `clean install` with `-am` and `-DskipTests` one time and perform `test` after changing code on specific modules, e.g.\n```\nbuild/mvn -Pscala-2.13 -pl externals/kyuubi-spark-sql-engine clean install -am -DskipTests\nbuild/mvn -Pscala-2.13 -pl externals/kyuubi-spark-sql-engine test\n```\n\nCurrently, the second step wrongly resolves to `*_2.12` dependencies due to the installed `pom.xml` referring to `*_${scala.binary.version}` while the default value of `scala.binary.version` is `2.12`.\n\n## Types of changes :bookmark:\n\n- [x] Bugfix (non-breaking change which fixes an issue)\n- [ ] New feature (non-breaking change which adds functionality)\n- [ ] Breaking change (fix or feature that would cause existing functionality to change)\n\n## Test Plan πŸ§ͺ\n\nThis change is verified locally by the above case.\n\n---\n\n# Checklist πŸ“\n\n- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)\n\n**Be nice. Be informative.**\n\nCloses #6416 from pan3793/flatten.\n\nCloses #6416\n\ndb7020c92 [Cheng Pan] Generate flattened pom.xml\n\nAuthored-by: Cheng Pan \nSigned-off-by: Cheng Pan \n(cherry picked from commit 4702f5a12bc4fb847e704c66c869341c41f9ce18)\nSigned-off-by: Cheng Pan ","shortMessageHtmlLink":"[KYUUBI #6416] Generate flattened POM"}},{"before":"e95faabe6cb09d597b5c0bae135cc4220ef06e67","after":"4702f5a12bc4fb847e704c66c869341c41f9ce18","ref":"refs/heads/master","pushedAt":"2024-05-24T10:20:29.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"pan3793","name":"Cheng Pan","path":"/pan3793","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/26535726?s=80&v=4"},"commit":{"message":"[KYUUBI #6416] Generate flattened POM\n\n# :mag: Description\n\nLeverage [Flatten Maven Plugin](https://www.mojohaus.org/flatten-maven-plugin/) to generate resolved `pom.xml`.\n\nThis recovers a dev case for Scala 2.13.\n\nWe usually perform `clean install` with `-am` and `-DskipTests` one time and perform `test` after changing code on specific modules, e.g.\n```\nbuild/mvn -Pscala-2.13 -pl externals/kyuubi-spark-sql-engine clean install -am -DskipTests\nbuild/mvn -Pscala-2.13 -pl externals/kyuubi-spark-sql-engine test\n```\n\nCurrently, the second step wrongly resolves to `*_2.12` dependencies due to the installed `pom.xml` referring to `*_${scala.binary.version}` while the default value of `scala.binary.version` is `2.12`.\n\n## Types of changes :bookmark:\n\n- [x] Bugfix (non-breaking change which fixes an issue)\n- [ ] New feature (non-breaking change which adds functionality)\n- [ ] Breaking change (fix or feature that would cause existing functionality to change)\n\n## Test Plan πŸ§ͺ\n\nThis change is verified locally by the above case.\n\n---\n\n# Checklist πŸ“\n\n- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)\n\n**Be nice. Be informative.**\n\nCloses #6416 from pan3793/flatten.\n\nCloses #6416\n\ndb7020c92 [Cheng Pan] Generate flattened pom.xml\n\nAuthored-by: Cheng Pan \nSigned-off-by: Cheng Pan ","shortMessageHtmlLink":"[KYUUBI #6416] Generate flattened POM"}},{"before":"7e6503d4563df8906351ec2341de368b301e294d","after":"89ea49f2035cc7d2978265bc2a5686a1816ea320","ref":"refs/heads/branch-1.9","pushedAt":"2024-05-24T09:14:11.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"cxzl25","name":null,"path":"/cxzl25","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3898450?s=80&v=4"},"commit":{"message":"[KYUUBI #6396] Add caching for KerberosAuthentication using ticketCache key\n\nThis pull request fixes #6396\n\nBy using a cache to store CachingKerberosAuthentication objects keyed by the ticket cache path, we ensure that each unique ticket cache path generates a distinct authentication object.\n\n- [ ] Bugfix (non-breaking change which fixes an issue)\n- [ ] New feature (non-breaking change which adds functionality)\n- [ ] Breaking change (fix or feature that would cause existing functionality to change)\n\n---\n\n- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)\n\n**Be nice. Be informative.**\n\nCloses #6401 from lsm1/branch-kyuubi-6396.\n\nCloses #6396\n\nbb8f738e1 [senmiaoliu] fix kyuubiClientTicketCache\n\nAuthored-by: senmiaoliu \nSigned-off-by: Shaoyun Chen \n(cherry picked from commit 4285abc3ae44e15465587e1afdf82cd003c5654a)","shortMessageHtmlLink":"[KYUUBI #6396] Add caching for KerberosAuthentication using ticketCac…"}},{"before":"67b09f3eb0a378788455ad1bd70fe7bf08b24878","after":"7e6503d4563df8906351ec2341de368b301e294d","ref":"refs/heads/branch-1.9","pushedAt":"2024-05-24T08:28:23.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"pan3793","name":"Cheng Pan","path":"/pan3793","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/26535726?s=80&v=4"},"commit":{"message":"[KYUUBI #6415] Manage Jersey in kyuubi-server module\n\n# :mag: Description\n\nWe can not manage Jersey dependencies globally because SPARK-47118 (4.0.0) upgrades Jersey from 2 to 3, using an unmatched Jersey version in modules like Spark SQL engine would fail Spark to start thus abort UT.\n\n## Types of changes :bookmark:\n\n- [x] Bugfix (non-breaking change which fixes an issue)\n- [ ] New feature (non-breaking change which adds functionality)\n- [ ] Breaking change (fix or feature that would cause existing functionality to change)\n\n## Test Plan πŸ§ͺ\n\nTested with Spark 4.0\n\n---\n\n# Checklist πŸ“\n\n- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)\n\n**Be nice. Be informative.**\n\nCloses #6415 from pan3793/server-jersey.\n\nCloses #6415\n\n804d6367b [Cheng Pan] nit\nfcb6ee573 [Cheng Pan] manage jersey in kyuubi-server module\n\nAuthored-by: Cheng Pan \nSigned-off-by: Cheng Pan \n(cherry picked from commit e95faabe6cb09d597b5c0bae135cc4220ef06e67)\nSigned-off-by: Cheng Pan ","shortMessageHtmlLink":"[KYUUBI #6415] Manage Jersey in kyuubi-server module"}},{"before":"b4065511892cbeb9be09084d94f172287d62700b","after":"e95faabe6cb09d597b5c0bae135cc4220ef06e67","ref":"refs/heads/master","pushedAt":"2024-05-24T08:28:08.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"pan3793","name":"Cheng Pan","path":"/pan3793","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/26535726?s=80&v=4"},"commit":{"message":"[KYUUBI #6415] Manage Jersey in kyuubi-server module\n\n# :mag: Description\n\nWe can not manage Jersey dependencies globally because SPARK-47118 (4.0.0) upgrades Jersey from 2 to 3, using an unmatched Jersey version in modules like Spark SQL engine would fail Spark to start thus abort UT.\n\n## Types of changes :bookmark:\n\n- [x] Bugfix (non-breaking change which fixes an issue)\n- [ ] New feature (non-breaking change which adds functionality)\n- [ ] Breaking change (fix or feature that would cause existing functionality to change)\n\n## Test Plan πŸ§ͺ\n\nTested with Spark 4.0\n\n---\n\n# Checklist πŸ“\n\n- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)\n\n**Be nice. Be informative.**\n\nCloses #6415 from pan3793/server-jersey.\n\nCloses #6415\n\n804d6367b [Cheng Pan] nit\nfcb6ee573 [Cheng Pan] manage jersey in kyuubi-server module\n\nAuthored-by: Cheng Pan \nSigned-off-by: Cheng Pan ","shortMessageHtmlLink":"[KYUUBI #6415] Manage Jersey in kyuubi-server module"}},{"before":"38d67dd2e117cd250cc45885f30339f770999933","after":"67b09f3eb0a378788455ad1bd70fe7bf08b24878","ref":"refs/heads/branch-1.9","pushedAt":"2024-05-24T08:27:22.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"pan3793","name":"Cheng Pan","path":"/pan3793","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/26535726?s=80&v=4"},"commit":{"message":"[KYUUBI #6417] Remove scala compile args -Yno-adapted-args\n\n# :mag: Description\n\nIDEA complains when enabling `scala-2.13` profile.\n```\nscala: bad option: '-Yno-adapted-args'\n```\n\n\"image\"\n\nI also found Spark removes it in SPARK-29413\n\n## Types of changes :bookmark:\n\n- [x] Bugfix (non-breaking change which fixes an issue)\n- [ ] New feature (non-breaking change which adds functionality)\n- [ ] Breaking change (fix or feature that would cause existing functionality to change)\n\n## Test Plan πŸ§ͺ\n\nPass GHA, and verify locally with IDEA.\n\n---\n\n# Checklist πŸ“\n\n- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)\n\n**Be nice. Be informative.**\n\nCloses #6417 from pan3793/scala-opt.\n\nCloses #6417\n\n1464c8194 [Cheng Pan] Remove scala compile args -Yno-adapted-args\n\nAuthored-by: Cheng Pan \nSigned-off-by: Cheng Pan \n(cherry picked from commit b4065511892cbeb9be09084d94f172287d62700b)\nSigned-off-by: Cheng Pan ","shortMessageHtmlLink":"[KYUUBI #6417] Remove scala compile args -Yno-adapted-args"}},{"before":"4cbecdc12f1f930f5375dddf02c3e545716b3400","after":"b4065511892cbeb9be09084d94f172287d62700b","ref":"refs/heads/master","pushedAt":"2024-05-24T08:27:08.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"pan3793","name":"Cheng Pan","path":"/pan3793","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/26535726?s=80&v=4"},"commit":{"message":"[KYUUBI #6417] Remove scala compile args -Yno-adapted-args\n\n# :mag: Description\n\nIDEA complains when enabling `scala-2.13` profile.\n```\nscala: bad option: '-Yno-adapted-args'\n```\n\n\"image\"\n\nI also found Spark removes it in SPARK-29413\n\n## Types of changes :bookmark:\n\n- [x] Bugfix (non-breaking change which fixes an issue)\n- [ ] New feature (non-breaking change which adds functionality)\n- [ ] Breaking change (fix or feature that would cause existing functionality to change)\n\n## Test Plan πŸ§ͺ\n\nPass GHA, and verify locally with IDEA.\n\n---\n\n# Checklist πŸ“\n\n- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)\n\n**Be nice. Be informative.**\n\nCloses #6417 from pan3793/scala-opt.\n\nCloses #6417\n\n1464c8194 [Cheng Pan] Remove scala compile args -Yno-adapted-args\n\nAuthored-by: Cheng Pan \nSigned-off-by: Cheng Pan ","shortMessageHtmlLink":"[KYUUBI #6417] Remove scala compile args -Yno-adapted-args"}},{"before":"b89c185eec330143c204eb84ddb60f24624d1079","after":"4cbecdc12f1f930f5375dddf02c3e545716b3400","ref":"refs/heads/master","pushedAt":"2024-05-24T04:15:49.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"pan3793","name":"Cheng Pan","path":"/pan3793","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/26535726?s=80&v=4"},"commit":{"message":"[KYUUBI #6367] Flink SQL engine supports RenewDelegationToken\n\n# :mag: Description\n## Issue References πŸ”—\n\nThis pull request fixes #6367\n\n## Describe Your Solution πŸ”§\n\n+ Implement `RenewDelegationToken` method in `FlinkTBinaryFrontendService`.\n+ Pass `kyuubi.engine.credentials` configuration when starting flink engine.\n\n## Types of changes :bookmark:\n\n- [ ] Bugfix (non-breaking change which fixes an issue)\n- [X] New feature (non-breaking change which adds functionality)\n- [ ] Breaking change (fix or feature that would cause existing functionality to change)\n\n## Test Plan πŸ§ͺ\n\n#### Behavior Without This Pull Request :coffin:\n\n#### Behavior With This Pull Request :tada:\ntest connection:\n\n```\n\"jdbc:hive2://hadoop-master1.orb.local:10009/default;hive.server2.proxy.user=spark;principal=kyuubi/_HOSTTEST.ORG?kyuubi.engine.type=FLINK_SQL;flink.execution.target=yarn-application\"\n```\n\nflink engine builder command:\n\n![image](https://github.com/apache/kyuubi/assets/17894939/dcdb8466-c423-464d-8119-9c4236f17ce7)\n\njobmanager log:\n\n```\n2024-05-22 07:46:46,545 INFO org.apache.kyuubi.engine.flink.FlinkTBinaryFrontendService [] - Add new unknown token Kind: HIVE_DELEGATION_TOKEN, Service: , Ident: 00 05 73 70 61 72 6b 04 68 69 76 65 28 6b 79 75 75 62 69 2f 68 61 64 6f 6f 70 2d 6d 61 73 74 65 72 31 2e 6f 72 62 2e 6c 6f 63 61 6c 40 54 45 53 54 2e 4f 52 47 8a 01 8f 9f 3f d5 4c 8a 01 8f c3 4c 59 4c 0b 06\n2024-05-22 07:46:46,547 WARN org.apache.kyuubi.engine.flink.FlinkTBinaryFrontendService [] - Ignore token with earlier issue date: Kind: HDFS_DELEGATION_TOKEN, Service: 172.20.0.5:8020, Ident: (token for spark: HDFS_DELEGATION_TOKEN owner=spark, renewer=spark, realUser=kyuubi/hadoop-master1.orb.localTEST.ORG, issueDate=1716363711750, maxDate=1716968511750, sequenceNumber=15, masterKeyId=7)\n2024-05-22 07:46:46,548 INFO org.apache.kyuubi.engine.flink.FlinkTBinaryFrontendService [] - Update delegation tokens. The number of tokens sent by the server is 2. The actual number of updated tokens is 1.\n\n```\n\n#### Related Unit Tests\n\n---\n\n# Checklist πŸ“\n\n- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)\n\n**Be nice. Be informative.**\n\nCloses #6371 from wForget/KYUUBI-6367.\n\nCloses #6367\n\n83b402aa0 [wforget] Revert \"change Base64 encoder/decoder\"\nf5c08eb45 [wforget] change Base64 encoder/decoder\ne8c66dfc5 [wforget] fix test\ne59820b3e [wforget] [KYUUBI #6367] Support RenewDelegationToken for flink sql engine\n\nAuthored-by: wforget <643348094@qq.com>\nSigned-off-by: Cheng Pan ","shortMessageHtmlLink":"[KYUUBI #6367] Flink SQL engine supports RenewDelegationToken"}},{"before":"5fffdc048923d119121ec10dd8628abfcc851a22","after":"38d67dd2e117cd250cc45885f30339f770999933","ref":"refs/heads/branch-1.9","pushedAt":"2024-05-23T06:27:04.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"pan3793","name":"Cheng Pan","path":"/pan3793","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/26535726?s=80&v=4"},"commit":{"message":"[KYUUBI #6413] SPARK_HOME detection supports Spark 4\n\n# Description\n\nWhen `SPARK_HOME` is not set explicitly, the Kyuubi server supports detecting it based on Scala versions, while the rules are not applicable for Spark 4.\n\nThis PR enhances the SPARK_HOME detection logic to make it support both Spark 3 and Spark 4.\n\nThe above logic is mainly used for testing purposes, the change does not affect users who configure `SPARK_HOME` in `kyuubi-env.sh`.\n\n## Types of changes\n\n- [ ] Bugfix (non-breaking change which fixes an issue)\n- [x] New feature (non-breaking change which adds functionality)\n- [ ] Breaking change (fix or feature that would cause existing functionality to change)\n\n## Test Plan\n\n#### Related Unit Tests\n\n- `SparkProcessBuilderSuite`\n\n---\n\n# Checklist πŸ“\n\n- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)\n\n**Be nice. Be informative.**\n\nCloses #6413 from pan3793/spark4-home.\n\nCloses #6413\n\n20e71fd7d [Cheng Pan] SPARK_HOME detection supports Spark 4\n\nAuthored-by: Cheng Pan \nSigned-off-by: Cheng Pan \n(cherry picked from commit b89c185eec330143c204eb84ddb60f24624d1079)\nSigned-off-by: Cheng Pan ","shortMessageHtmlLink":"[KYUUBI #6413] SPARK_HOME detection supports Spark 4"}},{"before":"a95ff125fc7a6bc1ebc78cda551e1ae9e5e0dac9","after":"b89c185eec330143c204eb84ddb60f24624d1079","ref":"refs/heads/master","pushedAt":"2024-05-23T06:26:44.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"pan3793","name":"Cheng Pan","path":"/pan3793","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/26535726?s=80&v=4"},"commit":{"message":"[KYUUBI #6413] SPARK_HOME detection supports Spark 4\n\n# Description\n\nWhen `SPARK_HOME` is not set explicitly, the Kyuubi server supports detecting it based on Scala versions, while the rules are not applicable for Spark 4.\n\nThis PR enhances the SPARK_HOME detection logic to make it support both Spark 3 and Spark 4.\n\nThe above logic is mainly used for testing purposes, the change does not affect users who configure `SPARK_HOME` in `kyuubi-env.sh`.\n\n## Types of changes\n\n- [ ] Bugfix (non-breaking change which fixes an issue)\n- [x] New feature (non-breaking change which adds functionality)\n- [ ] Breaking change (fix or feature that would cause existing functionality to change)\n\n## Test Plan\n\n#### Related Unit Tests\n\n- `SparkProcessBuilderSuite`\n\n---\n\n# Checklist πŸ“\n\n- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)\n\n**Be nice. Be informative.**\n\nCloses #6413 from pan3793/spark4-home.\n\nCloses #6413\n\n20e71fd7d [Cheng Pan] SPARK_HOME detection supports Spark 4\n\nAuthored-by: Cheng Pan \nSigned-off-by: Cheng Pan ","shortMessageHtmlLink":"[KYUUBI #6413] SPARK_HOME detection supports Spark 4"}},{"before":"1620b6df950c097c47a2bbbbc26a1d37d09aaf1a","after":"5f733a2a43d99c207fc1f2f57810427806eab900","ref":"refs/heads/branch-1.8","pushedAt":"2024-05-23T06:25:05.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"pan3793","name":"Cheng Pan","path":"/pan3793","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/26535726?s=80&v=4"},"commit":{"message":"[KYUUBI #6410] Check if asyncRequestExecutor is initialized before shutdown\n\n# :mag: Description\n## Issue References πŸ”—\n\nThis pull request fixes #6410\n\n## Describe Your Solution πŸ”§\n\nCheck if asyncRequestExecutor is initialized before shutdown to avoid NPE.\n\n## Types of changes :bookmark:\n\n- [X] Bugfix (non-breaking change which fixes an issue)\n- [ ] New feature (non-breaking change which adds functionality)\n- [ ] Breaking change (fix or feature that would cause existing functionality to change)\n\n## Test Plan πŸ§ͺ\n\n#### Behavior Without This Pull Request :coffin:\n\n#### Behavior With This Pull Request :tada:\n\n#### Related Unit Tests\n\n---\n\n# Checklist πŸ“\n\n- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)\n\n**Be nice. Be informative.**\n\nCloses #6414 from wForget/KYUUBI-6410.\n\nCloses #6410\n\n1b7d35efe [wforget] comment\nb83ea22cf [wforget] Check if asyncRequestExecutor is initialized before shutdown\n\nAuthored-by: wforget <643348094@qq.com>\nSigned-off-by: Cheng Pan \n(cherry picked from commit a95ff125fc7a6bc1ebc78cda551e1ae9e5e0dac9)\nSigned-off-by: Cheng Pan ","shortMessageHtmlLink":"[KYUUBI #6410] Check if asyncRequestExecutor is initialized before sh…"}},{"before":"ca9fa49b37a32b451466b577fdb1065fbca1b1e7","after":"5fffdc048923d119121ec10dd8628abfcc851a22","ref":"refs/heads/branch-1.9","pushedAt":"2024-05-23T06:24:48.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"pan3793","name":"Cheng Pan","path":"/pan3793","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/26535726?s=80&v=4"},"commit":{"message":"[KYUUBI #6410] Check if asyncRequestExecutor is initialized before shutdown\n\n# :mag: Description\n## Issue References πŸ”—\n\nThis pull request fixes #6410\n\n## Describe Your Solution πŸ”§\n\nCheck if asyncRequestExecutor is initialized before shutdown to avoid NPE.\n\n## Types of changes :bookmark:\n\n- [X] Bugfix (non-breaking change which fixes an issue)\n- [ ] New feature (non-breaking change which adds functionality)\n- [ ] Breaking change (fix or feature that would cause existing functionality to change)\n\n## Test Plan πŸ§ͺ\n\n#### Behavior Without This Pull Request :coffin:\n\n#### Behavior With This Pull Request :tada:\n\n#### Related Unit Tests\n\n---\n\n# Checklist πŸ“\n\n- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)\n\n**Be nice. Be informative.**\n\nCloses #6414 from wForget/KYUUBI-6410.\n\nCloses #6410\n\n1b7d35efe [wforget] comment\nb83ea22cf [wforget] Check if asyncRequestExecutor is initialized before shutdown\n\nAuthored-by: wforget <643348094@qq.com>\nSigned-off-by: Cheng Pan \n(cherry picked from commit a95ff125fc7a6bc1ebc78cda551e1ae9e5e0dac9)\nSigned-off-by: Cheng Pan ","shortMessageHtmlLink":"[KYUUBI #6410] Check if asyncRequestExecutor is initialized before sh…"}},{"before":"4285abc3ae44e15465587e1afdf82cd003c5654a","after":"a95ff125fc7a6bc1ebc78cda551e1ae9e5e0dac9","ref":"refs/heads/master","pushedAt":"2024-05-23T06:24:35.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"pan3793","name":"Cheng Pan","path":"/pan3793","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/26535726?s=80&v=4"},"commit":{"message":"[KYUUBI #6410] Check if asyncRequestExecutor is initialized before shutdown\n\n# :mag: Description\n## Issue References πŸ”—\n\nThis pull request fixes #6410\n\n## Describe Your Solution πŸ”§\n\nCheck if asyncRequestExecutor is initialized before shutdown to avoid NPE.\n\n## Types of changes :bookmark:\n\n- [X] Bugfix (non-breaking change which fixes an issue)\n- [ ] New feature (non-breaking change which adds functionality)\n- [ ] Breaking change (fix or feature that would cause existing functionality to change)\n\n## Test Plan πŸ§ͺ\n\n#### Behavior Without This Pull Request :coffin:\n\n#### Behavior With This Pull Request :tada:\n\n#### Related Unit Tests\n\n---\n\n# Checklist πŸ“\n\n- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)\n\n**Be nice. Be informative.**\n\nCloses #6414 from wForget/KYUUBI-6410.\n\nCloses #6410\n\n1b7d35efe [wforget] comment\nb83ea22cf [wforget] Check if asyncRequestExecutor is initialized before shutdown\n\nAuthored-by: wforget <643348094@qq.com>\nSigned-off-by: Cheng Pan ","shortMessageHtmlLink":"[KYUUBI #6410] Check if asyncRequestExecutor is initialized before sh…"}},{"before":"cb91dfaa4f8f1024beea2f90823b18073ead8c0d","after":"4285abc3ae44e15465587e1afdf82cd003c5654a","ref":"refs/heads/master","pushedAt":"2024-05-23T03:16:28.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"cxzl25","name":null,"path":"/cxzl25","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3898450?s=80&v=4"},"commit":{"message":"[KYUUBI #6396] Add caching for KerberosAuthentication using ticketCache key\n\n# :mag: Description\n## Issue References πŸ”—\n\nThis pull request fixes #6396\n\n## Describe Your Solution πŸ”§\n\nBy using a cache to store CachingKerberosAuthentication objects keyed by the ticket cache path, we ensure that each unique ticket cache path generates a distinct authentication object.\n\n## Types of changes :bookmark:\n\n- [ ] Bugfix (non-breaking change which fixes an issue)\n- [ ] New feature (non-breaking change which adds functionality)\n- [ ] Breaking change (fix or feature that would cause existing functionality to change)\n\n## Test Plan πŸ§ͺ\n\n#### Behavior Without This Pull Request :coffin:\n\n#### Behavior With This Pull Request :tada:\n\n#### Related Unit Tests\n\n---\n\n# Checklist πŸ“\n\n- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)\n\n**Be nice. Be informative.**\n\nCloses #6401 from lsm1/branch-kyuubi-6396.\n\nCloses #6396\n\nbb8f738e1 [senmiaoliu] fix kyuubiClientTicketCache\n\nAuthored-by: senmiaoliu \nSigned-off-by: Shaoyun Chen ","shortMessageHtmlLink":"[KYUUBI #6396] Add caching for KerberosAuthentication using ticketCac…"}},{"before":"5b592d07cab9c77cac136b496e3649f511627e57","after":"cb91dfaa4f8f1024beea2f90823b18073ead8c0d","ref":"refs/heads/master","pushedAt":"2024-05-22T09:55:03.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"pan3793","name":"Cheng Pan","path":"/pan3793","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/26535726?s=80&v=4"},"commit":{"message":"[KYUUBI #6408] Change to Base64 RFC4648 for handle guid and credential\n\n# :mag: Description\n\n`Base64.getMimeEncoder`(RFC2045) might generate newline when encoded chars exceed 76, so I changed it to `Base64.getEncoder`(RFC4648).\n## Types of changes :bookmark:\n\n- [x] Bugfix (non-breaking change which fixes an issue)\n- [ ] New feature (non-breaking change which adds functionality)\n- [ ] Breaking change (fix or feature that would cause existing functionality to change)\n\n## Test Plan πŸ§ͺ\n\nUse the changed BeeLine to connect to Kyuubi Server 1.9.0, everything goes well.\n\n```\n$ bin/beeline -u 'jdbc:kyuubi://0.0.0.0:10009/'\n...\nConnected to: Spark SQL (version 3.4.1)\nDriver: Kyuubi Project Hive JDBC Client (version 1.10.0-SNAPSHOT)\nBeeline version 1.10.0-SNAPSHOT by Apache Kyuubi\n0: jdbc:kyuubi://0.0.0.0:10009/>\n```\n\n---\n\n# Checklist πŸ“\n\n- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)\n\n**Be nice. Be informative.**\n\nCloses #6408 from wForget/base64.\n\nCloses #6408\n\na19f6f64f [wforget] change to Base64.getEncoder/getDecoder\n7052a9346 [wforget] change to Base64.getEncoder/getDecoder\n1b94ad991 [wforget] Change to Base64 urlEncoder/urlDecoder for handle guid and credential\n\nAuthored-by: wforget <643348094@qq.com>\nSigned-off-by: Cheng Pan ","shortMessageHtmlLink":"[KYUUBI #6408] Change to Base64 RFC4648 for handle guid and credential"}},{"before":"67e9639a73f3ee6c00efe0b5ecfb1bb691c0ec9d","after":"ca9fa49b37a32b451466b577fdb1065fbca1b1e7","ref":"refs/heads/branch-1.9","pushedAt":"2024-05-22T07:07:53.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"pan3793","name":"Cheng Pan","path":"/pan3793","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/26535726?s=80&v=4"},"commit":{"message":"[KYUUBI #6404] Fix HiveResult.toHiveString compatibility for Spark 4.0\n\n# :mag: Description\n\nSPARK-47911 introduced breaking changes for `HiveResult.toHiveString`, here we use reflection to fix the compatibility.\n\n## Types of changes :bookmark:\n\n- [x] Bugfix (non-breaking change which fixes an issue)\n- [ ] New feature (non-breaking change which adds functionality)\n- [ ] Breaking change (fix or feature that would cause existing functionality to change)\n\n## Test Plan πŸ§ͺ\n\n```\nbuild/mvn clean install -Pscala-2.13 -Pspark-master \\\n -pl externals/kyuubi-spark-sql-engine -am \\\n -Dtest=none -DwildcardSuites=org.apache.kyuubi.engine.spark.schema.RowSetSuite\n```\n\nbefore - compilation error\n```\n[INFO] --- scala-maven-plugin:4.8.0:compile (scala-compile-first) kyuubi-spark-sql-engine_2.13 ---\n...\n[ERROR] [Error] /home/kyuubi/apache-kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/schema/RowSet.scala:30: not enough arguments for method toHiveString: (a: (Any, org.apache.spark.sql.types.DataType), nested: Boolean, formatters: org.apache.spark.sql.execution.HiveResult.TimeFormatters, binaryFormatter: org.apache.spark.sql.execution.HiveResult.BinaryFormatter): String.\nUnspecified value parameter binaryFormatter.\n```\n\nafter - UT pass\n```\n[INFO] --- scalatest-maven-plugin:2.2.0:test (test) kyuubi-spark-sql-engine_2.13 ---\n[INFO] ScalaTest report directory: /home/kyuubi/apache-kyuubi/externals/kyuubi-spark-sql-engine/target/surefire-reports\nDiscovery starting.\nDiscovery completed in 1 second, 959 milliseconds.\nRun starting. Expected test count is: 3\nRowSetSuite:\n- column based set\n- row based set\n- to row set\nRun completed in 2 seconds, 712 milliseconds.\nTotal number of tests run: 3\nSuites: completed 2, aborted 0\nTests: succeeded 3, failed 0, canceled 0, ignored 0, pending 0\nAll tests passed.\n```\n\n---\n\n# Checklist πŸ“\n\n- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)\n\n**Be nice. Be informative.**\n\nCloses #6404 from pan3793/hive-string.\n\nCloses #6404\n\n6b3c743eb [Cheng Pan] fix breaking change of HiveResult.toHiveString caused by SPARK-47911\n\nAuthored-by: Cheng Pan \nSigned-off-by: Cheng Pan \n(cherry picked from commit 5b592d07cab9c77cac136b496e3649f511627e57)\nSigned-off-by: Cheng Pan ","shortMessageHtmlLink":"[KYUUBI #6404] Fix HiveResult.toHiveString compatibility for Spark 4.0"}},{"before":"9c1b779b1004736d65f8c2811b62a66aa1e9125b","after":"5b592d07cab9c77cac136b496e3649f511627e57","ref":"refs/heads/master","pushedAt":"2024-05-22T07:07:41.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"pan3793","name":"Cheng Pan","path":"/pan3793","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/26535726?s=80&v=4"},"commit":{"message":"[KYUUBI #6404] Fix HiveResult.toHiveString compatibility for Spark 4.0\n\n# :mag: Description\n\nSPARK-47911 introduced breaking changes for `HiveResult.toHiveString`, here we use reflection to fix the compatibility.\n\n## Types of changes :bookmark:\n\n- [x] Bugfix (non-breaking change which fixes an issue)\n- [ ] New feature (non-breaking change which adds functionality)\n- [ ] Breaking change (fix or feature that would cause existing functionality to change)\n\n## Test Plan πŸ§ͺ\n\n```\nbuild/mvn clean install -Pscala-2.13 -Pspark-master \\\n -pl externals/kyuubi-spark-sql-engine -am \\\n -Dtest=none -DwildcardSuites=org.apache.kyuubi.engine.spark.schema.RowSetSuite\n```\n\nbefore - compilation error\n```\n[INFO] --- scala-maven-plugin:4.8.0:compile (scala-compile-first) kyuubi-spark-sql-engine_2.13 ---\n...\n[ERROR] [Error] /home/kyuubi/apache-kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/schema/RowSet.scala:30: not enough arguments for method toHiveString: (a: (Any, org.apache.spark.sql.types.DataType), nested: Boolean, formatters: org.apache.spark.sql.execution.HiveResult.TimeFormatters, binaryFormatter: org.apache.spark.sql.execution.HiveResult.BinaryFormatter): String.\nUnspecified value parameter binaryFormatter.\n```\n\nafter - UT pass\n```\n[INFO] --- scalatest-maven-plugin:2.2.0:test (test) kyuubi-spark-sql-engine_2.13 ---\n[INFO] ScalaTest report directory: /home/kyuubi/apache-kyuubi/externals/kyuubi-spark-sql-engine/target/surefire-reports\nDiscovery starting.\nDiscovery completed in 1 second, 959 milliseconds.\nRun starting. Expected test count is: 3\nRowSetSuite:\n- column based set\n- row based set\n- to row set\nRun completed in 2 seconds, 712 milliseconds.\nTotal number of tests run: 3\nSuites: completed 2, aborted 0\nTests: succeeded 3, failed 0, canceled 0, ignored 0, pending 0\nAll tests passed.\n```\n\n---\n\n# Checklist πŸ“\n\n- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)\n\n**Be nice. Be informative.**\n\nCloses #6404 from pan3793/hive-string.\n\nCloses #6404\n\n6b3c743eb [Cheng Pan] fix breaking change of HiveResult.toHiveString caused by SPARK-47911\n\nAuthored-by: Cheng Pan \nSigned-off-by: Cheng Pan ","shortMessageHtmlLink":"[KYUUBI #6404] Fix HiveResult.toHiveString compatibility for Spark 4.0"}},{"before":"ac78125c1cc75deab8bcf04b631d9812856d46c8","after":"67e9639a73f3ee6c00efe0b5ecfb1bb691c0ec9d","ref":"refs/heads/branch-1.9","pushedAt":"2024-05-22T06:29:38.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"pan3793","name":"Cheng Pan","path":"/pan3793","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/26535726?s=80&v=4"},"commit":{"message":"[KYUUBI #6405] Spark engine supports both javax and jakarta ws.rs namespaces\n\n# :mag: Description\n\nSpark 4.0 upgraded Jersey from 2 to 3, and also migrated from `javax.ws.rs` to `jakarta.ws.rs` in SPARK-47118, this break the Spark SQL engine complication with Spark 4.0\n\n## Types of changes :bookmark:\n\n- [x] Bugfix (non-breaking change which fixes an issue)\n- [ ] New feature (non-breaking change which adds functionality)\n- [ ] Breaking change (fix or feature that would cause existing functionality to change)\n\n## Test Plan πŸ§ͺ\n\n```\nbuild/mvn clean install -Pscala-2.13 -Pspark-master \\\n -pl externals/kyuubi-spark-sql-engine -am -DskipTests\n```\n\nbefore\n```\n[INFO] --- scala-maven-plugin:4.8.0:compile (scala-compile-first) kyuubi-spark-sql-engine_2.13 ---\n[INFO] Compiler bridge file: /home/kyuubi/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.13-1.8.0-bin_2.13.8__61.0-1.8.0_20221110T195421.jar\n[INFO] compiler plugin: BasicArtifact(com.github.ghik,silencer-plugin_2.13.8,1.7.13,null)\n[INFO] compiling 61 Scala sources to /home/kyuubi/apache-kyuubi/externals/kyuubi-spark-sql-engine/target/scala-2.13/classes ...\n[ERROR] [Error] /home/kyuubi/apache-kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/ExecutePython.scala:27: object ws is not a member of package javax\n[ERROR] [Error] /home/kyuubi/apache-kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/ExecutePython.scala:307: not found: value UriBuilder\n[ERROR] [Error] /home/kyuubi/apache-kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/ExecutePython.scala:320: not found: value UriBuilder\n```\n\nafter\n```\n[INFO] --- scala-maven-plugin:4.8.0:compile (scala-compile-first) kyuubi-spark-sql-engine_2.13 ---\n[INFO] Compiler bridge file: /home/kyuubi/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.13-1.8.0-bin_2.13.8__61.0-1.8.0_20221110T195421.jar\n[INFO] compiler plugin: BasicArtifact(com.github.ghik,silencer-plugin_2.13.8,1.7.13,null)\n[INFO] compiling 61 Scala sources to /home/kyuubi/apache-kyuubi/externals/kyuubi-spark-sql-engine/target/scala-2.13/classes ...\n[INFO] compile in 19.2 s\n```\n\n---\n\n# Checklist πŸ“\n\n- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)\n\n**Be nice. Be informative.**\n\nCloses #6405 from pan3793/jersey.\n\nCloses #6405\n\n6cce23b01 [Cheng Pan] SPARK-47118 Jersey\n\nAuthored-by: Cheng Pan \nSigned-off-by: Cheng Pan \n(cherry picked from commit 9c1b779b1004736d65f8c2811b62a66aa1e9125b)\nSigned-off-by: Cheng Pan ","shortMessageHtmlLink":"[KYUUBI #6405] Spark engine supports both javax and jakarta ws.rs nam…"}},{"before":"586f6008bda6aabec4e1de7c60609ab2caadbc59","after":"9c1b779b1004736d65f8c2811b62a66aa1e9125b","ref":"refs/heads/master","pushedAt":"2024-05-22T06:29:17.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"pan3793","name":"Cheng Pan","path":"/pan3793","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/26535726?s=80&v=4"},"commit":{"message":"[KYUUBI #6405] Spark engine supports both javax and jakarta ws.rs namespaces\n\n# :mag: Description\n\nSpark 4.0 upgraded Jersey from 2 to 3, and also migrated from `javax.ws.rs` to `jakarta.ws.rs` in SPARK-47118, this break the Spark SQL engine complication with Spark 4.0\n\n## Types of changes :bookmark:\n\n- [x] Bugfix (non-breaking change which fixes an issue)\n- [ ] New feature (non-breaking change which adds functionality)\n- [ ] Breaking change (fix or feature that would cause existing functionality to change)\n\n## Test Plan πŸ§ͺ\n\n```\nbuild/mvn clean install -Pscala-2.13 -Pspark-master \\\n -pl externals/kyuubi-spark-sql-engine -am -DskipTests\n```\n\nbefore\n```\n[INFO] --- scala-maven-plugin:4.8.0:compile (scala-compile-first) kyuubi-spark-sql-engine_2.13 ---\n[INFO] Compiler bridge file: /home/kyuubi/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.13-1.8.0-bin_2.13.8__61.0-1.8.0_20221110T195421.jar\n[INFO] compiler plugin: BasicArtifact(com.github.ghik,silencer-plugin_2.13.8,1.7.13,null)\n[INFO] compiling 61 Scala sources to /home/kyuubi/apache-kyuubi/externals/kyuubi-spark-sql-engine/target/scala-2.13/classes ...\n[ERROR] [Error] /home/kyuubi/apache-kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/ExecutePython.scala:27: object ws is not a member of package javax\n[ERROR] [Error] /home/kyuubi/apache-kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/ExecutePython.scala:307: not found: value UriBuilder\n[ERROR] [Error] /home/kyuubi/apache-kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/ExecutePython.scala:320: not found: value UriBuilder\n```\n\nafter\n```\n[INFO] --- scala-maven-plugin:4.8.0:compile (scala-compile-first) kyuubi-spark-sql-engine_2.13 ---\n[INFO] Compiler bridge file: /home/kyuubi/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.13-1.8.0-bin_2.13.8__61.0-1.8.0_20221110T195421.jar\n[INFO] compiler plugin: BasicArtifact(com.github.ghik,silencer-plugin_2.13.8,1.7.13,null)\n[INFO] compiling 61 Scala sources to /home/kyuubi/apache-kyuubi/externals/kyuubi-spark-sql-engine/target/scala-2.13/classes ...\n[INFO] compile in 19.2 s\n```\n\n---\n\n# Checklist πŸ“\n\n- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)\n\n**Be nice. Be informative.**\n\nCloses #6405 from pan3793/jersey.\n\nCloses #6405\n\n6cce23b01 [Cheng Pan] SPARK-47118 Jersey\n\nAuthored-by: Cheng Pan \nSigned-off-by: Cheng Pan ","shortMessageHtmlLink":"[KYUUBI #6405] Spark engine supports both javax and jakarta ws.rs nam…"}},{"before":"304e222f9f2a7a84a7ed93bacf1e932f70c9f4cb","after":"ac78125c1cc75deab8bcf04b631d9812856d46c8","ref":"refs/heads/branch-1.9","pushedAt":"2024-05-22T05:03:32.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"pan3793","name":"Cheng Pan","path":"/pan3793","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/26535726?s=80&v=4"},"commit":{"message":"[KYUUBI #6399] Spark Kyuubi UI supports both javax and jakarta servlet namespaces\n\n# :mag: Description\n\nSpark 4.0 migrated from `javax.servlet` to `jakarta.servlet` in SPARK-47118, which breaks the binary compatibility of `SparkUITab` and `WebUIPage` that Kyuubi used, thus breaking the previous assumption of Kyuubi Spark SQL engine: single jar built with default Spark version, compatible with all supported versions of Spark runtime.\n\n## Describe Your Solution πŸ”§\n\nThis PR uses bytebuddy to dynamically generate classes and Java reflection find and dispatch method invocation in runtime, to recover the existing compatibility of Kyuubi Spark SQL engine.\n\n## Types of changes :bookmark:\n\n- [ ] Bugfix (non-breaking change which fixes an issue)\n- [x] New feature (non-breaking change which adds functionality)\n- [ ] Breaking change (fix or feature that would cause existing functionality to change)\n\n## Test Plan πŸ§ͺ\n\nBuild with Spark 3.5\n```\nbuild/dist --tgz --web-ui --spark-provided --flink-provided --hive-provided -Pspark-3.5\n```\n\nIt produces both Scala 2.12 and 2.13 Spark SQL engine jars\n- `kyuubi-spark-sql-engine_2.12-1.10.0-SNAPSHOT.jar`\n- `kyuubi-spark-sql-engine_2.13-1.10.0-SNAPSHOT.jar`\n\nRun with Spark 3.4 Scala 2.12\n\n\"image\"\n\nRun with Spark 3.5 Scala 2.13\n\n\"image\"\n\nRun with Spark 4.0.0-preview1 Scala 2.13\n\n\"image\"\n\n---\n\n# Checklist πŸ“\n\n- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)\n\n**Be nice. Be informative.**\n\nCloses #6399 from pan3793/ui-4.0.\n\nCloses #6399\n\ne0104f6df [Cheng Pan] nit\na2f9df4fa [Cheng Pan] nit\nc369ab2e3 [Cheng Pan] nit\nec1c45f66 [Cheng Pan] nit\n3e05744d6 [Cheng Pan] fix\na7e38cc1e [Cheng Pan] nit\nfa14a0d98 [Cheng Pan] refactor\n9d0ce6111 [Cheng Pan] A work version\nfc78b58e4 [Cheng Pan] fix startup\nd74c1c0fe [Cheng Pan] fix\n50066f563 [Cheng Pan] nit\nf5ad4c760 [Cheng Pan] Kyuubi UI supports Spark 4.0\n\nAuthored-by: Cheng Pan \nSigned-off-by: Cheng Pan \n(cherry picked from commit 586f6008bda6aabec4e1de7c60609ab2caadbc59)\nSigned-off-by: Cheng Pan ","shortMessageHtmlLink":"[KYUUBI #6399] Spark Kyuubi UI supports both javax and jakarta servle…"}},{"before":"b810bcc7eaf9e5eeae0dea5d6832d7fb1feadb62","after":"586f6008bda6aabec4e1de7c60609ab2caadbc59","ref":"refs/heads/master","pushedAt":"2024-05-22T05:03:10.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"pan3793","name":"Cheng Pan","path":"/pan3793","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/26535726?s=80&v=4"},"commit":{"message":"[KYUUBI #6399] Spark Kyuubi UI supports both javax and jakarta servlet namespaces\n\n# :mag: Description\n\nSpark 4.0 migrated from `javax.servlet` to `jakarta.servlet` in SPARK-47118, which breaks the binary compatibility of `SparkUITab` and `WebUIPage` that Kyuubi used, thus breaking the previous assumption of Kyuubi Spark SQL engine: single jar built with default Spark version, compatible with all supported versions of Spark runtime.\n\n## Describe Your Solution πŸ”§\n\nThis PR uses bytebuddy to dynamically generate classes and Java reflection find and dispatch method invocation in runtime, to recover the existing compatibility of Kyuubi Spark SQL engine.\n\n## Types of changes :bookmark:\n\n- [ ] Bugfix (non-breaking change which fixes an issue)\n- [x] New feature (non-breaking change which adds functionality)\n- [ ] Breaking change (fix or feature that would cause existing functionality to change)\n\n## Test Plan πŸ§ͺ\n\nBuild with Spark 3.5\n```\nbuild/dist --tgz --web-ui --spark-provided --flink-provided --hive-provided -Pspark-3.5\n```\n\nIt produces both Scala 2.12 and 2.13 Spark SQL engine jars\n- `kyuubi-spark-sql-engine_2.12-1.10.0-SNAPSHOT.jar`\n- `kyuubi-spark-sql-engine_2.13-1.10.0-SNAPSHOT.jar`\n\nRun with Spark 3.4 Scala 2.12\n\n\"image\"\n\nRun with Spark 3.5 Scala 2.13\n\n\"image\"\n\nRun with Spark 4.0.0-preview1 Scala 2.13\n\n\"image\"\n\n---\n\n# Checklist πŸ“\n\n- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)\n\n**Be nice. Be informative.**\n\nCloses #6399 from pan3793/ui-4.0.\n\nCloses #6399\n\ne0104f6df [Cheng Pan] nit\na2f9df4fa [Cheng Pan] nit\nc369ab2e3 [Cheng Pan] nit\nec1c45f66 [Cheng Pan] nit\n3e05744d6 [Cheng Pan] fix\na7e38cc1e [Cheng Pan] nit\nfa14a0d98 [Cheng Pan] refactor\n9d0ce6111 [Cheng Pan] A work version\nfc78b58e4 [Cheng Pan] fix startup\nd74c1c0fe [Cheng Pan] fix\n50066f563 [Cheng Pan] nit\nf5ad4c760 [Cheng Pan] Kyuubi UI supports Spark 4.0\n\nAuthored-by: Cheng Pan \nSigned-off-by: Cheng Pan ","shortMessageHtmlLink":"[KYUUBI #6399] Spark Kyuubi UI supports both javax and jakarta servle…"}},{"before":"6883552a8142c06b1781c5e3d862386f949dac15","after":"304e222f9f2a7a84a7ed93bacf1e932f70c9f4cb","ref":"refs/heads/branch-1.9","pushedAt":"2024-05-21T14:24:15.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"pan3793","name":"Cheng Pan","path":"/pan3793","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/26535726?s=80&v=4"},"commit":{"message":"Revert \"[KYUUBI #6390] Temporarily disable UI Tab for Spark 4.0 and above\"\n\nThis reverts commit bd33b7b176cf11a647650798456f51c405de9f81.","shortMessageHtmlLink":"Revert \"[KYUUBI #6390] Temporarily disable UI Tab for Spark 4.0 and a…"}},{"before":"97aa4f7e1fd758adbe1314217ef51dd6a23cc9fd","after":"b810bcc7eaf9e5eeae0dea5d6832d7fb1feadb62","ref":"refs/heads/master","pushedAt":"2024-05-21T14:23:29.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"pan3793","name":"Cheng Pan","path":"/pan3793","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/26535726?s=80&v=4"},"commit":{"message":"Revert \"[KYUUBI #6390] Temporarily disable UI Tab for Spark 4.0 and above\"\n\nThis reverts commit 70eadc15add3fe102c3a67a199d1796751eed0ca.","shortMessageHtmlLink":"Revert \"[KYUUBI #6390] Temporarily disable UI Tab for Spark 4.0 and a…"}},{"before":"c36e535703ce127b7034ab29b663a52f7895e19f","after":"6883552a8142c06b1781c5e3d862386f949dac15","ref":"refs/heads/branch-1.9","pushedAt":"2024-05-21T02:30:48.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"turboFei","name":"Fei Wang","path":"/turboFei","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6757692?s=80&v=4"},"commit":{"message":"[KYUUBI #6400] Fix memory leak when using saveToFile\n\n# :mag: Description\n## Issue References πŸ”—\n\nFix memory leak when using saveToFile mode.\n\nFYI: https://stackoverflow.com/questions/45649044/scala-stream-iterate-and-memory-management\n\nStream is IterableAgain, which means, that it will keep all the elements you iterate through in case you want to see them again.\n\n## Describe Your Solution πŸ”§\n\nPlease include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.\n\n## Types of changes :bookmark:\n\n- [ ] Bugfix (non-breaking change which fixes an issue)\n- [ ] New feature (non-breaking change which adds functionality)\n- [ ] Breaking change (fix or feature that would cause existing functionality to change)\n\n## Test Plan πŸ§ͺ\n\n#### Behavior Without This Pull Request :coffin:\n\n#### Behavior With This Pull Request :tada:\n\n#### Related Unit Tests\n\n---\n\n# Checklist πŸ“\n\n- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)\n\n**Be nice. Be informative.**\n\nCloses #6400 from turboFei/memory_leak.\n\nCloses #6400\n\ncdea358d6 [Wang, Fei] fix memory leak\n\nAuthored-by: Wang, Fei \nSigned-off-by: Wang, Fei \n(cherry picked from commit 97aa4f7e1fd758adbe1314217ef51dd6a23cc9fd)\nSigned-off-by: Wang, Fei ","shortMessageHtmlLink":"[KYUUBI #6400] Fix memory leak when using saveToFile"}},{"before":"522a28e1d56808776d0090108f51fbd52dcdf6cc","after":"97aa4f7e1fd758adbe1314217ef51dd6a23cc9fd","ref":"refs/heads/master","pushedAt":"2024-05-21T02:30:41.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"turboFei","name":"Fei Wang","path":"/turboFei","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/6757692?s=80&v=4"},"commit":{"message":"[KYUUBI #6400] Fix memory leak when using saveToFile\n\n# :mag: Description\n## Issue References πŸ”—\n\nFix memory leak when using saveToFile mode.\n\nFYI: https://stackoverflow.com/questions/45649044/scala-stream-iterate-and-memory-management\n\nStream is IterableAgain, which means, that it will keep all the elements you iterate through in case you want to see them again.\n\n## Describe Your Solution πŸ”§\n\nPlease include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.\n\n## Types of changes :bookmark:\n\n- [ ] Bugfix (non-breaking change which fixes an issue)\n- [ ] New feature (non-breaking change which adds functionality)\n- [ ] Breaking change (fix or feature that would cause existing functionality to change)\n\n## Test Plan πŸ§ͺ\n\n#### Behavior Without This Pull Request :coffin:\n\n#### Behavior With This Pull Request :tada:\n\n#### Related Unit Tests\n\n---\n\n# Checklist πŸ“\n\n- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)\n\n**Be nice. Be informative.**\n\nCloses #6400 from turboFei/memory_leak.\n\nCloses #6400\n\ncdea358d6 [Wang, Fei] fix memory leak\n\nAuthored-by: Wang, Fei \nSigned-off-by: Wang, Fei ","shortMessageHtmlLink":"[KYUUBI #6400] Fix memory leak when using saveToFile"}},{"before":"c9e19f0a62496a6866030c73506978c49186dd13","after":"522a28e1d56808776d0090108f51fbd52dcdf6cc","ref":"refs/heads/master","pushedAt":"2024-05-20T14:03:54.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"pan3793","name":"Cheng Pan","path":"/pan3793","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/26535726?s=80&v=4"},"commit":{"message":"[KYUUBI #6398] Fix lineage plugin UT for Spark 4.0\n\n# :mag: Description\n\n```\nbuild/mvn clean test -Pscala-2.13 -Pspark-master -pl :kyuubi-spark-lineage_2.13\n```\n\n```\n- test group by *** FAILED ***\n org.apache.spark.sql.catalyst.ExtendedAnalysisException: [DATATYPE_MISMATCH.BINARY_OP_WRONG_TYPE] Cannot resolve \"(b + c)\" due to data type mismatch: the binary operator requires the input type (\"NUMERIC\" or \"INTERVAL DAY TO SECOND\" or \"INTERVAL YEAR TO MONTH\" or \"INTERVAL\"), not \"STRING\". SQLSTATE: 42K09; line 1 pos 59;\n'InsertIntoStatement RelationV2[a#546, b#547, c#548] v2_catalog.db.t1 v2_catalog.db.t1, false, false, false\n+- 'Aggregate [a#543], [a#543, unresolvedalias('count(distinct (b#544 + c#545))), (count(distinct b#544) * count(distinct c#545)) AS (count(DISTINCT b) * count(DISTINCT c))#551L]\n +- SubqueryAlias v2_catalog.db.t2\n +- RelationV2[a#543, b#544, c#545] v2_catalog.db.t2 v2_catalog.db.t2\n at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.dataTypeMismatch(package.scala:73)\n at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis0$7(CheckAnalysis.scala:315)\n at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis0$7$adapted(CheckAnalysis.scala:302)\n at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:244)\n at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1(TreeNode.scala:243)\n at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1$adapted(TreeNode.scala:243)\n at scala.collection.immutable.Vector.foreach(Vector.scala:1856)\n at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:243)\n at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1(TreeNode.scala:243)\n at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1$adapted(TreeNode.scala:243)\n ...\n```\n\n## Types of changes :bookmark:\n\n- [x] Bugfix (non-breaking change which fixes an issue)\n- [ ] New feature (non-breaking change which adds functionality)\n- [ ] Breaking change (fix or feature that would cause existing functionality to change)\n\n## Test Plan πŸ§ͺ\n\nPass UT.\n\n---\n\n# Checklist πŸ“\n\n- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)\n\n**Be nice. Be informative.**\n\nCloses #6398 from pan3793/lineage-fix.\n\nCloses #6398\n\nafce6b880 [Cheng Pan] Fix lineage plugin UT for Spark 4.0\n\nAuthored-by: Cheng Pan \nSigned-off-by: Cheng Pan ","shortMessageHtmlLink":"[KYUUBI #6398] Fix lineage plugin UT for Spark 4.0"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAEUunIkgA","startCursor":null,"endCursor":null}},"title":"Activity Β· apache/kyuubi"}