{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":436912897,"defaultBranch":"main","name":"celeborn","ownerLogin":"apache","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2021-12-10T08:57:16.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/47359?v=4","public":true,"private":false,"isOrgOwned":true},"refInfo":{"name":"","listCacheKey":"v0:1718159305.0","currentOid":""},"activityList":{"items":[{"before":"bbb1572e21243369371f4b17e7c6f2e66e181b18","after":"a604b6c3ae3e99c0706efbed71f5977b16c93570","ref":"refs/heads/branch-0.5","pushedAt":"2024-06-12T08:15:19.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"SteNicholas","name":"Nicholas Jiang","path":"/SteNicholas","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/10048174?s=80&v=4"},"commit":{"message":"[CELEBORN-1457] Avoid NPE during shuffle data cleanup\n\n### What changes were proposed in this pull request?\nAvoid NPE during shuffle data cleanup by checking for null LevelDB.\n\n### Why are the changes needed?\nIf the LevelDB in StorageManager fails to initialize, the db will be null. This will cause a java.lang.NullPointerException when storageManager.cleanupExpiredShuffleKey(expiredShuffleKeys) is called, and the shuffle data in expiredShuffleKeys will not be cleaned up. The worker's disk may be filled up as a result.\n\n### Does this PR introduce _any_ user-facing change?\nNo\n\n### How was this patch tested?\nManual Testing\n\nCloses #2553 from jiang13021/celeborn-1457.\n\nAuthored-by: jiang13021 \nSigned-off-by: SteNicholas \n(cherry picked from commit 8e2fe74a60a690911d07db6744b167d66e4c45d1)\nSigned-off-by: SteNicholas ","shortMessageHtmlLink":"[CELEBORN-1457] Avoid NPE during shuffle data cleanup"}},{"before":"52c82bb60cc5d710ca4fa0d72c3a20cb9f5c4b9d","after":"8e2fe74a60a690911d07db6744b167d66e4c45d1","ref":"refs/heads/main","pushedAt":"2024-06-12T08:15:01.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"SteNicholas","name":"Nicholas Jiang","path":"/SteNicholas","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/10048174?s=80&v=4"},"commit":{"message":"[CELEBORN-1457] Avoid NPE during shuffle data cleanup\n\n### What changes were proposed in this pull request?\nAvoid NPE during shuffle data cleanup by checking for null LevelDB.\n\n### Why are the changes needed?\nIf the LevelDB in StorageManager fails to initialize, the db will be null. This will cause a java.lang.NullPointerException when storageManager.cleanupExpiredShuffleKey(expiredShuffleKeys) is called, and the shuffle data in expiredShuffleKeys will not be cleaned up. The worker's disk may be filled up as a result.\n\n### Does this PR introduce _any_ user-facing change?\nNo\n\n### How was this patch tested?\nManual Testing\n\nCloses #2553 from jiang13021/celeborn-1457.\n\nAuthored-by: jiang13021 \nSigned-off-by: SteNicholas ","shortMessageHtmlLink":"[CELEBORN-1457] Avoid NPE during shuffle data cleanup"}},{"before":"916add749d2dc1b2cdea11cb5090cb5a916d21eb","after":"bbb1572e21243369371f4b17e7c6f2e66e181b18","ref":"refs/heads/branch-0.5","pushedAt":"2024-06-12T02:28:25.000Z","pushType":"push","commitsCount":2,"pusher":{"login":"FMX","name":"Ethan Feng","path":"/FMX","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4150993?s=80&v=4"},"commit":{"message":"Bump 0.5.1-SNAPSHOT","shortMessageHtmlLink":"Bump 0.5.1-SNAPSHOT"}},{"before":"bd6a314a9284461de1dfeee6ddecc788f4cbfc00","after":"916add749d2dc1b2cdea11cb5090cb5a916d21eb","ref":"refs/heads/branch-0.5","pushedAt":"2024-06-12T02:19:10.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"FMX","name":"Ethan Feng","path":"/FMX","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4150993?s=80&v=4"},"commit":{"message":"[CELEBORN-1456][FOLLOWUP] Fix license issue\n\n### What changes were proposed in this pull request?\nFix license issues.\n\n### Why are the changes needed?\nFix license and notice for release 0.5.0.\n\n### Does this PR introduce _any_ user-facing change?\nNO.\n\n### How was this patch tested?\nNO.\n\nCloses #2556 from FMX/fix-license.\n\nLead-authored-by: mingji \nCo-authored-by: Ethan Feng \nSigned-off-by: mingji \n(cherry picked from commit 52c82bb60cc5d710ca4fa0d72c3a20cb9f5c4b9d)\nSigned-off-by: mingji ","shortMessageHtmlLink":"[CELEBORN-1456][FOLLOWUP] Fix license issue"}},{"before":"6f77bd9ca2796560d914385b8308d6ffaa876d48","after":"52c82bb60cc5d710ca4fa0d72c3a20cb9f5c4b9d","ref":"refs/heads/main","pushedAt":"2024-06-12T02:18:40.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"FMX","name":"Ethan Feng","path":"/FMX","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4150993?s=80&v=4"},"commit":{"message":"[CELEBORN-1456][FOLLOWUP] Fix license issue\n\n### What changes were proposed in this pull request?\nFix license issues.\n\n### Why are the changes needed?\nFix license and notice for release 0.5.0.\n\n### Does this PR introduce _any_ user-facing change?\nNO.\n\n### How was this patch tested?\nNO.\n\nCloses #2556 from FMX/fix-license.\n\nLead-authored-by: mingji \nCo-authored-by: Ethan Feng \nSigned-off-by: mingji ","shortMessageHtmlLink":"[CELEBORN-1456][FOLLOWUP] Fix license issue"}},{"before":null,"after":"734c42a81c7ed9ebe0c7dbe389a85f31cba116d2","ref":"refs/heads/detached","pushedAt":"2024-06-11T08:17:09.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"FMX","name":"Ethan Feng","path":"/FMX","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4150993?s=80&v=4"},"commit":{"message":"Bump 0.5.0","shortMessageHtmlLink":"Bump 0.5.0"}},{"before":"4eea40e1ee82c9e9a394c40c7142ca9edf17030b","after":"bd6a314a9284461de1dfeee6ddecc788f4cbfc00","ref":"refs/heads/branch-0.5","pushedAt":"2024-06-11T08:13:20.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"FMX","name":"Ethan Feng","path":"/FMX","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4150993?s=80&v=4"},"commit":{"message":"[CELEBORN-1456] Fix LICENSE dependencies in LICENSE-binary\n\n### What changes were proposed in this pull request?\n\nFix LICENSE dependencies in `LICENSE-binary`.\n\n### Why are the changes needed?\n\nLICENSE of dependencies including ratis-metrics-default, ap-loader-all, jersey-server, jersey-container-servlet-core, jersey-hk2, jersey-media-json-jackson, jersey-media-multipart, mybatis are missing in `LICENSE-binary`. Meanwhile, dependency javax.servlet is removed, therefore the LICENSE of javax.servlet should be removed from `LICENSE-binary`.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nNo.\n\nCloses #2551 from SteNicholas/CELEBORN-1456.\n\nAuthored-by: SteNicholas \nSigned-off-by: mingji \n(cherry picked from commit 6f77bd9ca2796560d914385b8308d6ffaa876d48)\nSigned-off-by: mingji ","shortMessageHtmlLink":"[CELEBORN-1456] Fix LICENSE dependencies in LICENSE-binary"}},{"before":"22ecbbd157c972141080f67f0bec12527c4537a0","after":"6f77bd9ca2796560d914385b8308d6ffaa876d48","ref":"refs/heads/main","pushedAt":"2024-06-11T08:12:37.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"FMX","name":"Ethan Feng","path":"/FMX","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4150993?s=80&v=4"},"commit":{"message":"[CELEBORN-1456] Fix LICENSE dependencies in LICENSE-binary\n\n### What changes were proposed in this pull request?\n\nFix LICENSE dependencies in `LICENSE-binary`.\n\n### Why are the changes needed?\n\nLICENSE of dependencies including ratis-metrics-default, ap-loader-all, jersey-server, jersey-container-servlet-core, jersey-hk2, jersey-media-json-jackson, jersey-media-multipart, mybatis are missing in `LICENSE-binary`. Meanwhile, dependency javax.servlet is removed, therefore the LICENSE of javax.servlet should be removed from `LICENSE-binary`.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nNo.\n\nCloses #2551 from SteNicholas/CELEBORN-1456.\n\nAuthored-by: SteNicholas \nSigned-off-by: mingji ","shortMessageHtmlLink":"[CELEBORN-1456] Fix LICENSE dependencies in LICENSE-binary"}},{"before":"20c730f0f6ff5287f7e3ce83a0a2c1884ca05249","after":"4eea40e1ee82c9e9a394c40c7142ca9edf17030b","ref":"refs/heads/branch-0.5","pushedAt":"2024-06-11T07:44:59.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"FMX","name":"Ethan Feng","path":"/FMX","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4150993?s=80&v=4"},"commit":{"message":"[CELEBORN-1403] Update celeborn configuration about offheap memory\n\n### What changes were proposed in this pull request?\nUpdate configuration guide.\n\n### Why are the changes needed?\nRemove outdated informations about worker's offheap memory.\n\n### Does this PR introduce _any_ user-facing change?\nNO.\n\n### How was this patch tested?\nNo need to test.\n\nCloses #2552 from FMX/b1403.\n\nAuthored-by: mingji \nSigned-off-by: mingji \n(cherry picked from commit 22ecbbd157c972141080f67f0bec12527c4537a0)\nSigned-off-by: mingji ","shortMessageHtmlLink":"[CELEBORN-1403] Update celeborn configuration about offheap memory"}},{"before":"3e875d0177b47d5e6ffb0720006ade99bf764f09","after":"22ecbbd157c972141080f67f0bec12527c4537a0","ref":"refs/heads/main","pushedAt":"2024-06-11T07:44:38.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"FMX","name":"Ethan Feng","path":"/FMX","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4150993?s=80&v=4"},"commit":{"message":"[CELEBORN-1403] Update celeborn configuration about offheap memory\n\n### What changes were proposed in this pull request?\nUpdate configuration guide.\n\n### Why are the changes needed?\nRemove outdated informations about worker's offheap memory.\n\n### Does this PR introduce _any_ user-facing change?\nNO.\n\n### How was this patch tested?\nNo need to test.\n\nCloses #2552 from FMX/b1403.\n\nAuthored-by: mingji \nSigned-off-by: mingji ","shortMessageHtmlLink":"[CELEBORN-1403] Update celeborn configuration about offheap memory"}},{"before":"28089ebb448a0ee39d127cd2a8fc2dadfcb03200","after":"20c730f0f6ff5287f7e3ce83a0a2c1884ca05249","ref":"refs/heads/branch-0.5","pushedAt":"2024-06-11T07:38:09.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"FMX","name":"Ethan Feng","path":"/FMX","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4150993?s=80&v=4"},"commit":{"message":"[CELEBORN-1455] Remove improper configs from config template\n\n### What changes were proposed in this pull request?\nTo remove improper default configs.\n\n### Why are the changes needed?\nMany users won't change default configs in the template causing bad performance in test scenarios.\n\n### Does this PR introduce _any_ user-facing change?\nNO.\n\n### How was this patch tested?\nGA.\n\nCloses #2550 from FMX/b1455.\n\nAuthored-by: mingji \nSigned-off-by: mingji \n(cherry picked from commit 3e875d0177b47d5e6ffb0720006ade99bf764f09)\nSigned-off-by: mingji ","shortMessageHtmlLink":"[CELEBORN-1455] Remove improper configs from config template"}},{"before":"4f039d5f71be743bd86439a61af0423f006bc190","after":"3e875d0177b47d5e6ffb0720006ade99bf764f09","ref":"refs/heads/main","pushedAt":"2024-06-11T07:36:14.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"FMX","name":"Ethan Feng","path":"/FMX","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4150993?s=80&v=4"},"commit":{"message":"[CELEBORN-1455] Remove improper configs from config template\n\n### What changes were proposed in this pull request?\nTo remove improper default configs.\n\n### Why are the changes needed?\nMany users won't change default configs in the template causing bad performance in test scenarios.\n\n### Does this PR introduce _any_ user-facing change?\nNO.\n\n### How was this patch tested?\nGA.\n\nCloses #2550 from FMX/b1455.\n\nAuthored-by: mingji \nSigned-off-by: mingji ","shortMessageHtmlLink":"[CELEBORN-1455] Remove improper configs from config template"}},{"before":"dfca58b428fa852d7874e915ffe15818bc209201","after":"28089ebb448a0ee39d127cd2a8fc2dadfcb03200","ref":"refs/heads/branch-0.5","pushedAt":"2024-06-08T03:31:06.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"waitinfuture","name":"Keyong Zhou","path":"/waitinfuture","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/948245?s=80&v=4"},"commit":{"message":"[CELEBORN-1453] Fix the thread safety bug in getMetrics\n\n### What changes were proposed in this pull request?\nFix the thread safety bug in getMetrics of AbstractSource by changing the lock scope\n\n### Why are the changes needed?\nWhen two threads access the getMetrics method in AbstractSource at the same time, one of the threads may get fewer metrics than the actual value, because the actual execution order may be like this: Thread A gets the lock, adds the metrics of the worker source to the innerMetrics queue and releases the lock, Thread B gets the lock, adds the metrics of the worker source to the innerMetrics queue and releases the lock, Thread A gets the lock, adds the metrics of other sources to the innerMetrics queue, assembles the values of innerMetrics, clears innerMetrics and releases the lock, Thread B gets the lock, adds the metrics of other sources to the innerMetrics queue, assembles the values of innerMetrics, clears innerMetrics and releases the lock. The result of this is that Thread A gets two sets of metrics data from the worker source, while Thread B doesn't get any.\n\n### Does this PR introduce _any_ user-facing change?\nNo\n\n### How was this patch tested?\nmanual test\n\nCloses #2548 from littlexyw/get_metrics_fix.\n\nAuthored-by: xinyuwang1 \nSigned-off-by: zky.zhoukeyong \n(cherry picked from commit 4f039d5f71be743bd86439a61af0423f006bc190)\nSigned-off-by: zky.zhoukeyong ","shortMessageHtmlLink":"[CELEBORN-1453] Fix the thread safety bug in getMetrics"}},{"before":"999510b2652befe1600c81e0c13424dcb29124c0","after":"4f039d5f71be743bd86439a61af0423f006bc190","ref":"refs/heads/main","pushedAt":"2024-06-08T03:30:32.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"waitinfuture","name":"Keyong Zhou","path":"/waitinfuture","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/948245?s=80&v=4"},"commit":{"message":"[CELEBORN-1453] Fix the thread safety bug in getMetrics\n\n### What changes were proposed in this pull request?\nFix the thread safety bug in getMetrics of AbstractSource by changing the lock scope\n\n### Why are the changes needed?\nWhen two threads access the getMetrics method in AbstractSource at the same time, one of the threads may get fewer metrics than the actual value, because the actual execution order may be like this: Thread A gets the lock, adds the metrics of the worker source to the innerMetrics queue and releases the lock, Thread B gets the lock, adds the metrics of the worker source to the innerMetrics queue and releases the lock, Thread A gets the lock, adds the metrics of other sources to the innerMetrics queue, assembles the values of innerMetrics, clears innerMetrics and releases the lock, Thread B gets the lock, adds the metrics of other sources to the innerMetrics queue, assembles the values of innerMetrics, clears innerMetrics and releases the lock. The result of this is that Thread A gets two sets of metrics data from the worker source, while Thread B doesn't get any.\n\n### Does this PR introduce _any_ user-facing change?\nNo\n\n### How was this patch tested?\nmanual test\n\nCloses #2548 from littlexyw/get_metrics_fix.\n\nAuthored-by: xinyuwang1 \nSigned-off-by: zky.zhoukeyong ","shortMessageHtmlLink":"[CELEBORN-1453] Fix the thread safety bug in getMetrics"}},{"before":"7188e845f764b546d5e2b30b0f47cdcf715c70fe","after":"999510b2652befe1600c81e0c13424dcb29124c0","ref":"refs/heads/main","pushedAt":"2024-06-08T03:12:44.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"waitinfuture","name":"Keyong Zhou","path":"/waitinfuture","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/948245?s=80&v=4"},"commit":{"message":"[CELEBORN-1444] Introduce worker decommission metrics and corresponding REST API\n\n### What changes were proposed in this pull request?\n\nIntroduce worker decommission metrics and corresponding REST API.\n\n### Why are the changes needed?\n\nIn a production environment, due to certain hardware or environmental reasons, our script will automatically decommission the node. At this time, we need to distinguish between graceful shutdown nodes and decommissioned nodes.\n\nIf we distinguish shutdown worker and decommission worker metrics, we can achieve better operation and maintenance.\n\n### Does this PR introduce _any_ user-facing change?\n\nYes.\n\n### How was this patch tested?\n\n- `DefaultMetaSystemSuiteJ#testHandleReportWorkerDecommission`\n- `RatisMasterStatusSystemSuiteJ#testHandleReportWorkerDecommission`\n- `ApiMasterResourceSuite#decommissionWorkers`\n- `ApiWorkerResourceSuite#isDecommissioning`\n\nCloses #2535 from leixm/issue_1444.\n\nLead-authored-by: Xianming Lei \nCo-authored-by: Xianming Lei <31424839+leixm@users.noreply.github.com>\nSigned-off-by: zky.zhoukeyong ","shortMessageHtmlLink":"[CELEBORN-1444] Introduce worker decommission metrics and correspondi…"}},{"before":"a5d3f0f30eb97767d98ae617f397ba9cda93718e","after":"7188e845f764b546d5e2b30b0f47cdcf715c70fe","ref":"refs/heads/main","pushedAt":"2024-06-07T08:23:47.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"FMX","name":"Ethan Feng","path":"/FMX","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4150993?s=80&v=4"},"commit":{"message":"[CELEBORN-1327][FOLLOWUP] Simplify DirectByteBuffer constructor lookup logic\n\n### What changes were proposed in this pull request?\n\nSimplify `DirectByteBuffer` constructor lookup logic in `Platform`. Meanwhile, bump `commons-lang3` version from `3.12.0` to `3.13.0`.\n\n### Why are the changes needed?\n\n`try-catch` statement is not needed because we know version number already.\n\nBackport:\n\n- https://github.com/apache/spark/pull/41780\n- https://github.com/apache/spark/pull/42269\n- https://github.com/apache/spark/pull/44444\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nGA.\n\nCloses #2544 from SteNicholas/CELEBORN-1327.\n\nAuthored-by: SteNicholas \nSigned-off-by: mingji ","shortMessageHtmlLink":"[CELEBORN-1327][FOLLOWUP] Simplify DirectByteBuffer constructor looku…"}},{"before":"74d982b20f959339b1fb34e9399b97147031462a","after":"dfca58b428fa852d7874e915ffe15818bc209201","ref":"refs/heads/branch-0.5","pushedAt":"2024-06-07T03:05:46.000Z","pushType":"push","commitsCount":2,"pusher":{"login":"FMX","name":"Ethan Feng","path":"/FMX","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4150993?s=80&v=4"},"commit":{"message":"Bump 0.5.1-SNAPSHOT","shortMessageHtmlLink":"Bump 0.5.1-SNAPSHOT"}},{"before":"e06505c70c0c005120932df61fb47087b7a662f9","after":null,"ref":"refs/tags/v0.5.0-rc0","pushedAt":"2024-06-06T09:29:47.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"FMX","name":"Ethan Feng","path":"/FMX","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4150993?s=80&v=4"}},{"before":"fd9e77abc06768817c648fb8c2598c1382f4465c","after":"74d982b20f959339b1fb34e9399b97147031462a","ref":"refs/heads/branch-0.5","pushedAt":"2024-06-06T09:26:28.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"FMX","name":"Ethan Feng","path":"/FMX","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4150993?s=80&v=4"},"commit":{"message":"[CELEBORN-1182][FOLLOWUP] WorkerSource should use Counter to support application dimension ActiveConnectionCount metric\n\n### What changes were proposed in this pull request?\n\n`WorkerSource` should use Counter to support application dimension `ActiveConnectionCount` metric.\n\nFollow up #2167.\n\n### Why are the changes needed?\n\n`WorkerSource` uses `Gauge` for application dimension ActiveConnectionCount metric via `appActiveConnections` at present, which has performance problem in metrics REST API as follows:\n\n```\n\"worker-JettyThreadPool-11242\" #11242 daemon prio=5 os_prio=0 tid=0x00007f410800c000 nid=0x2d80 runnable [0x00007f3426de2000]\n java.lang.Thread.State: RUNNABLE\n\tat scala.collection.Iterator.foreach(Iterator.scala:941)\n\tat scala.collection.Iterator.foreach$(Iterator.scala:941)\n\tat scala.collection.AbstractIterator.foreach(Iterator.scala:1429)\n\tat scala.collection.IterableLike.foreach(IterableLike.scala:74)\n\tat scala.collection.IterableLike.foreach$(IterableLike.scala:73)\n\tat scala.collection.AbstractIterable.foreach(Iterable.scala:56)\n\tat scala.collection.TraversableOnce.count(TraversableOnce.scala:118)\n\tat scala.collection.TraversableOnce.count$(TraversableOnce.scala:116)\n\tat scala.collection.AbstractTraversable.count(Traversable.scala:108)\n\tat org.apache.celeborn.service.deploy.worker.WorkerSource.$anonfun$recordAppActiveConnection$1(WorkerSource.scala:104)\n\tat org.apache.celeborn.service.deploy.worker.WorkerSource$$Lambda$787/1074905995.apply$mcI$sp(Unknown Source)\n\tat scala.runtime.java8.JFunction0$mcI$sp.apply(JFunction0$mcI$sp.java:23)\n\tat org.apache.celeborn.common.metrics.source.GaugeSupplier$$anon$3.getValue(AbstractSource.scala:466)\n\tat org.apache.celeborn.common.metrics.source.AbstractSource.recordGauge(AbstractSource.scala:342)\n\tat org.apache.celeborn.common.metrics.source.AbstractSource.$anonfun$getMetrics$2(AbstractSource.scala:401)\n\tat org.apache.celeborn.common.metrics.source.AbstractSource.$anonfun$getMetrics$2$adapted(AbstractSource.scala:401)\n\tat org.apache.celeborn.common.metrics.source.AbstractSource$$Lambda$956/1021547679.apply(Unknown Source)\n\tat scala.collection.immutable.List.foreach(List.scala:392)\n\tat org.apache.celeborn.common.metrics.source.AbstractSource.getMetrics(AbstractSource.scala:401)\n\tat org.apache.celeborn.common.metrics.sink.AbstractServlet.$anonfun$getMetricsSnapshot$1(AbstractServlet.scala:34)\n\tat org.apache.celeborn.common.metrics.sink.AbstractServlet$$Lambda$954/1559941228.apply(Unknown Source)\n\tat scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)\n\tat scala.collection.TraversableLike$$Lambda$33/829149076.apply(Unknown Source)\n\tat scala.collection.Iterator.foreach(Iterator.scala:941)\n\tat scala.collection.Iterator.foreach$(Iterator.scala:941)\n\tat scala.collection.AbstractIterator.foreach(Iterator.scala:1429)\n\tat scala.collection.IterableLike.foreach(IterableLike.scala:74)\n\tat scala.collection.IterableLike.foreach$(IterableLike.scala:73)\n\tat scala.collection.AbstractIterable.foreach(Iterable.scala:56)\n\tat scala.collection.TraversableLike.map(TraversableLike.scala:238)\n\tat scala.collection.TraversableLike.map$(TraversableLike.scala:231)\n\tat scala.collection.AbstractTraversable.map(Traversable.scala:108)\n\tat org.apache.celeborn.common.metrics.sink.AbstractServlet.getMetricsSnapshot(AbstractServlet.scala:34)\n\tat org.apache.celeborn.common.metrics.sink.PrometheusServlet.$anonfun$createServletHandler$1(PrometheusServlet.scala:38)\n\tat org.apache.celeborn.common.metrics.sink.PrometheusServlet$$Lambda$721/2120532393.apply(Unknown Source)\n\tat org.apache.celeborn.server.common.http.HttpUtils$$anon$1.doGet(HttpUtils.scala:51)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:497)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:584)\n\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:516)\n\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)\n\tat org.eclipse.jetty.server.HttpChannel$$Lambda$636/1962809899.dispatch(Unknown Source)\n\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)\n\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)\n\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)\n\tat java.lang.Thread.run(Thread.java:748)\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nCluster test.\n\n```\n$ curl http://bigdata-rss-worker:9096/metrics|grep ActiveConnectionCount|grep application\n % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n100 54003 0 54003 0 0 2924k 0 --:--:-- --:--:-- --:--:-- 3102k\nmetrics_ActiveConnectionCount_Count{applicationId=\"application_1688369676084_17462520_1\",hostName=\"bigdata-rss-worker\",role=\"Worker\"} 15 1717590356773\nmetrics_ActiveConnectionCount_Count{applicationId=\"application_1650016801129_32165809_1\",hostName=\"bigdata-rss-worker\",role=\"Worker\"} 7 171759035677\n$ curl http://bigdata-rss-worker:9096/metrics|grep ActiveConnectionCount|grep application\n % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n100 54025 0 54025 0 0 2891k 0 --:--:-- --:--:-- --:--:-- 2931k\nmetrics_ActiveConnectionCount_Count{applicationId=\"application_1688369676084_17462520_1\",hostName=\"bigdata-rss-worker\",role=\"Worker\"} 25 1717590431544\nmetrics_ActiveConnectionCount_Count{applicationId=\"application_1650016801129_32165809_1\",hostName=\"bigdata-rss-worker\",role=\"Worker\"} 14 1717590431544\n$ curl http://bigdata-rss-worker:9096/metrics|grep ActiveConnectionCount|grep application\n % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n100 54014 0 54014 0 0 2727k 0 --:--:-- --:--:-- --:--:-- 2776k\nmetrics_ActiveConnectionCount_Count{applicationId=\"application_1688369676084_17462520_1\",hostName=\"bigdata-rss-worker\",role=\"Worker\"} 19 1717590480837\nmetrics_ActiveConnectionCount_Count{applicationId=\"application_1650016801129_32165809_1\",hostName=\"bigdata-rss-worker\",role=\"Worker\"} 9 1717590480837\n```\n\nCloses #2546 from SteNicholas/CELEBORN-1182.\n\nAuthored-by: SteNicholas \nSigned-off-by: mingji \n(cherry picked from commit a5d3f0f30eb97767d98ae617f397ba9cda93718e)\nSigned-off-by: mingji ","shortMessageHtmlLink":"[CELEBORN-1182][FOLLOWUP] WorkerSource should use Counter to support …"}},{"before":"9e5f7e5a4da70d77a6b5f7a5745718a678b56d8f","after":"a5d3f0f30eb97767d98ae617f397ba9cda93718e","ref":"refs/heads/main","pushedAt":"2024-06-06T09:25:55.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"FMX","name":"Ethan Feng","path":"/FMX","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4150993?s=80&v=4"},"commit":{"message":"[CELEBORN-1182][FOLLOWUP] WorkerSource should use Counter to support application dimension ActiveConnectionCount metric\n\n### What changes were proposed in this pull request?\n\n`WorkerSource` should use Counter to support application dimension `ActiveConnectionCount` metric.\n\nFollow up #2167.\n\n### Why are the changes needed?\n\n`WorkerSource` uses `Gauge` for application dimension ActiveConnectionCount metric via `appActiveConnections` at present, which has performance problem in metrics REST API as follows:\n\n```\n\"worker-JettyThreadPool-11242\" #11242 daemon prio=5 os_prio=0 tid=0x00007f410800c000 nid=0x2d80 runnable [0x00007f3426de2000]\n java.lang.Thread.State: RUNNABLE\n\tat scala.collection.Iterator.foreach(Iterator.scala:941)\n\tat scala.collection.Iterator.foreach$(Iterator.scala:941)\n\tat scala.collection.AbstractIterator.foreach(Iterator.scala:1429)\n\tat scala.collection.IterableLike.foreach(IterableLike.scala:74)\n\tat scala.collection.IterableLike.foreach$(IterableLike.scala:73)\n\tat scala.collection.AbstractIterable.foreach(Iterable.scala:56)\n\tat scala.collection.TraversableOnce.count(TraversableOnce.scala:118)\n\tat scala.collection.TraversableOnce.count$(TraversableOnce.scala:116)\n\tat scala.collection.AbstractTraversable.count(Traversable.scala:108)\n\tat org.apache.celeborn.service.deploy.worker.WorkerSource.$anonfun$recordAppActiveConnection$1(WorkerSource.scala:104)\n\tat org.apache.celeborn.service.deploy.worker.WorkerSource$$Lambda$787/1074905995.apply$mcI$sp(Unknown Source)\n\tat scala.runtime.java8.JFunction0$mcI$sp.apply(JFunction0$mcI$sp.java:23)\n\tat org.apache.celeborn.common.metrics.source.GaugeSupplier$$anon$3.getValue(AbstractSource.scala:466)\n\tat org.apache.celeborn.common.metrics.source.AbstractSource.recordGauge(AbstractSource.scala:342)\n\tat org.apache.celeborn.common.metrics.source.AbstractSource.$anonfun$getMetrics$2(AbstractSource.scala:401)\n\tat org.apache.celeborn.common.metrics.source.AbstractSource.$anonfun$getMetrics$2$adapted(AbstractSource.scala:401)\n\tat org.apache.celeborn.common.metrics.source.AbstractSource$$Lambda$956/1021547679.apply(Unknown Source)\n\tat scala.collection.immutable.List.foreach(List.scala:392)\n\tat org.apache.celeborn.common.metrics.source.AbstractSource.getMetrics(AbstractSource.scala:401)\n\tat org.apache.celeborn.common.metrics.sink.AbstractServlet.$anonfun$getMetricsSnapshot$1(AbstractServlet.scala:34)\n\tat org.apache.celeborn.common.metrics.sink.AbstractServlet$$Lambda$954/1559941228.apply(Unknown Source)\n\tat scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)\n\tat scala.collection.TraversableLike$$Lambda$33/829149076.apply(Unknown Source)\n\tat scala.collection.Iterator.foreach(Iterator.scala:941)\n\tat scala.collection.Iterator.foreach$(Iterator.scala:941)\n\tat scala.collection.AbstractIterator.foreach(Iterator.scala:1429)\n\tat scala.collection.IterableLike.foreach(IterableLike.scala:74)\n\tat scala.collection.IterableLike.foreach$(IterableLike.scala:73)\n\tat scala.collection.AbstractIterable.foreach(Iterable.scala:56)\n\tat scala.collection.TraversableLike.map(TraversableLike.scala:238)\n\tat scala.collection.TraversableLike.map$(TraversableLike.scala:231)\n\tat scala.collection.AbstractTraversable.map(Traversable.scala:108)\n\tat org.apache.celeborn.common.metrics.sink.AbstractServlet.getMetricsSnapshot(AbstractServlet.scala:34)\n\tat org.apache.celeborn.common.metrics.sink.PrometheusServlet.$anonfun$createServletHandler$1(PrometheusServlet.scala:38)\n\tat org.apache.celeborn.common.metrics.sink.PrometheusServlet$$Lambda$721/2120532393.apply(Unknown Source)\n\tat org.apache.celeborn.server.common.http.HttpUtils$$anon$1.doGet(HttpUtils.scala:51)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:497)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:584)\n\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:516)\n\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)\n\tat org.eclipse.jetty.server.HttpChannel$$Lambda$636/1962809899.dispatch(Unknown Source)\n\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)\n\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)\n\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)\n\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)\n\tat java.lang.Thread.run(Thread.java:748)\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nCluster test.\n\n```\n$ curl http://bigdata-rss-worker:9096/metrics|grep ActiveConnectionCount|grep application\n % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n100 54003 0 54003 0 0 2924k 0 --:--:-- --:--:-- --:--:-- 3102k\nmetrics_ActiveConnectionCount_Count{applicationId=\"application_1688369676084_17462520_1\",hostName=\"bigdata-rss-worker\",role=\"Worker\"} 15 1717590356773\nmetrics_ActiveConnectionCount_Count{applicationId=\"application_1650016801129_32165809_1\",hostName=\"bigdata-rss-worker\",role=\"Worker\"} 7 171759035677\n$ curl http://bigdata-rss-worker:9096/metrics|grep ActiveConnectionCount|grep application\n % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n100 54025 0 54025 0 0 2891k 0 --:--:-- --:--:-- --:--:-- 2931k\nmetrics_ActiveConnectionCount_Count{applicationId=\"application_1688369676084_17462520_1\",hostName=\"bigdata-rss-worker\",role=\"Worker\"} 25 1717590431544\nmetrics_ActiveConnectionCount_Count{applicationId=\"application_1650016801129_32165809_1\",hostName=\"bigdata-rss-worker\",role=\"Worker\"} 14 1717590431544\n$ curl http://bigdata-rss-worker:9096/metrics|grep ActiveConnectionCount|grep application\n % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n100 54014 0 54014 0 0 2727k 0 --:--:-- --:--:-- --:--:-- 2776k\nmetrics_ActiveConnectionCount_Count{applicationId=\"application_1688369676084_17462520_1\",hostName=\"bigdata-rss-worker\",role=\"Worker\"} 19 1717590480837\nmetrics_ActiveConnectionCount_Count{applicationId=\"application_1650016801129_32165809_1\",hostName=\"bigdata-rss-worker\",role=\"Worker\"} 9 1717590480837\n```\n\nCloses #2546 from SteNicholas/CELEBORN-1182.\n\nAuthored-by: SteNicholas \nSigned-off-by: mingji ","shortMessageHtmlLink":"[CELEBORN-1182][FOLLOWUP] WorkerSource should use Counter to support …"}},{"before":"0d761da3c9af8b72ff15006259fecb1800b41507","after":"fd9e77abc06768817c648fb8c2598c1382f4465c","ref":"refs/heads/branch-0.5","pushedAt":"2024-06-05T06:04:59.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"cxzl25","name":null,"path":"/cxzl25","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3898450?s=80&v=4"},"commit":{"message":"[CELEBORN-1449] Fix JavaUtils#deleteRecursivelyUsingJavaIO to skip non-existing file input\n\n### What changes were proposed in this pull request?\n\nFix `JavaUtils#deleteRecursivelyUsingJavaIO` to skip non-existing file input. Meanwhile, reduce multiple file attribute calls of `JavaUtils#deleteRecursivelyUsingJavaIO`.\n\n### Why are the changes needed?\n\n`deleteRecursivelyUsingJavaIO` is a fallback of `deleteRecursivelyUsingUnixNative` in `JavaUtils`. We should have identical capability for `JavaUtils#deleteRecursivelyUsingJavaIO` which should skip non-existing file input. Meanwhile, `JavaUtils#deleteRecursivelyUsingJavaIO` method performs multiple file attribute calls.\n\nBackport:\n\n- https://github.com/apache/spark/pull/36636\n- https://github.com/apache/spark/pull/45346\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nGA.\n\nCloses #2543 from SteNicholas/CELEBORN-1449.\n\nAuthored-by: SteNicholas \nSigned-off-by: Shaoyun Chen \n(cherry picked from commit 9e5f7e5a4da70d77a6b5f7a5745718a678b56d8f)\nSigned-off-by: Shaoyun Chen ","shortMessageHtmlLink":"[CELEBORN-1449] Fix JavaUtils#deleteRecursivelyUsingJavaIO to skip no…"}},{"before":"50eaaf723f3cda528b9c0252a8d5cc193ce71397","after":"9e5f7e5a4da70d77a6b5f7a5745718a678b56d8f","ref":"refs/heads/main","pushedAt":"2024-06-05T06:04:44.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"cxzl25","name":null,"path":"/cxzl25","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3898450?s=80&v=4"},"commit":{"message":"[CELEBORN-1449] Fix JavaUtils#deleteRecursivelyUsingJavaIO to skip non-existing file input\n\n### What changes were proposed in this pull request?\n\nFix `JavaUtils#deleteRecursivelyUsingJavaIO` to skip non-existing file input. Meanwhile, reduce multiple file attribute calls of `JavaUtils#deleteRecursivelyUsingJavaIO`.\n\n### Why are the changes needed?\n\n`deleteRecursivelyUsingJavaIO` is a fallback of `deleteRecursivelyUsingUnixNative` in `JavaUtils`. We should have identical capability for `JavaUtils#deleteRecursivelyUsingJavaIO` which should skip non-existing file input. Meanwhile, `JavaUtils#deleteRecursivelyUsingJavaIO` method performs multiple file attribute calls.\n\nBackport:\n\n- https://github.com/apache/spark/pull/36636\n- https://github.com/apache/spark/pull/45346\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nGA.\n\nCloses #2543 from SteNicholas/CELEBORN-1449.\n\nAuthored-by: SteNicholas \nSigned-off-by: Shaoyun Chen ","shortMessageHtmlLink":"[CELEBORN-1449] Fix JavaUtils#deleteRecursivelyUsingJavaIO to skip no…"}},{"before":"e06505c70c0c005120932df61fb47087b7a662f9","after":null,"ref":"refs/tags/v0.5.0","pushedAt":"2024-06-05T02:26:18.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"FMX","name":"Ethan Feng","path":"/FMX","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4150993?s=80&v=4"}},{"before":"3edfa491d697c588f7553382d9af12f50a83c2ca","after":"0d761da3c9af8b72ff15006259fecb1800b41507","ref":"refs/heads/branch-0.5","pushedAt":"2024-06-05T02:05:39.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"FMX","name":"Ethan Feng","path":"/FMX","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4150993?s=80&v=4"},"commit":{"message":"[CELEBORN-1450] MRAppMasterWithCeleborn should get FileSystem via mapreduce.job.dir for HDFS federation\n\n### What changes were proposed in this pull request?\n\n`MRAppMasterWithCeleborn` gets `FileSystem` via `mapreduce.job.dir` for HDFS federation.\n\n### Why are the changes needed?\n\n`MRAppMasterWithCeleborn` should get `FileSystem` via `mapreduce.job.dir` for HDFS federation. Otherwise, the Celeborn conf path of `celeborn.xml` is wrong for the following example configuration:\n\n```\n\n fs.defaultFS\n viewfs://bigdata-cluster\n\n\n fs.viewfs.mounttable.bigdata-cluster.link./application\n hdfs://bigdata-proxy-ns/application\n\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nCluster test.\n\nCloses #2545 from SteNicholas/CELEBORN-1450.\n\nAuthored-by: SteNicholas \nSigned-off-by: mingji \n(cherry picked from commit 50eaaf723f3cda528b9c0252a8d5cc193ce71397)\nSigned-off-by: mingji ","shortMessageHtmlLink":"[CELEBORN-1450] MRAppMasterWithCeleborn should get FileSystem via map…"}},{"before":"4fc42d7fef9d9e38bad10437cd61665b81ed9a3b","after":"50eaaf723f3cda528b9c0252a8d5cc193ce71397","ref":"refs/heads/main","pushedAt":"2024-06-05T02:05:10.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"FMX","name":"Ethan Feng","path":"/FMX","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4150993?s=80&v=4"},"commit":{"message":"[CELEBORN-1450] MRAppMasterWithCeleborn should get FileSystem via mapreduce.job.dir for HDFS federation\n\n### What changes were proposed in this pull request?\n\n`MRAppMasterWithCeleborn` gets `FileSystem` via `mapreduce.job.dir` for HDFS federation.\n\n### Why are the changes needed?\n\n`MRAppMasterWithCeleborn` should get `FileSystem` via `mapreduce.job.dir` for HDFS federation. Otherwise, the Celeborn conf path of `celeborn.xml` is wrong for the following example configuration:\n\n```\n\n fs.defaultFS\n viewfs://bigdata-cluster\n\n\n fs.viewfs.mounttable.bigdata-cluster.link./application\n hdfs://bigdata-proxy-ns/application\n\n```\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nCluster test.\n\nCloses #2545 from SteNicholas/CELEBORN-1450.\n\nAuthored-by: SteNicholas \nSigned-off-by: mingji ","shortMessageHtmlLink":"[CELEBORN-1450] MRAppMasterWithCeleborn should get FileSystem via map…"}},{"before":"9a8752300f45f4b48649b6a1804df2e89c4e84a7","after":"3edfa491d697c588f7553382d9af12f50a83c2ca","ref":"refs/heads/branch-0.5","pushedAt":"2024-06-04T11:28:15.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"FMX","name":"Ethan Feng","path":"/FMX","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4150993?s=80&v=4"},"commit":{"message":"[CELEBORN-1389] Bump Dropwizard version from 3.2.6 to 4.2.25\n\n### What changes were proposed in this pull request?\n\nBump Dropwizard version from 3.2.6 to 4.2.25. Meanwhile, introduce `metrics_jvm_thread_peak_count_Value` and `metrics_jvm_thread_total_started_count_Value` in `celeborn-jvm-dashboard.json`.\n\n### Why are the changes needed?\n\nDropwizard metrics has released v4.2.25 including some bugfixes and improvements including:\n\n* [JVM] Fix maximum/total memory calculation: https://github.com/dropwizard/metrics/pull/3125\n* [Thread] Add peak and total started thread count to `ThreadStatesGaugeSet`: https://github.com/dropwizard/metrics/pull/1601\n\nMeanwhile, Ratis version has upgraded to 3.0.1 which has no compatibility problem with Dropwizard 4.2.25.\n\nBackport:\n\n- https://github.com/apache/spark/pull/26332\n- https://github.com/apache/spark/pull/29426\n- https://github.com/apache/spark/pull/37372\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nManual test.\n\nCloses #2540 from SteNicholas/CELEBORN-1389.\n\nAuthored-by: SteNicholas \nSigned-off-by: mingji \n(cherry picked from commit 4fc42d7fef9d9e38bad10437cd61665b81ed9a3b)\nSigned-off-by: mingji ","shortMessageHtmlLink":"[CELEBORN-1389] Bump Dropwizard version from 3.2.6 to 4.2.25"}},{"before":"aa8ff3c17ca99cfca5603a93d7190cfc3d28cfa4","after":"4fc42d7fef9d9e38bad10437cd61665b81ed9a3b","ref":"refs/heads/main","pushedAt":"2024-06-04T11:27:03.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"FMX","name":"Ethan Feng","path":"/FMX","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4150993?s=80&v=4"},"commit":{"message":"[CELEBORN-1389] Bump Dropwizard version from 3.2.6 to 4.2.25\n\n### What changes were proposed in this pull request?\n\nBump Dropwizard version from 3.2.6 to 4.2.25. Meanwhile, introduce `metrics_jvm_thread_peak_count_Value` and `metrics_jvm_thread_total_started_count_Value` in `celeborn-jvm-dashboard.json`.\n\n### Why are the changes needed?\n\nDropwizard metrics has released v4.2.25 including some bugfixes and improvements including:\n\n* [JVM] Fix maximum/total memory calculation: https://github.com/dropwizard/metrics/pull/3125\n* [Thread] Add peak and total started thread count to `ThreadStatesGaugeSet`: https://github.com/dropwizard/metrics/pull/1601\n\nMeanwhile, Ratis version has upgraded to 3.0.1 which has no compatibility problem with Dropwizard 4.2.25.\n\nBackport:\n\n- https://github.com/apache/spark/pull/26332\n- https://github.com/apache/spark/pull/29426\n- https://github.com/apache/spark/pull/37372\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nManual test.\n\nCloses #2540 from SteNicholas/CELEBORN-1389.\n\nAuthored-by: SteNicholas \nSigned-off-by: mingji ","shortMessageHtmlLink":"[CELEBORN-1389] Bump Dropwizard version from 3.2.6 to 4.2.25"}},{"before":"db87c23e2bd7919fc1210e75483ef4b3adf8dd80","after":"9a8752300f45f4b48649b6a1804df2e89c4e84a7","ref":"refs/heads/branch-0.5","pushedAt":"2024-06-04T06:26:10.000Z","pushType":"push","commitsCount":2,"pusher":{"login":"FMX","name":"Ethan Feng","path":"/FMX","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4150993?s=80&v=4"},"commit":{"message":"Bump 0.5.1-SNAPSHOT","shortMessageHtmlLink":"Bump 0.5.1-SNAPSHOT"}},{"before":"fd325140d6aa1aa150d3cddb88a824537857012f","after":null,"ref":"refs/tags/v0.5.0-rc0","pushedAt":"2024-06-04T06:22:15.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"FMX","name":"Ethan Feng","path":"/FMX","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4150993?s=80&v=4"}},{"before":"1a9205be71e44888fafd03d64d82832623665eee","after":"94801b6f3963bf4ca7a90d4e4f2d2c21e8558d34","ref":"refs/heads/branch-0.4","pushedAt":"2024-06-04T02:51:02.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"FMX","name":"Ethan Feng","path":"/FMX","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4150993?s=80&v=4"},"commit":{"message":"[CELEBORN-1182][FOLLOWUP] Fix WorkerSource record application active connection for application dimension ActiveConnectionCount metric\n\n### What changes were proposed in this pull request?\n\nFix `WorkerSource` record application active connection for application dimension `ActiveConnectionCount` metric.\n\nFollow up #2167.\n\n### Why are the changes needed?\n\nApplication dimension `ActiveConnectionCount` metric does not have value because the check of recording application active connection is wrong for `WorkerSource`.\n\n### Does this PR introduce _any_ user-facing change?\n\nNo.\n\n### How was this patch tested?\n\nGA.\n\n```\ncelebornceleborn-worker-1:/data/service/celeborn$ curl http://celeborn-worker-1:9096/metrics|grep application|grep ActiveConnectionCount\n % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n100 53673 0 53673 0 0 2710k 0 --:--:-- --:--:-- --:--:-- 2758k\nmetrics_ActiveConnectionCount_Value{applicationId=\"application_1692685933461_14488489\",hostName=\"celeborn-worker-1\",role=\"Worker\"} 68 1717418613619\nmetrics_ActiveConnectionCount_Value{applicationId=\"application_1692685933461_14488489\",hostName=\"celeborn-worker-1\",role=\"Worker\"} 68 1717418613619\n```\n\nCloses #2542 from SteNicholas/CELEBORN-1182.\n\nAuthored-by: SteNicholas \nSigned-off-by: mingji \n(cherry picked from commit aa8ff3c17ca99cfca5603a93d7190cfc3d28cfa4)\nSigned-off-by: mingji ","shortMessageHtmlLink":"[CELEBORN-1182][FOLLOWUP] Fix WorkerSource record application active …"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAEYwt5UAA","startCursor":null,"endCursor":null}},"title":"Activity · apache/celeborn"}