Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

YARN-11469. [BackPort] Refactor FederationStateStoreFacade Cache Code. (#5570) #6796

Open
wants to merge 3 commits into
base: branch-3.3
Choose a base branch
from
Open

YARN-11469. [BackPort] Refactor FederationStateStoreFacade Cache Code. (#5570) #6796

wants to merge 3 commits into from
+926 −224

Conversation

slfan1989
Copy link
Contributor

@slfan1989 slfan1989 commented May 5, 2024 •
edited

Description of PR

We expect to upgrade ehcache from 3.3.1 to 3.8.2 in branch-3.3, but before upgrading this jar package version, we need to backport 2 pr.

How was this patch tested?

Add Junit Test.

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@slfan1989 slfan1989 marked this pull request as draft May 5, 2024 08:12
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 6m 46s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ branch-3.3 Compile Tests _
+1 💚 mvninstall 46m 23s branch-3.3 passed
+1 💚 compile 0m 39s branch-3.3 passed
+1 💚 checkstyle 0m 31s branch-3.3 passed
+1 💚 mvnsite 0m 44s branch-3.3 passed
+1 💚 javadoc 0m 39s branch-3.3 passed
+1 💚 spotbugs 1m 26s branch-3.3 passed
+1 💚 shadedclient 38m 12s branch has no errors when building and testing our client artifacts.
-0 ⚠️ patch 38m 33s Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 35s the patch passed
+1 💚 compile 0m 31s the patch passed
+1 💚 javac 0m 31s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 17s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 0 new + 0 unchanged - 11 fixed = 0 total (was 11)
+1 💚 mvnsite 0m 33s the patch passed
+1 💚 javadoc 0m 26s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common generated 0 new + 44 unchanged - 1 fixed = 44 total (was 45)
+1 💚 spotbugs 1m 21s the patch passed
+1 💚 shadedclient 36m 9s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 2m 55s /patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt hadoop-yarn-server-common in the patch passed.
+1 💚 asflicense 0m 35s The patch does not generate ASF License warnings.
142m 21s
Reason Tests
Failed junit tests hadoop.yarn.server.federation.policies.amrmproxy.TestLocalityMulticastAMRMProxyPolicy
Subsystem Report/Notes
Docker ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6796/1/artifact/out/Dockerfile
GITHUB PR #6796
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 7c05cf879860 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / edccf15
Default Java Private Build-1.8.0_362-8u372-gaus1-0ubuntu118.04-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6796/1/testReport/
Max. process+thread count 557 (vs. ulimit of 5500)
modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6796/1/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 29s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ branch-3.3 Compile Tests _
+1 💚 mvninstall 47m 49s branch-3.3 passed
+1 💚 compile 0m 37s branch-3.3 passed
+1 💚 checkstyle 0m 29s branch-3.3 passed
+1 💚 mvnsite 0m 42s branch-3.3 passed
+1 💚 javadoc 0m 38s branch-3.3 passed
+1 💚 spotbugs 1m 22s branch-3.3 passed
+1 💚 shadedclient 36m 32s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 35s the patch passed
+1 💚 compile 0m 31s the patch passed
+1 💚 javac 0m 31s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 18s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 0 new + 0 unchanged - 11 fixed = 0 total (was 11)
+1 💚 mvnsite 0m 34s the patch passed
+1 💚 javadoc 0m 25s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common generated 0 new + 44 unchanged - 1 fixed = 44 total (was 45)
+1 💚 spotbugs 1m 21s the patch passed
+1 💚 shadedclient 35m 20s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 2m 57s /patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt hadoop-yarn-server-common in the patch passed.
+1 💚 asflicense 0m 38s The patch does not generate ASF License warnings.
134m 58s
Reason Tests
Failed junit tests hadoop.yarn.server.federation.policies.amrmproxy.TestLocalityMulticastAMRMProxyPolicy
Subsystem Report/Notes
Docker ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6796/2/artifact/out/Dockerfile
GITHUB PR #6796
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 77569b6f3797 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / f417397
Default Java Private Build-1.8.0_362-8u372-gaus1-0ubuntu118.04-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6796/2/testReport/
Max. process+thread count 747 (vs. ulimit of 5500)
modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6796/2/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@slfan1989 slfan1989 marked this pull request as ready for review May 7, 2024 00:07
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 6m 17s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ branch-3.3 Compile Tests _
+1 💚 mvninstall 46m 52s branch-3.3 passed
+1 💚 compile 0m 39s branch-3.3 passed
+1 💚 checkstyle 0m 31s branch-3.3 passed
+1 💚 mvnsite 0m 45s branch-3.3 passed
+1 💚 javadoc 0m 40s branch-3.3 passed
+1 💚 spotbugs 1m 26s branch-3.3 passed
+1 💚 shadedclient 35m 38s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 36s the patch passed
+1 💚 compile 0m 31s the patch passed
+1 💚 javac 0m 31s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 18s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 0 new + 0 unchanged - 11 fixed = 0 total (was 11)
+1 💚 mvnsite 0m 34s the patch passed
+1 💚 javadoc 0m 27s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common generated 0 new + 44 unchanged - 1 fixed = 44 total (was 45)
+1 💚 spotbugs 1m 22s the patch passed
+1 💚 shadedclient 34m 59s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 3m 6s /patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt hadoop-yarn-server-common in the patch passed.
+1 💚 asflicense 0m 37s The patch does not generate ASF License warnings.
139m 0s
Reason Tests
Failed junit tests hadoop.yarn.server.federation.policies.amrmproxy.TestLocalityMulticastAMRMProxyPolicy
Subsystem Report/Notes
Docker ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6796/3/artifact/out/Dockerfile
GITHUB PR #6796
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 30922528e317 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / 8019977
Default Java Private Build-1.8.0_362-8u372-gaus1-0ubuntu118.04-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6796/3/testReport/
Max. process+thread count 552 (vs. ulimit of 5500)
modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6796/3/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 6m 25s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ branch-3.3 Compile Tests _
+1 💚 mvninstall 53m 43s branch-3.3 passed
+1 💚 compile 0m 39s branch-3.3 passed
+1 💚 checkstyle 0m 29s branch-3.3 passed
+1 💚 mvnsite 0m 44s branch-3.3 passed
+1 💚 javadoc 0m 40s branch-3.3 passed
+1 💚 spotbugs 1m 25s branch-3.3 passed
+1 💚 shadedclient 35m 24s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 35s the patch passed
+1 💚 compile 0m 31s the patch passed
+1 💚 javac 0m 31s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 18s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 0 new + 0 unchanged - 11 fixed = 0 total (was 11)
+1 💚 mvnsite 0m 34s the patch passed
+1 💚 javadoc 0m 26s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common generated 0 new + 44 unchanged - 1 fixed = 44 total (was 45)
+1 💚 spotbugs 1m 22s the patch passed
+1 💚 shadedclient 34m 49s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 3m 12s hadoop-yarn-server-common in the patch passed.
+1 💚 asflicense 0m 37s The patch does not generate ASF License warnings.
145m 34s
Subsystem Report/Notes
Docker ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6796/4/artifact/out/Dockerfile
GITHUB PR #6796
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 945bbd036316 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / e3cfb1a
Default Java Private Build-1.8.0_362-8u372-gaus1-0ubuntu118.04-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6796/4/testReport/
Max. process+thread count 552 (vs. ulimit of 5500)
modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6796/4/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@slfan1989
Copy link
Contributor Author

@steveloughran @goiri We are backporting YARN-11469 to branch-3.3 in order to upgrade ehcache from 3.3.1 to 3.8.2. However, there are significant differences between the code in branch-3.3 and branch-3.4. I had to redevelop YARN-11469 based on the code in branch-3.3, which has resulted in some variances between YARN-11469 in branch-3.3 and branch-3.4, but they are generally consistent.

@steveloughran
Copy link
Contributor

I don't write/review yarn code, or hdfs. better for everyone. fewer regressions and I can stay out of support escalations. sorry

@slfan1989
Copy link
Contributor Author

I don't write/review yarn code, or hdfs. better for everyone. fewer regressions and I can stay out of support escalations. sorry

Thank you for feedback! I understand and respect your principles.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@slfan1989 @hadoop-yetus @steveloughran