Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FLINK-35465][runtime] Introduce BatchJobRecoveryHandler for recovery of batch jobs from JobMaster failures. #24771

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

[FLINK-35465][runtime] Introduce BatchJobRecoveryHandler for recovery of batch jobs from JobMaster failures. #24771

wants to merge 3 commits into from
+3,923 −139

Conversation

JunRuiLee
Copy link
Contributor

What is the purpose of the change

(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)

Brief change log

Support Job Recovery from JobMaster Failures for Batch Jobs.

Verifying this change

This change added tests and can be verified by BatchJobRecoveryTest and JMFailoverITCase.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
  • The serializers: (yes / no / don't know)
  • The runtime per-record code paths (performance sensitive): (yes / no / don't know)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
  • The S3 file system connector: (yes / no / don't know)

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@JunRuiLee JunRuiLee marked this pull request as draft May 11, 2024 07:28
@flinkbot
Copy link
Collaborator

flinkbot commented May 11, 2024 •
edited

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@JunRuiLee JunRuiLee force-pushed the jm_fo_main branch 2 times, most recently from 20ab25f to 36c0da2 Compare May 13, 2024 02:37
@JunRuiLee JunRuiLee marked this pull request as ready for review May 13, 2024 02:38
@JunRuiLee JunRuiLee force-pushed the jm_fo_main branch 4 times, most recently from 6368530 to 13c231d Compare May 19, 2024 10:48
zhuzhurk
zhuzhurk reviewed May 21, 2024
View reviewed changes
flink-runtime/src/main/java/org/apache/flink/runtime/shuffle/JobShuffleContext.java Show resolved Hide resolved
flink-runtime/src/main/java/org/apache/flink/runtime/shuffle/ShuffleMaster.java Show resolved Hide resolved
flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/JobMasterTest.java Outdated Show resolved Hide resolved
...apache/flink/runtime/executiongraph/failover/RestartPipelinedRegionFailoverStrategyTest.java Outdated Show resolved Hide resolved
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultScheduler.java Outdated Show resolved Hide resolved
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultScheduler.java Outdated Show resolved Hide resolved
flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/JobVertex.java Outdated Show resolved Hide resolved
flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/JobVertex.java Outdated Show resolved Hide resolved
@JunRuiLee JunRuiLee force-pushed the jm_fo_main branch 2 times, most recently from eb02365 to e50adee Compare May 22, 2024 05:12
zhuzhurk
zhuzhurk reviewed May 22, 2024
View reviewed changes
...main/java/org/apache/flink/runtime/scheduler/adaptivebatch/DummyBatchJobRecoveryHandler.java Outdated Show resolved Hide resolved
.../src/main/java/org/apache/flink/runtime/scheduler/strategy/VertexwiseSchedulingStrategy.java Outdated Show resolved Hide resolved
...e/src/main/java/org/apache/flink/runtime/scheduler/adaptivebatch/AdaptiveBatchScheduler.java Outdated Show resolved Hide resolved
...e/src/main/java/org/apache/flink/runtime/scheduler/adaptivebatch/AdaptiveBatchScheduler.java Outdated Show resolved Hide resolved
...e/src/main/java/org/apache/flink/runtime/scheduler/adaptivebatch/AdaptiveBatchScheduler.java Outdated Show resolved Hide resolved
...e/src/main/java/org/apache/flink/runtime/scheduler/adaptivebatch/AdaptiveBatchScheduler.java Show resolved Hide resolved
.../src/main/java/org/apache/flink/runtime/scheduler/adaptivebatch/BatchJobRecoveryHandler.java Outdated Show resolved Hide resolved
.../src/main/java/org/apache/flink/runtime/scheduler/adaptivebatch/BatchJobRecoveryHandler.java Outdated Show resolved Hide resolved
...e/src/main/java/org/apache/flink/runtime/scheduler/adaptivebatch/AdaptiveBatchScheduler.java Show resolved Hide resolved
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/strategy/SchedulingStrategy.java Outdated Show resolved Hide resolved
zhuzhurk
zhuzhurk reviewed May 22, 2024
View reviewed changes
...in/java/org/apache/flink/runtime/scheduler/adaptivebatch/DefaultBatchJobRecoveryHandler.java Outdated Show resolved Hide resolved
...in/java/org/apache/flink/runtime/scheduler/adaptivebatch/DefaultBatchJobRecoveryHandler.java Outdated Show resolved Hide resolved
...in/java/org/apache/flink/runtime/scheduler/adaptivebatch/DefaultBatchJobRecoveryHandler.java Outdated Show resolved Hide resolved
...in/java/org/apache/flink/runtime/scheduler/adaptivebatch/DefaultBatchJobRecoveryHandler.java Outdated Show resolved Hide resolved
...in/java/org/apache/flink/runtime/scheduler/adaptivebatch/DefaultBatchJobRecoveryHandler.java Outdated Show resolved Hide resolved
...in/java/org/apache/flink/runtime/scheduler/adaptivebatch/DefaultBatchJobRecoveryHandler.java Outdated Show resolved Hide resolved
...in/java/org/apache/flink/runtime/scheduler/adaptivebatch/DefaultBatchJobRecoveryHandler.java Outdated Show resolved Hide resolved
...in/java/org/apache/flink/runtime/scheduler/adaptivebatch/DefaultBatchJobRecoveryHandler.java Outdated Show resolved Hide resolved
...in/java/org/apache/flink/runtime/scheduler/adaptivebatch/DefaultBatchJobRecoveryHandler.java Outdated Show resolved Hide resolved
...e/src/main/java/org/apache/flink/runtime/scheduler/adaptivebatch/AdaptiveBatchScheduler.java Outdated Show resolved Hide resolved
@JunRuiLee
Copy link
Contributor Author

Thanks @zhuzhurk for reviewing, I've updated this pr accordingly. PTAL.

@JunRuiLee
[FLINK-33892][runtime] Reset dynamic decided parallelism before compu…
daa5ce5 
…te parallelism store.

This commit is to maintain consistency between the job graph loaded from the file and the one in memory.
zhuzhurk
zhuzhurk reviewed May 23, 2024
View reviewed changes
...e/src/main/java/org/apache/flink/runtime/scheduler/adaptivebatch/AdaptiveBatchScheduler.java Outdated Show resolved Hide resolved
...e/src/main/java/org/apache/flink/runtime/scheduler/adaptivebatch/AdaptiveBatchScheduler.java Outdated Show resolved Hide resolved
...in/java/org/apache/flink/runtime/scheduler/adaptivebatch/DefaultBatchJobRecoveryHandler.java Outdated Show resolved Hide resolved
...in/java/org/apache/flink/runtime/scheduler/adaptivebatch/DefaultBatchJobRecoveryHandler.java Show resolved Hide resolved
...in/java/org/apache/flink/runtime/scheduler/adaptivebatch/DefaultBatchJobRecoveryHandler.java Outdated Show resolved Hide resolved
...in/java/org/apache/flink/runtime/scheduler/adaptivebatch/DefaultBatchJobRecoveryHandler.java Outdated Show resolved Hide resolved
...in/java/org/apache/flink/runtime/scheduler/adaptivebatch/DefaultBatchJobRecoveryHandler.java Outdated Show resolved Hide resolved
zhuzhurk
zhuzhurk reviewed May 24, 2024
View reviewed changes
...e/src/main/java/org/apache/flink/runtime/scheduler/adaptivebatch/AdaptiveBatchScheduler.java Outdated Show resolved Hide resolved
flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobMaster.java Show resolved Hide resolved
flink-tests/src/test/java/org/apache/flink/test/scheduling/JMFailoverITCase.java Outdated Show resolved Hide resolved
flink-tests/src/test/java/org/apache/flink/test/scheduling/JMFailoverITCase.java Outdated Show resolved Hide resolved
flink-tests/src/test/java/org/apache/flink/test/scheduling/JMFailoverITCase.java Outdated Show resolved Hide resolved
flink-tests/src/test/java/org/apache/flink/test/scheduling/JMFailoverITCase.java Outdated Show resolved Hide resolved
flink-tests/src/test/java/org/apache/flink/test/scheduling/JMFailoverITCase.java Outdated Show resolved Hide resolved
flink-tests/src/test/java/org/apache/flink/test/scheduling/JMFailoverITCase.java Outdated Show resolved Hide resolved
flink-tests/src/test/java/org/apache/flink/test/scheduling/JMFailoverITCase.java Outdated Show resolved Hide resolved
flink-tests/src/test/java/org/apache/flink/test/scheduling/JMFailoverITCase.java Outdated Show resolved Hide resolved
@JunRuiLee JunRuiLee force-pushed the jm_fo_main branch 2 times, most recently from 5e2ba9b to eaffd59 Compare May 24, 2024 09:41
zhuzhurk
zhuzhurk reviewed May 24, 2024
View reviewed changes
flink-tests/src/test/java/org/apache/flink/test/scheduling/JMFailoverITCase.java Outdated Show resolved Hide resolved
flink-tests/src/test/java/org/apache/flink/test/scheduling/JMFailoverITCase.java Outdated Show resolved Hide resolved
flink-tests/src/test/java/org/apache/flink/test/scheduling/JMFailoverITCase.java Outdated Show resolved Hide resolved
...ime/src/test/java/org/apache/flink/runtime/scheduler/adaptivebatch/BatchJobRecoveryTest.java Outdated Show resolved Hide resolved
...ime/src/test/java/org/apache/flink/runtime/scheduler/adaptivebatch/BatchJobRecoveryTest.java Outdated Show resolved Hide resolved
...ime/src/test/java/org/apache/flink/runtime/scheduler/adaptivebatch/BatchJobRecoveryTest.java Outdated Show resolved Hide resolved
...ime/src/test/java/org/apache/flink/runtime/scheduler/adaptivebatch/BatchJobRecoveryTest.java Outdated Show resolved Hide resolved
...ime/src/test/java/org/apache/flink/runtime/scheduler/adaptivebatch/BatchJobRecoveryTest.java Outdated Show resolved Hide resolved
@JunRuiLee
Copy link
Contributor Author

Thanks @zhuzhurk for the thorough review. I have refactored the BatchJobRecoveryTest and JMFailoverITCase based on your comments. PTAL.

zhuzhurk
zhuzhurk reviewed May 27, 2024
View reviewed changes
...ime/src/test/java/org/apache/flink/runtime/scheduler/adaptivebatch/BatchJobRecoveryTest.java Outdated Show resolved Hide resolved
...ime/src/test/java/org/apache/flink/runtime/scheduler/adaptivebatch/BatchJobRecoveryTest.java Outdated Show resolved Hide resolved
...ime/src/test/java/org/apache/flink/runtime/scheduler/adaptivebatch/BatchJobRecoveryTest.java Outdated Show resolved Hide resolved
...ime/src/test/java/org/apache/flink/runtime/scheduler/adaptivebatch/BatchJobRecoveryTest.java Outdated Show resolved Hide resolved
...ime/src/test/java/org/apache/flink/runtime/scheduler/adaptivebatch/BatchJobRecoveryTest.java Outdated Show resolved Hide resolved
...ime/src/test/java/org/apache/flink/runtime/scheduler/adaptivebatch/BatchJobRecoveryTest.java Outdated Show resolved Hide resolved
...ime/src/test/java/org/apache/flink/runtime/scheduler/adaptivebatch/BatchJobRecoveryTest.java Outdated Show resolved Hide resolved
...ime/src/test/java/org/apache/flink/runtime/scheduler/adaptivebatch/BatchJobRecoveryTest.java Outdated Show resolved Hide resolved
...ime/src/test/java/org/apache/flink/runtime/scheduler/adaptivebatch/BatchJobRecoveryTest.java Show resolved Hide resolved
...ime/src/test/java/org/apache/flink/runtime/scheduler/adaptivebatch/BatchJobRecoveryTest.java Outdated Show resolved Hide resolved
zhuzhurk
zhuzhurk approved these changes May 27, 2024
View reviewed changes
Copy link
Contributor

@zhuzhurk zhuzhurk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing all the comments. @JunRuiLee
LGTM.

@JunRuiLee JunRuiLee changed the title [FLINK-33892][runtime] Support Job Recovery from JobMaster Failures for Batch Jobs. [FLINK-35465][runtime] Introduce BatchJobRecoveryHandler for recovery of batch jobs from JobMaster failures. May 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhuzhurk zhuzhurk zhuzhurk approved these changes

Assignees
No one assigned
Labels
component=Runtime/Coordination
Projects
None yet
Milestone
No milestone
3 participants
@JunRuiLee @flinkbot @zhuzhurk