Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[vpj] send out pushjob details while hitting exception. #878

Closed
wants to merge 3 commits into from

Conversation

majisourav99
Copy link
Contributor

Send out pushjob details while hitting exception

During target colo push if the later push job fails, it does not send out the pushjob details to the controller, which later breaks the h2v availability report. This PR sends out pushjob details before throwing exceptions.

How was this PR tested?

GHCI

Does this PR introduce any user-facing changes?

  • No. You can skip the rest of this section.
  • Yes. Make sure to explain your proposed changes and call out the behavior change.

@@ -2450,6 +2455,7 @@ void pollStatusUntilComplete(
VenicePushJob.this.pushJobSetting.storeResponse.getStore().getBootstrapToOnlineTimeoutInHours();
long durationMs = System.currentTimeMillis() - pollStartTimeMs;
if (durationMs > TimeUnit.HOURS.toMillis(bootstrapToOnlineTimeoutInHours)) {
sendPushJobDetailsToController();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain a bit about how this will help? IIUC, if an exception is thrown, the catch clause will send push job details after marking it as an ERROR

@@ -859,6 +859,8 @@ public void run() {
sendPushJobDetailsToController();
closeVeniceWriter();
} catch (Exception ex) {
sendPushJobDetailsToController();
closeVeniceWriter();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about moving 859-863 to the final block if we have to invoke them anyways?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we veify that the colo status is being constructed correctly? I don't see how adding more sendPushJobDetailsToController() will fix the issue of incomplete colo statuses

StringBuilder errorMsg = new StringBuilder().append("Push job error reported by controller: ")
.append(pushJobSetting.veniceControllerUrl)
.append("\ncontroller response: ")
.append(response);

sendPushJobDetailsToController();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this needed if we already have line 862-863?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants