New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[vpj] send out pushjob details while hitting exception. #878
Conversation
@@ -2450,6 +2455,7 @@ void pollStatusUntilComplete( | |||
VenicePushJob.this.pushJobSetting.storeResponse.getStore().getBootstrapToOnlineTimeoutInHours(); | |||
long durationMs = System.currentTimeMillis() - pollStartTimeMs; | |||
if (durationMs > TimeUnit.HOURS.toMillis(bootstrapToOnlineTimeoutInHours)) { | |||
sendPushJobDetailsToController(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you explain a bit about how this will help? IIUC, if an exception is thrown, the catch clause will send push job details after marking it as an ERROR
@@ -859,6 +859,8 @@ public void run() { | |||
sendPushJobDetailsToController(); | |||
closeVeniceWriter(); | |||
} catch (Exception ex) { | |||
sendPushJobDetailsToController(); | |||
closeVeniceWriter(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How about moving 859-863 to the final block if we have to invoke them anyways?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we veify that the colo status is being constructed correctly? I don't see how adding more sendPushJobDetailsToController()
will fix the issue of incomplete colo statuses
StringBuilder errorMsg = new StringBuilder().append("Push job error reported by controller: ") | ||
.append(pushJobSetting.veniceControllerUrl) | ||
.append("\ncontroller response: ") | ||
.append(response); | ||
|
||
sendPushJobDetailsToController(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this needed if we already have line 862-863?
Send out pushjob details while hitting exception
During target colo push if the later push job fails, it does not send out the pushjob details to the controller, which later breaks the h2v availability report. This PR sends out pushjob details before throwing exceptions.
How was this PR tested?
GHCI
Does this PR introduce any user-facing changes?