Progress now account for more things #228
Conversation
shuhaowu
force-pushed
the
more-progress-info
branch
from
ccb262c
to
bc38482
Compare
| @@ -32,6 +33,9 @@ type Progress struct { | |||
| BinlogStreamerLag float64 // seconds | |||
| Throttled bool | |||
|
|
|||
| // The number of data iterators currently active. | |||
| ActiveDataIterators int | |||
There was a problem hiding this comment.
Would be nice to show especially this one in the UI.
There was a problem hiding this comment.
Yeah I think a ControlServer refactor is coming after #227 cc @Manan007224
| @@ -20,6 +20,8 @@ def test_progress_callback | |||
| assert_equal 1111, progress.last["Tables"]["gftest.test_table_1"]["TargetPaginationKey"] | |||
| assert_equal "completed", progress.last["Tables"]["gftest.test_table_1"]["CurrentAction"] | |||
|
|
|||
| assert_equal 0, progress.last["ActiveDataIterators"] | |||
|
|
|||
There was a problem hiding this comment.
Is it possible that we receive more than one progress reports during the ghostferry run? If so, can you add a check that the ActiveDataIterators is >0 in one of the non-last progresses?
There was a problem hiding this comment.
We do sometimes, so it will introduce flakiness. I've manually verified these feature for now.
we really a better integration testing environment...
|
I agree with Forrest that we can hold off the code for |
shuhaowu
force-pushed
the
more-progress-info
branch
from
bc38482
to
670c571
Compare
Related: #226
Added the current number of active data iterators as well as the number of rows copied per table into the progress struct. This will allow us to track the performance of a Ghostferry move with a bit more precision.
The code is a bit crappy right now as we get this functionality working and measuring in real moves. The reason it's crappy is because the StateTracker now has two roles: tracking the state which is vital for interrupt and resume, as well as measuring some optional performance metrics. These two objectives ideally would be refactored so they're not as closely intertwined.