Increase timeout from 1 minute to 30 minutes #98
Conversation
| case out := <-tableState: | ||
| ch.Logger.Log(internal.LOGLEVEL_INFO, "done syncing "+out.shardName) | ||
| if out.err != nil { | ||
| ch.Logger.Error(err.Error()) |
There was a problem hiding this comment.
| ch.Logger.Error(err.Error()) | |
| ch.Logger.Error(out.err.Error()) |
| @@ -281,9 +282,11 @@ func (p PlanetScaleEdgeDatabase) sync(ctx context.Context, tc *psdbconnect.Table | |||
|
|
|||
| // stop when we've reached the well known stop position for this sync session. | |||
| watchForVgGtidChange := false | |||
| lastRestCount := -1 | |||
There was a problem hiding this comment.
what does Rest mean in this context?
There was a problem hiding this comment.
It's probably not a great variable name
| @@ -314,9 +318,13 @@ func (p PlanetScaleEdgeDatabase) sync(ctx context.Context, tc *psdbconnect.Table | |||
| } | |||
| } | |||
|
|
|||
| if watchForVgGtidChange && tc.Position != stopPosition { | |||
| if (watchForVgGtidChange && tc.Position != stopPosition) || (lastRestCount == 0 && len(res.Result) == 0 && tc.Position == stopPosition) { | |||
There was a problem hiding this comment.
am i reading this right: stop if there are two empty results in a row and we've reached the stop position?
There was a problem hiding this comment.
that's the idea. In testing, for a database with data and no active writes, this tended to be indicative of the stream being caught up.
The existing behavior will just wait for a timeout in that situation, which seems unwise if we are going to increase the timeout.
| return tc, io.EOF | ||
| } | ||
|
|
||
| if !(lastRestCount == -1 && len(res.Result) == 0) { |
There was a problem hiding this comment.
what's the reason for this guard?
There was a problem hiding this comment.
In testing, the initial copy phase would trigger the condition above without this, but I don't precisely know why. Experimental!
cmd/airbyte-source/read.go
Outdated
| @@ -157,7 +157,6 @@ func ReadCommand(ch *Helper) *cobra.Command { | |||
| } | |||
| } | |||
|
|
|||
| ch.Logger.Log(internal.LOGLEVEL_INFO, "done done") | |||
| @@ -37,6 +37,8 @@ type PlanetScaleEdgeDatabase struct { | |||
| Logger AirbyteLogger | |||
There was a problem hiding this comment.
For my own clarification, what will Logger be responsible for vs dataLogger?
There was a problem hiding this comment.
I think, if this ends up being good for the customer, we should separate the "data logger" and the info logger - one of them can be shared across every sync job, one of them can be specific per sync.
Experimental.