Sync optimizations #6234
Open
Sync optimizations #6234
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* CBG-2894: Reject user auth when channel threshold is over 500 in serverless mode * fix panic where authetciator was needed and it wasn't availible * linter issue * linter issue again * remove extra methods off interface * pass user into function * rebase * ensure 500 code is retruned for http error added * updates based off comments * fix panic * updates based off comments * updates based off dicussion yesterday * lint error * updates based of comments
* Add stats for transferred of bytes data ReplicationBytesReceived and ReplicationBytesSent * Stats are updated when a BlipSyncContext exits and at a threshold set by DatabaseContextOptions.BlipStatsReportingInterval (set to thirty seconds arbitrarily to see perf impact).
* Use sgMgrEventHandlers context to add cancellation handling for EOF feed errors * Move all importListener.Stop cbgtContext code into CbgtContext.Stop()
…hanges feed (#6243) * CBG-2853 Add requestPlus option for changes feeds Adds requestPlus option for changes feeds. When set, changes feeds will loop until the cached sequence (via DCP) is greater than the database sequence at the time the changes request was issued. requestPlus can be enabled for non-continuous changes requests in one of three ways: - by setting request_plus=true on a REST API changes call - by setting the requestPlus property to "true" on a subChanges message - by setting "changes_request_plus":true in the database config (default=false) The request setting is given priority - if not set on a request, the value will fall back to the database config value. Required minor refactoring of how options.Wait was used in changes.go, to support use of requestPlus and longpoll together. No functional changes to longpoll if requestPlus is not set. * Update docs for request_plus changes parameter. * lint fixes
…replication protocol (#6242)
* remove cached buckets * Add refcounting for parallel calls to get buckets
* Update waiting error message from generic message Co-authored-by: Ben Brooks <ben.brooks@couchbase.com>
* Tweak TestIncrCounter to cover non-equal `def` and `amt` values * Allow zero value in `amt` for Collection.Incr - will write `def` to bucket if counter does not exist
* CBG-2983 Close cbgt agents on database close Manager.Stop() doesn’t close cbgt’s gocb agents - partly because the fts use case is to have one manager that exists for the lifetime of the process. For SG’s usage, where manager lifecycle is bound to a database, we need to shut down these agents when we close the database/importListener. * Improve inline documentation for CloseStatsClients call
* CBG-3001 Avoid bucket retrieval error during OnFeedClose OnFeedError was checking for bucket existence to determine whether to call NotifyMgrOnClose(). This handling isn't necessary for SG, as we want database close to handle shutdown of the import feed (via importListener.Stop()) in the case of a deleted bucket.
- Increase maxInFlightChangesBatches from 2 to 4 - Send `changes` requests as Urgent, as LiteCore does
Updated go-blip, with new API for registering & calling request handlers.
`blipRevSender` manages a list of `revToSend` structs identifying revisions to be sent as `rev` messages, and sends them at a limited rate so that the number being written to the socket and their total size don't exceed a limit.
The struct has a Mutex in it now so it shouldn't be copied.
…6258) * CBG-2793: attachment compaction code erroneously sets failOnRollback * spelling errors and test logging removal + lint error * updates for comments and try fix race condition * linter error fix * changes for jenkins * updates to add handling for cleanup phase * remove logging line * fix race in Jenkins * updates to fix collections issue * updates based off comment * updates * updates to address complexity comments
|
Still awaiting performance testing by perf team: CBPS-1153. Review is blocked on this. |
see commit 3f1855f3bfe9 in go-blip
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
CBG-2952
Improvements to the sync code in 'db' and to 'go-blip', to optimize replication by reducing the amount of memory and goroutines used.
blipRevSenderwhich manages the fetching of doc revisions and sending of 'rev' messages. It throttles this to a limited rate so that the number being written to the socket and their total byte size don't exceed a limit.Pre-review checklist
fmt.Print,log.Print, ...)base.UD(docID),base.MD(dbName))docs/apiDependencies (if applicable)
Integration Tests
GSI=true,xattrs=truehttps://jenkins.sgwdev.com/job/SyncGateway-Integration/000/