Release/2.8.2 #2

* CBG-716: Fail on unknown fields on config PUT * Basic test

* CBG-758: Reliably flush to console on Fatalf * Switch waitgroup to pointer * Fix typo

…trap (#4538)

* CBG-724: Deletions of empty documents are not imported * CBG-724: Deletions of empty documents are not imported When the document is a delete, the bodyBytes would be replaced with “{}” and we can’t use bodyBytes in CRC-32 checksum comparison to determine whether the mutation was an SG write and the document has already been imported. Need to set the current value of CRC-32 checksum from bodyBytes when the document is non-delete and revert back to the correct CRC-32 checksum “0x00” before we replace the bodyBytes otherwise (if the document is delete). * CBG-724: Deletions of empty documents are not imported - Added additional unit tests (TestUpdateXattrWithDeleteBodyAndIsDelete, TestUpdateXattrWithDeleteBodyAndIsNotDelete) in bucket_gocb_test.go to validate the new behaviour of UpdateXattr. - Refactored XattrMacroCrc32cDelete to DeleteCrc32c and added short comment describes what it represents. - Removed obsolete comments left over from working session; added a brief description about the situation and need to modify currentBodyCrc32c in the delete case.

…mmitting TestDeletedEmptyDocumentImport along with CBG-724. (#4539)

* CBG-752: SGCollect avoid unnecessary unzip * Address PR comments

… compact (#4544) * CBG-812 - Fixes infinte loop caused by stale view query when running compact * Use consistency param * Remove stale parameter and hardcode

This speeds up full integration test runs significantly, by moving the bucket setup/teardown into an async worker which gets buckets ready in the background while tests are running using a different bucket. ## Usage: ``` export SG_TEST_BACKING_STORE=Couchbase export SG_TEST_COUCHBASE_SERVER_URL=couchbase://localhost go test -v -p=1 -count=1 ./... ``` ### Configuration #### Existing - `SG_TEST_BACKING_STORE` (default `Walrus`) - Can be set to `Couchbase` to enable integration tests - `SG_TEST_COUCHBASE_SERVER_URL` (default `http://localhost:8091`) - The connection string to connect with Couchbase Server #### New - `SG_TEST_USERNAME` (default `"Administrator"`) - The username to use when connecting to the test server - `SG_TEST_PASSWORD` (default `"password"`) - The password to use when connecting to the test server - `SG_TEST_BUCKET_POOL_SIZE` (default `3`) - Specify how many buckets to create and use. - `SG_TEST_BUCKET_QUOTA_MB` (default `150`) - Specify how many MB each bucket's RAM quota should be - `SG_TEST_BACKING_STORE_RECREATE` (default `false`) - Drops any existing test buckets, and creates new ones before continuing to add them to the pool. If this is not set, previous buckets are just readied as normal. - `SG_TEST_BUCKET_POOL_PRESERVE` (default `false`) - In the event of a failing test, the bucket used is not emptied and put back into the pool. This may result in exhaustion of buckets if more tests fail than the pool size. - `SG_TEST_BUCKET_POOL_DEBUG` (default `false`) - If "true", verbose bucket pool logging is turned on, to see what is happening to each bucket whilst running. - `SG_TEST_DISABLE_GSI` - Always set to true until CBG-818 --- * CBG-625 - Add bucket pooling test framework. * Simplify LoggingBucket log code, and include bucket name in log context * Prevent test failures when using walrus buckets * Check for TestBucket pointer in AsGoCBBucket * Fix WaitGroup data race by incrementing counter before sending bucket over bucketReadierQueue * Prevent data race with multiple concurrent cluster.Authenticate() calls by using a pre-authed cluster reference * Fix data race around initialisation of bucketReadierWaitGroup * Add getTestKeyNamespace to force unique doc IDs for xattr tests (avoids KV tombstone issues) * Increase timeout on retry loop for WaitForNumDocsViaChanges * Bump gocb/gocbcore for cluster ops pre-bucket open * Add SG_TEST_USE_VIEWS flag, but force it to be true all the time until 6.5.1 * Fix TestQuerySequencesStatsN1ql bucket type/view check * Use TestsUseViews in RestTester setup * Avoid vet catching unreachable code * Use TestsUseViews for ServerContext test * rename TestsUseViews to TestsDisableGSI and use in PostUpgrade * Handle View and Query stats in TestTombstoneCompaction * Bump bucket pool size from 100MB to 150MB * Parameterise bucket quota * Improve comments, rename things to be TBP scoped, rearrange to tidy * More cleanup * Tweak values in TestAddRawTimeoutRetry to avoid exceeding 150MB bucket quota, whilst still reproducing issue * Add retry loop around UpsertDesignDocument to handle sporadic Erlang 500 errors * Bump sg-bucket/walrus revisions * Handle non-nil empty error from gocb's GetBuckets for 401 status code * Add SG_TEST_USERNAME/SG_TEST_PASSWORD env vars * Add benchmark for GetCallersName * Skip printStats if no buckets opened (e.g. benchmarks) * Remove unused GetTestBucketSpec * Remove alignment chars from test logging * Add comments for unused functions * Drop default test pool size to 3 * Use t.Name() from getTestKeyNamespace * Simplify document keys in single-doc tests * Remove unnesesary query result Close * Clean up TestBucketPool naming/API * Move primary index creation out into test-only code * Make mockClaims expire far into the future * Fix missed InitializeIndex paramter removal * Refer to global TestBucketPool from db TestMain * Unskip TestPostUpgradeIndexesSimple

Switch to compareAndSwap when toggling compactRunning on channelCacheImpl

* CBG-808: Add LiteCore tests

The change to include the active-only criteria in the N1QL query, instead of post-cache retrieval filtering, resulted in invalid cache contents. Active-only query results should not be cached.

* CBG-825: Allow session create with invalid session * Clarify comments * Switch to use publicPrivs

…n non-default (#4551) * CBG-801: Auto-generated OIDC callback URL should include provider when non-default * CBG-801: Added changes to handle updating the provider value * CBG-801: Updated the error code returned by failures to set parameters to URL query string * CBG-801: Added changes in getOIDCCallbackURL to use SetURLQueryParam while building the callback URL * CBG-801: Changes track URL parsing errors and test case for provider name that would need URL encoding. - Added test case for a provider name that would need URL encoding. - Added changes to track underlying error * Fixed issue in adding provider to callback URL and included Unit tests * CBG-801: Handle error when writing mocked response body for discovery endpoints in TestGetOIDCCallbackURL * CBG-801: Revised getOIDCCallbackURL to remove nesting if clauses while building the callback URL.

Adds query pagination for channel queries triggered by changes processing. Default query limit for pagination is 5000, and can be modified with a new ‘query_limit’ config property in channel cache config. Pagination is done while building the per-channel feed (go-channel). This avoids storing the entire cross-query resultset in memory, allows streaming of results as soon as the first query completes, and optimizes interaction with any changes request limit. This has implications for cache warming - only the last (query_limit) entries will be prepended to the cache per changes request.

* Make GetTestKeyNamespace public * Remove getTestKeyNamespace(t) in favour of t.Name()

…sts (#4578)

* Expose pool bucket cred methods * Add x509 test framework * Split into util and tests file * configurable SSH username * Enable optional X.509 Client certs in CBS, skip failing CBG-844 test * Ignore errors from resp.Body.Close() * clarify error messages * Undo node rename approach, enforce tests have required hostname set before running * Fix default port handling * Adopt comments from #4577 into here, and remove unused function * assert on error returned from teardown * Rename the basic auth REST API URL helper * Tweak comments for couchbaseNodeConfiguredHostname

* Add go-jose.v2 for fixing the build failure and running the coverage * Resolve merge conflicts in base/util_testing.go * Verification of new OIDC coverage against the old library * oidc-coverage: Add wantError struct to assert the response * oidc-coverage: Change token expiry assertion * oidc-coverage: Add forceAuthError, forceRefreshError fields in the test struct and remove the switch conditional processing. - Additional functions to return a provider in the give name with different settings; mockProvider, mockProviderWithRegister, mockProviderWithRegisterWithAccessToken. - Remove 'mock' prefix on the function names (e.g. this can just be discoveryHandler, authHandler, tokenHandler), to remove redundancy and ambiguity. - Revise code comments - Rename 'wantError' to 'forceError' and 'wantErrKind' to 'forceErrorType'. * oidc-coverage: Change error message text to handle 'no such host' and 'i/o timeout' error * oidc-coverage: Remove TestFetchCustomProviderConfigWithBadURL; it doesn't add any value * oidc-coverage: Additional unit tests that covers registered user authentication

…4584)

* CBG-804: OpenID Connect Library Upgrade * CBG-804: Fix status codes to handle errors in auth and refresh scenarios - Remove gocbconnstr dependency changes committed against this PR from manifest file. - Remove client ID empty check which is obsolete. - Return status 500 internal server error when there is any failure in auth/refresh token exchange. - Return status 401 unauthorized error when no token_id is received from OAuth server in auth/refresh responses. - Fix the token expiration time sent from fake OAuth server and correct the assertion logic. * CBG-804: Status response correction against token exchange and auth failure scenarios * CBG-804: Check the token expiry is within 5 seconds of the expected value for auth and refresh responses

…ure (#4585)

…om ReadJSONFromMIME when request body is empty (#4587)

#4586) * CBG-876: Include Bearer prefix when sending token base authentication request in OIDC E2E tests - Move getRSASigner() from util_testing.go to oidc_test.go. - Include Bearer prefix when sending token base authentication request in OIDC E2E tests. - Attach public keys and signer to the mockAuthServer. - Add keysHandler to expose the public keys to validate a JSON Web Token (JWT) issued. - Include makeToken method on mockAuthServer to creates a default token. - E2E test that checks OpenID Connect Implicit Flow. * CBG-876: Fix the assertion error message "session expiry is not within 5 seconds of 24 hours" * CBG-876: Refactor makeToken and include additional coverage - Add comments against BearerToken stating that it is used for setting JWT token type as well as the prefix for the Authorization HTTP header. - Refactor makeToken method to keep one property per line while building the claims for making auth token. - Yeah, Last time the it was falling under the Guest user flow instead of validating the Bearer token and since the Guest user has access to everything, it was not complaining about any "login issue" inside checkAuth(), returning a successful response from db endpoint and eventually all those tests got succeeded. After adding the changes for sending Bearer token, I've gone through the E2E workflow and confirmed that the JWT token we are sending during Implicit Flow is undergoing successful validation. But when I was going through this review comments, I did realize that the test require an additional fix that is disabling the Admin Party when spinning up the fake SG server at the start of the test. This will cover the negative scenario; the test will fail if we send a valid token with "Bearer " prefix. - Refactor getCookie function to use enhanced for loop. - Since we're already checking whether the DefaultCookieName exists, skip the check for kDefaultSessionTTL and make sure the response body has the expected details like username, channels, authentication_handlers etc. - Refactor makeToken method to accept claimSet which contains both primary and secondary claims to make it convenient to adjust claims from the caller for specific use cases. - Include additional unit test cases to cover the edge cases.

* Pull shared BLIP code into new replicator package * CBG-779 and CBG-780: Add ActiveReplicator APIs and implement blipsync * Add basic ActiveReplicator test * Remove sleep, defer Close() * Use public REST API with user and rename test * Remove old comment (and try to force GitHub to update the PR) * Address PR comments * Automatically connect to target on ActiveReplicator Start

) * CBG-770-prereq Refactor heartbeater to support multiple listeners The existing heartbeat implementation was tightly coupled with a single listener (cbgt). Refactored to support sharing of a single heartbeater among multiple listeners. Simplified heartbeater interface to better encapsulate functionality. For shared heartbeat, a common node ID needs to be used for both heartbeater and its listeners - moved this from cbgt up to databaseContext. * Cleanup/fixes based on PR feedback

… Connect library upgrade (#4591)

* CBG-820: Maintain state between the OpenID Connect authentication request and the callback - Add disable_callback_state option to determines whether or not to maintain state between the Authentication request and the callback. - Add callback_state_cookie_http_only to determine whether or not to make the callback state cookie inaccessible to JavaScript's - - Document.cookie API to mitigate cross-site scripting (XSS) attacks. - Add 'sg-oidc-state' cookie to maintain the state token with a duration of 5 minutes. - Introduce ErrNoStateCookie, ErrStateMismatch, ErrReadStateCookie errors for callback state error handling. - Set state parameter to prevent cross-site request forgery (CSRF) when DisableCallbackState is not enabled. - Validate state parameter to prevent cross-site request forgery (CSRF) when callback state is enabled. - Delete the state cookie on successful state validation. - Add an error type invalidStateErr to force the /auth API to return an invalid state token in the callback URL for unit testing. - Add cookieJar to HttpClient instance to insert relevant cookies into every outbound Request and update it with the cookie values of every inbound Response. - Add coverage for the following scenarios: - Successful new user authentication with HttpOnly cookie enabled. - Successful new user authentication with HttpOnly cookie enabled. - Successful new user authentication with callback state disabled. - Unsuccessful new user authentication against CSRF attack. - Successful registered user authentication with HttpOnly cookie enabled. - Successful registered user authentication with callback state disabled. - Unsuccessful registered user authentication against CSRF attack. * CBG-820: Fix the comments over written while resolving merge conflict on rebase against master * CBG-820: make the callback state cookie always inaccessible to JavaScript's Document.cookie API * CBG-820: Refactor setSateCookie method and rename to makeStateCookie

* CBG-800: OnDemandImportForWrite migrate handling * Address PR comments * Fix test and added check * Better sync property check * Address PR comments * Reworked handling * Work on test * Run callback once * Temporary PoC test * Move test to use leaky bucket * Upgrade logging for paniced test * Rollback logging change * Address PR comments * Update manifest

* Add golangci-lint config file * Disable goconst linting in test files

) Moves heartbeater initialization to database, and shares use between sg-replicate and import. Includes addition of sgreplicate_enabled config property to allow nodes to opt out of sgreplicate.

) * Refactor generateChanges to move REST handler-specific handling out into h.generateContinuousChanges * Switch blip generateChanges context to TODO * Ignore typed error for changes send errors

* Move blipSyncContext into replicator package * Pull latest changes from #4594

* SGR2: Initial pull replication * Rename BlipSyncContext context ID parameter * Minor tidy of unused struct field * Simplify by only setting blipSyncContext field * Use db.SequenceID type instead of uint64 for ActiveReplicator since * Add all missing subChanges params, and Rename BidirectionalActiveReplicator to ActiveReplicator * Rename PassiveDB to PassiveDBURL * Rename variables for consistency after Struct/Field renames

… it along with auth code (#4602)

* Expand waitForIndex for more error cases Modifies waitForIndex to retry on all errors. Indefinite retry for timeouts, backoff retry for all other errors. * Reduce max retry for a better balance of SG retry and service retry.

* Reenable TestActiveReplicatorPullBasic * Bump go-blip manifest * Update manifest/default.xml

…pport cookies when OIDC callback state is enabled. (#4611)

* Send getCheckpoint to set since sequence before starting pull replication * Handle checkpointing from ActivePullReplicator * minor test tidy up * Add pull replicator checkpointing * Implement checkpoint ID hash * Bump walrus manifest for PR branch * Bump walrus manifest revision for merged PR * Address PR comments * Fix maxI handling Setting maxI as &i incremented maxI even when breaking out of the loop. Switched back to int, initialized to -1. * Use default checkpoint interval if not specified in config Co-authored-by: Adam Fraser <adam.fraser@couchbase.com>

Avoids cyclical dependencies between ActiveReplicator's use of a DB instance, and DB's collection of active replications. Straight move for most files, the only exception is that the contents of replicator/changes.go were appended to db/changes.go.

* Update ORDER BY to align with index * Avoid extra whitespace when active only is not being used

* CBG-838: Add sync_gateway user check * Add user definition too

* CBG-880: SGCollect Python3 compatibility * Address PR comments * Upload fixes * Remove remaining double parentheses

* CBG-864: Use addJSON errors and detection change * Add connection reset check * Switch error check back to strings

* Refactor getOrAddDatabaseFromConfig for clarity Moved generation of databaseContextOptions into helper method to clarify the sequence of operations during database creation. * Connect replication API with active replicator Initialize config-based replications on startup, start replications on updated config.

* CBG-768 Report stats for active replications Implement _replicationStatus for local, active replications. Non-active replication status is still stubbed out pending CBG-909. * Fix for replications assigned on startup Previously were not being added to activeReplications. Includes additional diagnostic logging.

* ConflictResolverFunc definition and API Adds ConflictResolverFunc API, implementations of default/localWins/RemoteWins, and constructor to return function based on config value. * Add test for default conflict resolver

* CBG-914 Implement basic push replication * Rip out unused checkpointer code * Restore accidental removal of constructor * TestActiveReplicatorBlipsync now opens 2 blipsync connections (PushAndPull) * Run sendChanges in the push replicator in a goroutine

* CBG-777 Javascript custom conflict resolver Javascript custom conflict resolver taking a conflict as input, and returning a document body. * Whitespace fix * Apply suggestions from code review Co-authored-by: Ben Brooks <ben.brooks@couchbase.com>

) * Share connect funciton for active replicators * Refactor blip.sendChanges for generic options * remove blipSyncContext sendChanges variables * Remove old values from blipSyncContext struct * Identify SGR2 and CBL2 clients

…4627) * CBG-778 Invocation of default policy from custom conflict resolver Supports calling defaultPolicy(conflict) from a custom conflict resolver in order to perform default conflict resolution. * Whitespace cleanup * Update sg-bucket revision for merged PR Co-authored-by: Ben Brooks <ben.brooks@couchbase.com>

* Adds adhoc flag to replicationConfig Currently just a flag, behaviour when replications are stopped is pending CBG-924. * Fix copy/paste error Co-authored-by: Ben Brooks <ben.brooks@couchbase.com>

* Rename stats accessors for consistency * Initial implementation of push checkpointing * Share common checkpointer code * Some more tidy up (removed unused functions) * Remove interim changes * Rename callbacks and move pre-response callback to be after a successful send

* Add fgprof debug endpoint * Use standard debug seconds query param * Add temporary felixge manifest remote for fgprof * Add google/pprof dependency * Switch to mirrored repos

* Add test to pull tombstone in SGR2 * Wait for change on rt1

…4634) * Purge document on removal when PurgeOnRemoval flag is set * Set checkpoints for purged docs * Include stat for SGR2 docs_purged

…ames) (#4633) * Wire up config options * Bump go-blip manifest * Bump go-blip manifest * Bump go-blip manifest * Bump go-blip manifest * Remove duplicate blip.NewContext/NewBlipSyncContext and tidy up Checkpointer context * WebsocketPingInterval config tidy from PR comments * Change websocket config name * Remove old/unused constant * Remove old field from ServerConfig * Update field name to _secs

Passive replication continues to use the database's context, active replication uses a replication-specific context for additional logging context. Also removes omitempty to always display continuous=true/false in _replication response.

…TPS (#4628) * CBG-802: Being able to configure OpenID Connect Test Provider with HTTPS * CBG-802: Clone the default http.Transport and apply selective changes to the copy while setting insecureSkipVerify * CBG-805: Check whether the transport.TLSClientConfig is nil, and update the existing if non-nil

* Invoke conflictResolver during PutExistingRev Modify PutExistingRev to support invocation of conflict resolver when provided. Adds wrapper for ConflictResolverFunc to identify whether result is local/remote/merge based on revID comparison. * Updating revision tree based on conflict resolution outcome * Fixes for pushAndPull with conflict resolution SGR2 push replications need to send revisions with the noconflicts property set to true in order to avoid causing branched revision trees on the remote when the remote Sync Gateway is running with allowConflicts=true. Adds unit test to validate pushAndPull results. * Error check fix * Updates based on PR feedback Removed unused code, reworked assignment of conflict resolver to replication to ensure it's only set for pull replications.

…#4644)

Adds batch_size config parameter, passes through to activeReplicator.

* Specify docIDs in subChanges only when we have any to send * Handle marshal error when sending subChanges

If delta sync is disabled for an sg-replicate replication, override the calculated sgCanUseDeltas value for the replication's blip sync context. Note that this implies that pull replications from a remote that supports delta sync will still not use deltas unless deltas are enabled on the local node. This matches CBL behaviour today (we don't accept deltas from CBL if deltas aren't enabled for the SG database).

)

…last_seq_checkpointed (#4652)

* Add rev response callback * Invoke callback for all response types * Address PR feedback * Tweak stat reporting * Add doc_write_conflict stat

…scoped (#4650)

Includes config definition in GET replicationStatus response when includeConfig=true query parameter is set.

* CBG-805: Update test OIDC provider to simulate real-world token * CBG-805: Simulate tokens for Microsoft Identity Platform and Yahoo! token format

…rror (#4659)

* One-shot stopping for pull replication * One-shot stopping for push replication * Initialize active replicator state to stopped * Test fixes * Avoid race on concurrent Stop, Stop and Replace * Fix races in activeReplicator Switch to a single lock for the replicator, and use for access to any internal properties.

Callback invocation on replication completion was attempting to re-acquire the activeReplicator lock. Switched onComplete to be invoked after releasing the lock, and also refactored to avoid unnecessary status call on replication complete.

Remove adhoc replications that are stopped (either via _replicationStatus?action=stop, or on completion of a one-shot replication).

* Delta sync config fix and stats Fixes config validation that was forcing replications to run with delta sync disabled, and adds stats for deltas sent/received/requested. * Simplify DeltasEnabled assignment

Co-authored-by: Sarath Kumar Sivan <4345604+sarathkumarsivan@users.noreply.github.com> Co-authored-by: Ben Brooks <ben.brooks@couchbase.com>

Adds support for localOnly, activeOnly, includeError as replicationStatus API query parameters.

#4657) * Accept 0 as a valid Expires and return the discovery data as not cacheable * Set ttl=MaxProviderConfigSyncInterval if cacheable() returns error, instead of failing to initialize the metadata * Change logging level to info instead of debug while reporting cachable error

)

) * CBG-905: Support using custom OIDC claim as Sync Gateway username * CBG-906: Handle non-string values as username claim in identity token * CBG-906: Refactor handling non-string values as username claim in identity token * Update auth/oidc.go Refactor comments against username_claim Co-authored-by: Adam Fraser <adam.fraser@couchbase.com> * CBG-906: Refactor handling non-string values as username claim in identity token Co-authored-by: Adam Fraser <adam.fraser@couchbase.com>

* Checkpoint when revisions are ignored by a peer. * Add missing 'Pull' prefix in stat constant * Add specific AddIgnoredSeq method to reduce lock/unlock churn * Address PR comments * Batch push expected/alreadyKnown seq tracking * Pluralise callback names that now do batching * Final ignoredSeqs -> alreadyKnownSeqs rename

…heckpoint write (#4671) * Update last known rev IDs when we encounter a conflict on write * Move local/remote setCheckpoint retry handling into shared function * Add get/setCheckpointFn parameter names

MaxNumRetries isn't changed, so overall retry time is shorter for default spec values of MaxNumRetries and InitialRetrySleepTimeMS (by about 2s).

…ng 'latin-1' encoding (#4672)

…rough import listener (#4675) Removed obsolete importListener.NotifyCfg, moved prefix removal into FireEvent.

Get1xRevAndChannels was attempting to return the transport-only revID property, which isn't set in this usage.

…ck to go-blip (#4680)

Avoids SG interaction with transaction metadata documents. No additional handling is needed for on-demand import scenarios, as the underscore-prefixed metadata documents are not have valid SG docIDs and already can't be accessed through SG's REST API.

Added missing Direction validation to ValidateReplication, reused ValidateReplication on replication start.

* Track sequences for pulled norevs for checkpointing purposes * Move IDAndRev handling for pull replications/norev seq lookups down into checkpointer * Minor tweak for readability

… function (#4678) * CBG-951: Deleted documents should set _deleted:true for import filter function * CBG-926: Set delete property when body is both empty and non-empty and refactor unit test * CBG-926: Take a shallow copy of the body and set _deleted property the import is for delete and the body is not empty

…fails (#4691)

#4690) * CBG-949: Improve error when non-upgradable HTTP request is sent to _blipsync endpoint * CBG-949: Exit early when the connection can't be switched to websocket protocol without incrementing the status * CBG-949: Fix Jenkins CI build failure due to wrong status code assertion in TestBlipSyncNonUpgradableConnection

The nodeListenerMap is useful when triaging node assignment issues, but inclusion of listener pointer addresses made it less readable.

…while running sgcollect (#4694)

Co-authored-by: Ben Brooks <ben.brooks@couchbase.com>

Allows for Hydrogen to Hydrogen push replication to a peer running in no conflicts mode.

#4696)

…#4699)

…sword) as separate config options (#4697)

#4703)

…ormation (#4700)

* Rely on dbContext/serverContext close to call bucket teardown, avoids incorrect ordering of closing a bucket before dbContext * Avoid testBucket.Bucket pattern in Authenticator/DatabaseContext * Handle getting underlying GoCB Bucket in CouchbaseURIToHttpURL

…4711)

…nError to prevent appearing in searches for 'FAIL:' (#4713)

…ion fails (#4692)

… is in the config (#4706)

Co-authored-by: Adam Fraser <adam.fraser@couchbase.com>

… lock is acquired (#4715)

Ensure allocated sequences are used or released prior to cache start

…annel (#4714) * Working session for CBG-946 * CBG-946: Repeated change entries on access grant to doc's previous channel * CBG-946: Refactor TestPostChangesAdminChannelGrantRemoval * CBG-946: Format package imports * CBG-946: Additional coverage in TestPostChangesAdminChannelGrantRemoval * CBG-946: Fix build failures in TestGetRemovedDoc and TestPostChangesAdminChannelGrantRemoval * CBG-946: Refactor TestPostChangesAdminChannelGrantRemoval include WaitForChanges enough to avoid this fixed sleep * CBG-946: Disable sequence batching for multi-RT tests * Update rest/replication_api_test.go Co-authored-by: Ben Brooks <ben.brooks@couchbase.com> * Update rest/replication_api_test.go Co-authored-by: Ben Brooks <ben.brooks@couchbase.com> * CBG-946: Keep the deleteDoc function limited to just validating the delete via the response * CBG-946: Fix build errors Co-authored-by: Adam Fraser <adam.fraser@couchbase.com> Co-authored-by: Ben Brooks <ben.brooks@couchbase.com>

* CBG-1012 Persist replication status with checkpoints Store replication status with checkpoints instead of as separate status. Supports status retrieval between nodes within a cluster, and status consistency across replication stop/start/rebalance. * Fix for test races Switch to state/error lock in a few places, add support for checkpointer to persist status and update lastLocal safely. * Error handling fix

* CBG-919: Integrated Prometheus Exporter * GSI and SharedBucket * Add float and delta stats * Replication stats and sync.Once * Added Value() in prep for atomic * Mutexes * Most database stats * Cache stats * Handle external expvars * Mutex GSI fix * Ensure all the names are inline with existing * Clean up * Modify when stats initialize * BLIP test tweaks * Rebase Blip Replicator Stats * Addressed some PR comments * address more * Const format for query view stat * Remove old dbStats * More addressing PR comments * Test fix * Last cleanup * Fix dbStats map data race * Rename initQueryStat * Added benchmark * Fix new tests * Added more benchmarks and switched to atomic * Replace sprintf with format * Add sgw_up for compat * Rename metrics config to be inline with others * Test fixes * Test fixes

* CBG-1022: Require replication to be stopped prior to upsert * Address PR comment

* Read full response body before checking response type * Log for error, and apply same handling in SGR2 checkpoint response code * Apply same handling in rev response

* Fix assertion of mixed int/int64 types * Fix import stats init Co-authored-by: JRasagneres <jacquesrascagneres@gmail.com>

…4738)

…nection (#4742)

* Fix reconnect ctx race * Return context error value from reconnect worker for correct ConnectsAborted stats * Last reference to a.ctx in reconnect fn * Revert accidental assert -> require change

* New test for replication status actions * Fix reset handling Reset replication stats and checkpointer when reset is invoked. * Avoid sleep in test

…ot available (#4747)

…er (#4724)

* CBG-1057: Deterministic prometheus stat label ordering * Change to two slices

…#4728) * Invoke reconnect loop when BLIP OnExitCallback is fired. * Bump go-blip manifest for branch * bump go-blip manifest * Prevent reconnect loop firing on intentional replicator stop * bump go-blip manifest for merged PR * Reverse context error check for reconnect * Fix replication rebalance races on RestTester teardown * On ActiveReplicator Stop, cancel the replicator's reconnect handling before disconnecting * Stop ActiveReplicator's reconnect handling on Complete callback * Treat empty body the way io.EOF used to be treated for BLIP changes responses * Compare stats vs. start of test instead of from zero

An insert race can return ErrNotStored (instead of ErrKeyExists) for xattr subdoc inserts. This should be routed through the standard CAS failure handling, instead of reporting an error.

…m conflict resolver (#4758) * Process attachments from a merge via the sg-replicate custom conflict resolver. * Wait for merged change in test * Use table driven tests for various combinations of attachment merges

* CBG-1053: Adhoc replicationStatus panic fix * Change check condition to wait * Race fix * Address PR comment

* CBG-1035: Add Grafana and update for Hydrogen * Additions

* Prevent falling back to the SGR1 checkpoint for a second time Avoids the case where some documents may not be replicated because of SGR1 checkpoint reuse when both local and remote SGR2 checkpoints have been removed through expiry, replicator reset, bucket flush, rollback, etc. * Write SGR2 checkpoint when performing SGR1 checkpoint upgrade * Improve logging around checkpoint seq retrieval * Switch test to reset replicator instead of deleting checkpoint * Simplify common assertions

…ombstone handling (#4756) * CBG-1028: Customer feedback:Default conflict resolver not behaving consistently * CBG-1028: Remove the methods which became obsolete after introducing table driven tests. * CBG-1028: Assert body value in requireBodyNotEmpty * CBG-1028: Adjust the requiredAdditionalRevs buffer value and incorporate the feedback on unit tests. * CBG-1028: Update the document on rt2 with the specified body values. * CBG-1028: Check for revID instead of docID in TestDefaultConflictResolverWithTombstone

* Fix race for allowed attachment processing Not seen this manifest in the real world, but there's a gap between sending the rev message, and adding the allowed attachments that the client will ask for. If the response gets back to us quick enough, the attachments won't be marked as allowed. * Remove allowed attachments in send error case

…4767) Couchbase Lite already limits the number of in-flight changes batches to prevent Sync Gateway from preparing the revs for batches too far ahead of time and having to keep them in memory before we're able to send them. This commit enforces that limit on the server side for each client, to prevent non-CBL clients from issuing too many changes requests before the revs for those batches have had chance to be sent. We keep this number >1 in order to keep the list of revs supplied for clients to allow for a continous stream of revisions to be available and prevent "batchiness" in the sync.

…uests (#4766) * Perform a retry loop on the active subChanges CAS operation to prevent races when a client is rapidly requesting sequential subChanges requests * Tweak CASRetry timing

Co-authored-by: JRasagneres <jacquesrascagneres@gmail.com>

* CBG-1044: Don't trigger conflict resolution for two tombstones * Add extra test to ensure revID * Change handling and add tests * Remove sleeps * Reworked and added passive SGR2 support * Change clientType type * Address PR comments

These were incorrectly swapped to the push stats as part of the stat format update.

… conflict resolution (#4763) * CBG-1049: Don't tombstone an already deleted revision whilst applying conflict resolution * CBG-1049: Assert expected tombstone revID in waitForTombstone * CBG-1049: Add additional coverage for tombstone handling when remote revision is tombstoned * CBG-1049: Format logging message * Only bypass conflict resolution for incoming tombstones when local is also a tombstone Co-authored-by: Adam Fraser <adam.fraser@couchbase.com>

sgReplicateManager.Stop() now blocks until clusterSubscribe and clusterUpdate goroutines have exited, avoiding potential races during test bucket teardown.

* CBG-1074: Grafana README updates * Address PR comment

* Work on TestSGR2TombstoneConflictHandling * Require no error and nil check * Remove error assert

…oint hash (#4784)

The value used to specify that a replication should start in stopped state is switched to initial_state. TargetState continues to be used internally for state transitions. Target state is set to initial state the first time a replication is added to the cfg.

* PrometheusRules: Added Prometheus directory * Address PR comments * Comment out SGR1 row * Remove extra space

* CBG-1005: Update examples & README * Fix config * Remove old replication example * Remove file * Add a couple extra examples

Sort recent sequences prior to updating syncData.

* CBG-1085 Re-register node when not present on cluster update When a SG node receives a cluster cfg update that doesn't it's own node UUID (e.g. it's been removed by another node due to heartbeat expiry), it should re-register itself to the node set. * Simplify localNodePresent calculation

…on (#4801) * CBG-1087 Preserve document metadata under localWins conflict resolution Fixes issue where document metadata (deleted, attachments) weren't being preserved under localWins conflict resolution. * Comment touch-up

If an import node is temporarily removed from the set of cbgt node definitions (e.g. due to temporary dropped heartbeats), have nodes re-register themselves.

* CBG-1127 Don't treat imported SDK resurrections as conflicts When a tombstoned document is resurrected via the SDK, the metadata history is lost and the document is resurrected as a new revision on a disconnected branch. When replicated via sg-replicate, this revision shouldn't be treated as being in conflict with a tombstoned document (no active leaves). Concurrent updates and deletes to the same revision continue to be treated as conflicts, as this doesn't meet the 'disconnected branch' criteria. * Port test fixes from master * Avoid timing issues for TestActiveReplicatorPushAndPullConflict

…rization header (#4836) (#4839)

…ding blipsync websocket request (#4841)

…rmation (#4914)

…iggered by change for existing channels (#4910)

…ption (#4909) * [2.8.1 Backport] Fix empty webhook payload logging due to buffer-reuse (#4891) * [2.8.1 Backport] Use standard config schema validation for event handler config validation (#4898) * [2.8.1 Backport] Add winning_rev_only option to DocumentChanged event handler config (#4901) * Add String method to EventType for better logging * Add winning_rev_changed event handler * Rework winning rev only option for DocumentChanged event to use generic options supplied to all event handlers/webhooks. * Add missing params in event/webhook functions * Remove old event type * Address PR comments

#4918) (#4919)

…B-32623 (#4920)

…after cfgEventCallback (#4922)

…e set (#4923) Backports CBG-1219 to 2.8.1.

…fill (#4925) Backports CBG-1246 to 2.8.1

…) (#4929)

Backports CBG-1222 to 2.8.1

…rom non-admin requests (#4935) * [2.8.1 Backport] CBG-759 - Add option to hide product version info from non-admin requests (#4928) * CBG-759 - Add option to hide product info from non-admin requests * Scope placeholders in version.go to prevent accidental usage * Have default ProductName const (for use in testing) * Omit product name from Basic auth realm when hide_product_info=true * Hide product version only * Fix test assertions for last change * Address PR comments (add rootResponse struct and build wwwAuthenticateHeader ahead of time) * Fix unused import from cherry-pick

…pty documents (#4884) (#4937) * CBG-1200: Tombstones v2 * Possibly handle pre-6.6 * Alter returned cas values * Additional Test * Added manual test * Remove debug * Test fix * Missed skip on manual test * Address PR comments and import handling * Remove commented out block * Additional cleanup * Change if requirements * Bump manifest * Assert specific error in tests

* CBG-1273: Add more prometheus alert examples * address PR comments

Fixes ISGR: Pull replications incompatible with active-side allow_conflicts=false

Fix for infinite loop when tombstoning a server tombstone.

Commits on May 11, 2020

CBG-769 Replication rebalance (#4573 )

adamcfraser committed May 11, 2020

Configuration menu

View commit details

Copy full SHA for 82489a3

Browse repository at this point

Copy the full SHA

82489a3 View commit details

Browse the repository at this point in the history

Release/2.8.2 #2

Are you sure you want to change the base?

Release/2.8.2 #2

Commits on Mar 19, 2020

Commits on Mar 20, 2020

Commits on Mar 25, 2020

Commits on Mar 27, 2020

Commits on Apr 1, 2020

Commits on Apr 7, 2020

Commits on Apr 8, 2020

Commits on Apr 14, 2020

Commits on Apr 20, 2020

Commits on Apr 27, 2020

Commits on Apr 28, 2020

Commits on May 1, 2020

Commits on May 5, 2020

Commits on May 6, 2020

Commits on May 7, 2020

Commits on May 11, 2020

Commits on May 13, 2020

Commits on May 14, 2020

Commits on May 15, 2020

Commits on May 19, 2020

Commits on May 20, 2020

Commits on May 21, 2020

Commits on May 22, 2020

Commits on May 26, 2020

Commits on May 27, 2020

Commits on May 28, 2020

Commits on May 29, 2020

Commits on Jun 1, 2020

Commits on Jun 2, 2020

Commits on Jun 3, 2020

Commits on Jun 5, 2020

Commits on Jun 8, 2020

Commits on Jun 9, 2020

Commits on Jun 12, 2020

Commits on Jun 15, 2020

Commits on Jun 16, 2020

Commits on Jun 17, 2020

Commits on Jun 19, 2020

Commits on Jun 22, 2020

Commits on Jun 23, 2020

Commits on Jun 24, 2020

Commits on Jun 25, 2020

Commits on Jun 26, 2020

Commits on Jun 29, 2020

Commits on Jun 30, 2020

Commits on Jul 1, 2020

Commits on Jul 2, 2020

Commits on Jul 3, 2020

Commits on Jul 6, 2020

Commits on Jul 7, 2020

Commits on Jul 8, 2020

Commits on Jul 9, 2020

Commits on Jul 10, 2020

Commits on Jul 11, 2020

Commits on Jul 13, 2020

Commits on Jul 14, 2020

Commits on Jul 16, 2020

Commits on Jul 17, 2020

Commits on Jul 20, 2020

Commits on Jul 21, 2020

Commits on Jul 22, 2020

Commits on Jul 23, 2020

Commits on Jul 24, 2020

Commits on Jul 27, 2020

Commits on Jul 29, 2020

Commits on Jul 30, 2020

Commits on Aug 4, 2020

Commits on Aug 5, 2020

Commits on Aug 10, 2020

Commits on Aug 13, 2020

Commits on Aug 14, 2020

Commits on Aug 17, 2020

Commits on Aug 18, 2020

Commits on Aug 19, 2020

Commits on Aug 21, 2020

Commits on Aug 24, 2020

Commits on Aug 25, 2020