CBG-3736: cv only delta sync #6661

gregns1 · 2024-02-02T13:13:40Z

Store CV (In CBL format) on the DocumentDelta (this needs to be stored alongside the ToRevID on the delta for future backwards compatibility reasons)
DocumentDelta needs now also stores history in CBL string format
oldRevisionKey: changes to have cv instead of revID here, to adfdress isseus with key length I have made is to CV is hased befgore being persisted to the bucket
When subprotocol version is below v4 and delta sync is requested, fall back to sending full revision
GetDelta: fromRevision switch the fetch from rev cache for this revision from using revID to CV, If delta is found on fromRevision we now check if this delta correlates to the top the CV we want the delta for, toRevision: switch the fetch from rev cache to use CV rather than revID
UpdateDelta: has a CV version entitled UpdateDeltaCV which uses the CV code path to update a delata on revcache
newRevCacheDelta: grabs HLV history off documentRevision and store in onto RevisionDelta for ease of sending to CBL inside sendDelta function
processRev: Switch to get by CV from revcache for deltaSrcRev
backupRevisionJSON: had to move the handling for backing up sdk writes as it was using the new rev to do this. From where this code is called we haven't yet macro expanded the version thus cannot use the new CV to back up here. Solution was to move the code to where we update hlv post doc update.
Some test changes to fetch the backup revisions by the new format, some tets are just too difficult to work around in this sense till backwards compatibility is done, thus filed a ticket for this work and liked in the skip comments of these tests

Pre-review checklist

Removed debug logging (fmt.Print, log.Print, ...)
Logging sensitive data? Make sure it's tagged (e.g. base.UD(docID), base.MD(dbName))
Updated relevant information in the API specifications (such as endpoint descriptions, schemas, ...) in docs/api

Integration Tests

GSI=true,xattrs=true https://jenkins.sgwdev.com/job/SyncGateway-Integration/2397/

* CBG-3209: changes for retreival of a doc from the rev cache via CV with backwards compatability in mind * fix failing test, add commnets * fix lint * updated to address comments * rebase chnages needed * updated to tests that call Get on revision cache * updates based of new direction with PR + addressing comments * updated to fix panic * updated to fix another panic * address comments * updates based off commnets * remove commnented out line * updates to skip test relying on import and update PutExistingRev doc update type to update HLV * updates to remove code adding rev id to value inside addToRevMapPostLoad. Added code to assign this inside value.store * remove redundent code

* CBG-3210: Updating HLV on Put And PutExistingRev (#6366) * CBG-3209: Add cv index and retrieval for revision cache (#6491) * CBG-3209: changes for retreival of a doc from the rev cache via CV with backwards compatability in mind * fix failing test, add commnets * fix lint * updated to address comments * rebase chnages needed * updated to tests that call Get on revision cache * updates based of new direction with PR + addressing comments * updated to fix panic * updated to fix another panic * address comments * updates based off commnets * remove commnented out line * updates to skip test relying on import and update PutExistingRev doc update type to update HLV * updates to remove code adding rev id to value inside addToRevMapPostLoad. Added code to assign this inside value.store * remove redundent code * Add support for PutExistingCurrentVersion * updated to remove function not used anymore * remove duplicated code from dev time * fix linter errors + add assertions on body of doc update * address commnets * updates to add further test cases for AddNewerVersions function + fix some incorrect logic * updates to chnage helper function for creation of doc for tests. Also adress further comments * lint error * address comments, add new merge function for merge versions when hlv is in conflict. * updates to remove test case and test * remove unused function * rebase * missed current version name change * more missing updates to name changes

* CBG-3255: Add current version to log entry for population on the channel cache. Pre-requisite for my work on adding CV to change entries. Only adds CV to log entry from docs seen over DCP at this time pending work on channel cache backfill * add comments and protect against panic in channel cache population * add more commnets * updated to move test and few lines populating log entry

#6590) * CBG-3607: disable the ability to set shared_bucket_access to false. In future we will probably look to fully remove the config param but for now this will protect against panics * remove comment slashes + add test comments * updates off review + failing test in integration test run * missed test update * lint error fix * add skip for lint?

* CBG-3356: add CV to change entry, test that it corretcly populates when calling for changes. Tests need to activate a channel cache as backfill for channel cache not yet implemented * updates to fix failing tests. Added cv to version type returned by Putting a doc and deleting a doc to make testing easier * minor changes * updates after rebase * fix for test failure * updates from rebase * updated comment * changes in response to commmets * updates to fix test failures * rebase + lint skip * updates to update the doc id changes test I have to actually test the codepath

…nts. (#6614) * - Rename `SourceAndVersion` to just `Version`, and rename `SourceAndVersion.Version` to `Value`. - Add/Improve comments. * Update db/hybrid_logical_vector.go

* CBG-3212: add api to fetch a document by its CV value * test fix * rebased SourceAndVersion -> Version rename * Update currentRevChannels on CV revcache load and doc.updateChannels * fix spelling * Remove currentRevChannels * Move common GetRev/GetCV work into documentRevisionForRequest function * Pass revision.RevID into authorizeUserForChannels * Update db/crud.go Co-authored-by: Tor Colvin <tor.colvin@couchbase.com> --------- Co-authored-by: Ben Brooks <ben.brooks@couchbase.com> Co-authored-by: Tor Colvin <tor.colvin@couchbase.com>

* `teh` -> `the` * Remove typo'd TODO by implementing assertion * `comapre` -> `compare` * `exsiting` -> `existing`

* CBG-3210: Updating HLV on Put And PutExistingRev (#6366) * CBG-3209: Add cv index and retrieval for revision cache (#6491) * CBG-3209: changes for retreival of a doc from the rev cache via CV with backwards compatability in mind * fix failing test, add commnets * fix lint * updated to address comments * rebase chnages needed * updated to tests that call Get on revision cache * updates based of new direction with PR + addressing comments * updated to fix panic * updated to fix another panic * address comments * updates based off commnets * remove commnented out line * updates to skip test relying on import and update PutExistingRev doc update type to update HLV * updates to remove code adding rev id to value inside addToRevMapPostLoad. Added code to assign this inside value.store * remove redundent code * CBG-3503 Update HLV on import (#6572) * Beryllium: Rename `SourceAndVersion` to `Version` / Improve HLV comments. (#6614) * - Rename `SourceAndVersion` to just `Version`, and rename `SourceAndVersion.Version` to `Value`. - Add/Improve comments. * Update db/hybrid_logical_vector.go * CBG-3254: pull replication for v4 protocol * updates to the btcRunner * tidy of comments * fix linter * updates to change way hlv is represented on doc revision in rev cache. Also added temporary methods to use db operation in btcRunner test to put and update docs * updates to fix linters and remove unused function * more lint stuff * address commnets after rebase * updates to fix failing test and comments * updates to address comments --------- Co-authored-by: Adam Fraser <adam.fraser@couchbase.com> Co-authored-by: Ben Brooks <ben.brooks@couchbase.com>

* CBG-3213 Version support for channel removals Adds cv (source and version) to removals in _sync.channels (ChannelMap). Uses RevAndVersion to support query (the same approached used for _sync.rev). Required moving RevAndVersion to channels package for usage within ChannelMap. Changes in crud.go required to support the case where the removal version needs to be set via macro expansion. * Use standard function to update testBackingStore document channels

…6655) * CBG-3719: change in memory format of hlv to match XDCR/CBL format * updates after rebase * updates to fix missed type swap in channels package * update to add encoded bucket UUID to db contect, this allows us to avoid overhead associated with encoding bucketUUID each time a HLV is updated * updates after rebase * updates based off review

* CBG-3788 Support HLV operations in BlipTesterClient Switches the BlipTesterCollectionClient to maintain client HLV and (linear) revtree per document. Switches the docs struct to a map of a new BlipTesterDoc struct, instead of a map of revs per document. BlipTesterDoc still maintains a history of all rev messages received (revMessageHistory) to support test evaluation of received messages, but also defines a linear revTreeId history or an HLV (depending on protocol enabled for the test). Includes a refactor of revID to revTreeID in RevAndVersion, as a step toward standardizing ‘revID’ as the generic property used during replication (which can be currentRev or cv), and revTreeID as a traditional revtree revision ID. * Fixes based on PR review

* CBG-3255 Push replication support for HLV Adds push replication support for HLV clients. Delta sync and attachments are not yet supported (pending CBG-3736, CBG-3797). On proposeChanges, checks whether the incoming CV and parent version represent a new document, known version, valid update, or conflict. Uses the same handling as revTreeID (conflict if parent version isn’t the server’s current version), with the additional non-conflict case where the incoming CV and server CV share the same source and the incoming CV is a newer version. For the incoming rev, detects conflict based on the incoming cv (based on the implicit hierarchy in an HLV, where cv > pv > mv). Includes some test helpers to support writing tests with simplified versions (e.g. 1@abc) while still asserting for encoded source and version. * Test fixes * Fixes/cleanup based on PR review --------- Co-authored-by: Gregory Newman-Smith <gregory.newmansmith@couchbase.com>

HLV clients don't consider revpos, and evaluate whether they need to request an attachment based on the existing set of attachments on the document. SGW still needs to persist revpos into _attachments to support revtree clients. For new attachments added by HLV client, revpos is set to the generation of SGW's computed revTreeID for the incoming revision. Co-authored-by: Gregory Newman-Smith <gregory.newmansmith@couchbase.com>

- Add new parameter to updateAndReturnDoc call from PutExistingCurrentVersion - Add nil check in rev handling in unmarshal JSON

gregns1 self-assigned this Feb 2, 2024

gregns1 requested a review from bbrks February 5, 2024 09:53

gregns1 assigned bbrks and unassigned gregns1 Feb 5, 2024

gregns1 force-pushed the CBG-3736 branch from 25c81ed to d2c0623 Compare February 26, 2024 10:30

gregns1 force-pushed the CBG-3736 branch 2 times, most recently from c996631 to 5a7962a Compare March 13, 2024 17:09

bbrks changed the base branch from beryllium to release/anemone March 20, 2024 12:03

bbrks force-pushed the release/anemone branch 2 times, most recently from 555a093 to a25664c Compare April 16, 2024 14:46

bbrks force-pushed the release/anemone branch from a25664c to 37d3cb5 Compare April 16, 2024 16:28

bbrks force-pushed the CBG-3736 branch from e763785 to d00e728 Compare April 16, 2024 16:30

bbrks force-pushed the CBG-3736 branch from d00e728 to 2bcf6be Compare May 1, 2024 17:20

adamcfraser force-pushed the release/anemone branch 2 times, most recently from f0850fd to 6ae5004 Compare May 10, 2024 18:01

gregns1 and others added 15 commits May 14, 2024 13:26

Rebase anemone on main

dfba369

4.0: Bump SG API version (#6578)

45c9a3d

CBG-3503 Update HLV on import (#6572)

52bc6af

Beryllium: Rename SourceAndVersion to Version / Improve HLV comme…

9746cf7

…nts. (#6614) * - Rename `SourceAndVersion` to just `Version`, and rename `SourceAndVersion.Version` to `Value`. - Add/Improve comments. * Update db/hybrid_logical_vector.go

Rebase on main

540a0e2

beryllium: fix misspell typos (#6648)

8504b4f

* `teh` -> `the` * Remove typo'd TODO by implementing assertion * `comapre` -> `compare` * `exsiting` -> `existing`

adamcfraser and others added 7 commits May 14, 2024 13:29

CBG-3808: vrs -> ver to match XDCR format (#6723)

2129bd3

Ignore lint error for now - filed ticket to fix (#6762)

9726dfd

Anemone fixes after rebase on main

72340cf

- Add new parameter to updateAndReturnDoc call from PutExistingCurrentVersion - Add nil check in rev handling in unmarshal JSON

CBG-3764-anemone Correct error type checking (#6810)

99bdd0d

adamcfraser force-pushed the release/anemone branch from 6ae5004 to 99bdd0d Compare May 14, 2024 20:37

adamcfraser and others added 13 commits May 15, 2024 13:17

CBG-3877 Persist HLV to _vv xattr (#6807)

3e502a3

CBG-3736: cv only delta sync changes

c10f01b

updates to back up revs for delta sync

32a0613

fix after rebase

951ece9

updates to fix lint

f532626

more lint

e15735c

linter again

27158f6

fix for intermittent test failure

1fa2d85

enable the delta sync tests that were disabled due to push replication

8800530

remove commnet

74a4ce5

mistake with last push

ca4fb0d

fix for incorrect fetch from the rev cache for delta src

ca7fd48

fix for lint

cffd026

bbrks force-pushed the CBG-3736 branch from 2bcf6be to cffd026 Compare May 16, 2024 11:46

torcolvin force-pushed the release/anemone branch from 3e502a3 to 72b0624 Compare May 21, 2024 15:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CBG-3736: cv only delta sync #6661

CBG-3736: cv only delta sync #6661

gregns1 commented Feb 2, 2024 •

edited by bbrks

CBG-3736: cv only delta sync #6661

Are you sure you want to change the base?

CBG-3736: cv only delta sync #6661

Conversation

gregns1 commented Feb 2, 2024 • edited by bbrks

Pre-review checklist

Integration Tests

gregns1 commented Feb 2, 2024 •

edited by bbrks