Add support for bucket to bucket sync of the latest versions of files #165

mlech-reef · 2020-10-15T12:31:20Z

An example of usage:

$ b2 ls --recursive ML-TestBucket
$ b2 ls --recursive ML-TestBucket2
$ b2 sync ./test b2://ML-TestBucket/test
upload 1.txt
upload 2.txt
upload 3.txt
$ b2 sync b2://ML-TestBucket/test b2://ML-TestBucket2/test
copy 1.txt
copy 2.txt
copy 3.txt
$ b2 ls --recursive ML-TestBucket2
test/1.txt
test/2.txt
test/3.txt

This reverts commit 2f0f77f.

Co-authored-by: Paweł Polewicz <p.polewicz@gmail.com>

bwbeach

This is a review of just the first commit. (It's time for the next meeting, and I want the comments to be visible.)

Overall question: What are the semantics of copying from a hidden file? It looks like the destination file gets deleted. Is that what's expected?

bwbeach · 2020-10-15T19:34:10Z

b2sdk/bucket.py

@@ -687,10 +687,10 @@ def copy(
        """

        copy_source = CopySource(file_id, offset=offset, length=length)
-        if length is None:
+        if not length:


This check is not the same when length is 0. I think this first branch should still happen in that case.

The first branch is using a simple copy manager and the second emerger created by @mzukowski-reef
The problem is that emerger doesn't work when copying 0-length files and actually is not needed for such files

bwbeach · 2020-10-15T19:42:14Z

b2sdk/sync/policy.py

+    """
+
+    def _get_hide_delete_actions(self):
+        for action in super()._get_hide_delete_actions():


What actions might the parent class have that we want to use here? (It looks like the actual base class just returns the empty list.)

If the parent class will start returning something, we'd like this class (and all descendants) to honor that without modification of the inheriting classes

bwbeach · 2020-10-15T19:46:58Z

b2sdk/transfer/outbound/copy_source.py

-        if length is None and offset > 0:
-            raise ValueError('Cannot copy with non zero offset and unknown length')
+        if not length and offset > 0:
+            raise ValueError('Cannot copy with non zero offset and unknown or zero length')


Why? Can't we copy a zero-length file?

We could, but we'd rather not. Uploads are A-class transactions (free), but b2_copy_file are C-class transactions (2500/day for free, then $0.004 per 1,000 calls), so since we are not going to save much bandwidth on transfer, it's better to re-upload the "whole" file.

This is an emerger code. See my previous comment.

bwbeach · 2020-10-15T19:47:55Z

b2sdk/transfer/outbound/copy_source.py

@@ -38,13 +38,15 @@ def is_copy(self):
        return True

    def get_bytes_range(self):
-        if self.length is None:
+        if not self.length:


bwbeach · 2020-10-15T20:50:04Z

I went through the rest of the commits, and they look fine.

I do think there needs to be documentation on what happens when you sync from bucket to bucket. I think it does not copy all the versions of files, just the most recent one. And when in "delete" mode, when the source file is hidden, all versions of the destination file will be deleted.

mlech-reef · 2020-10-15T21:07:49Z

Overall question: What are the semantics of copying from a hidden file? It looks like the destination file gets deleted. Is that what's expected?

By default the destination file is untouched.
If you pass --delete and there is no file on the source, the destination is deleted.
If you pass --delete and the file on the source is hidden, the destination file is untouched.

For me 1 and 2 are ok, but 3 is weird behavior but is not something that I introduced as it works exactly the same way when the destination is a local folder.

ppolewicz · 2020-10-15T21:20:00Z

Please file a separate issue for that so that we may decide whether this is a bug that needs to be fixed for local folders too

mlech-reef · 2020-10-15T21:27:36Z

I do think there needs to be documentation on what happens when you sync from bucket to bucket.

Where would you like to see such documentation? In CLI? I think that overall sync documentation in b2sdk is less informative comparing to CLI sync command description.

ppolewicz · 2020-10-16T22:49:24Z

both, actually.

Also, we'll add sync of all versions in a separare PR

mlech-reef · 2020-10-28T07:26:19Z

I've added a better docstring, updated the sphinx docs slightly, and created tickets for dealing with hidden files and sync of all versions (linked).

mlech-reef · 2020-10-28T07:46:43Z

An update for the CLI: Backblaze/B2_Command_Line_Tool#667

b2sdk/sync/sync.py

doc/source/api/sync.rst

Co-authored-by: Paweł Polewicz <p.polewicz@gmail.com>

ppolewicz · 2020-10-28T15:00:23Z

Codacy says:
test/unit/v1/test_sync.py
unnecessary from enum import Enum

apart from that I think we are ready to merge

mlech-reef · 2020-10-28T15:31:23Z

Codacy says:
test/unit/v1/test_sync.py
unnecessary from enum import Enum

apart from that I think we are ready to merge

Fixed.

mlech-reef and others added 19 commits September 7, 2020 17:46

Add support for bucket to bucket sync

de1f5aa

Add unit tests

5702cb6

Merge remote-tracking branch 'upstream/master' into b2b-sync

66cb55a

Improve unit testing

3eb5189

Fix emerge planner

cabba13

Increase the timeout for server-side copy

8b5328d

Add small improvements in sync

f089bd9

Update unit tests after review

dd3e6ec

Fix reporter for copy, few fixes

e196a02

Fix unit tests for windows

c1b51f9

Fix usage of progress listener

2f0f77f

Merge remote-tracking branch 'upstream/master' into b2b-sync

0df6236

Add better API handling in conftest

ded4499

Change error class in sync

3a3da88

Merge remote-tracking branch 'upstream/master' into b2b-sync

8da4823

Update after PR

9dba8b0

Revert "Fix usage of progress listener"

3d5a0b8

This reverts commit 2f0f77f.

Update b2sdk/sync/action.py

93013c1

Co-authored-by: Paweł Polewicz <p.polewicz@gmail.com>

Merge remote-tracking branch 'upstream/master' into b2b-sync

c96d741

mlech-reef requested review from bwbeach and ppolewicz October 15, 2020 12:31

bwbeach reviewed Oct 15, 2020

View reviewed changes

Update docstring of Synchronizer

5b0fdd2

mlech-reef changed the title ~~Add support for bucket to bucket sync~~ Add support for bucket to bucket sync of the latest versions of files Oct 28, 2020

mlech-reef mentioned this pull request Oct 28, 2020

Bucket to bucket "sync" of all versions #166

Open

mlech-reef mentioned this pull request Oct 28, 2020

How to deal with hidden files during sync #167

Open

mlech-reef mentioned this pull request Oct 28, 2020

Update the docs Backblaze/B2_Command_Line_Tool#667

Merged

mlech-reef requested a review from bwbeach October 28, 2020 07:46

ppolewicz reviewed Oct 28, 2020

View reviewed changes

b2sdk/sync/sync.py Outdated Show resolved Hide resolved

b2sdk/sync/sync.py Outdated Show resolved Hide resolved

b2sdk/sync/sync.py Show resolved Hide resolved

ppolewicz reviewed Oct 28, 2020

View reviewed changes

doc/source/api/sync.rst Outdated Show resolved Hide resolved

mlech-reef and others added 2 commits October 28, 2020 14:57

Update Synchronizer docstring

56c4dcb

Co-authored-by: Paweł Polewicz <p.polewicz@gmail.com>

Update after PR

2d6e955

mlech-reef requested a review from ppolewicz October 28, 2020 14:02

Remove unused import from the tests

a7429eb

ppolewicz approved these changes Nov 2, 2020

View reviewed changes

mlech-reef merged commit e30256e into master Nov 3, 2020

mlech-reef deleted the b2b-sync branch November 3, 2020 06:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for bucket to bucket sync of the latest versions of files #165

Add support for bucket to bucket sync of the latest versions of files #165

mlech-reef commented Oct 15, 2020

bwbeach left a comment

bwbeach Oct 15, 2020

mlech-reef Oct 15, 2020

bwbeach Oct 15, 2020

ppolewicz Oct 15, 2020

bwbeach Oct 15, 2020

ppolewicz Oct 15, 2020

mlech-reef Oct 15, 2020

bwbeach Oct 15, 2020

mlech-reef Oct 15, 2020

bwbeach commented Oct 15, 2020

mlech-reef commented Oct 15, 2020

ppolewicz commented Oct 15, 2020

mlech-reef commented Oct 15, 2020 •

edited

ppolewicz commented Oct 16, 2020

mlech-reef commented Oct 28, 2020

mlech-reef commented Oct 28, 2020

ppolewicz commented Oct 28, 2020

mlech-reef commented Oct 28, 2020

Add support for bucket to bucket sync of the latest versions of files #165

Add support for bucket to bucket sync of the latest versions of files #165

Conversation

mlech-reef commented Oct 15, 2020

bwbeach left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bwbeach commented Oct 15, 2020

mlech-reef commented Oct 15, 2020

ppolewicz commented Oct 15, 2020

mlech-reef commented Oct 15, 2020 • edited

ppolewicz commented Oct 16, 2020

mlech-reef commented Oct 28, 2020

mlech-reef commented Oct 28, 2020

ppolewicz commented Oct 28, 2020

mlech-reef commented Oct 28, 2020

mlech-reef commented Oct 15, 2020 •

edited