3.0.0 sync with GCloud suddenly stopped working - "Failed to synchronize with server"

[jcoffa]

Set up syncing with google cloud by following the steps outlined in man task-sync after upgrading from 2.6.2 -> 3.0.0. Verified that it was working and have been using this solution for syncing for a couple weeks now across 3 devices. Have performed successful syncs on all 3 of these devices over the past couple weeks. Everything was working as expected until today for some reason.

Attempted to run task sync as normal. Got an error message saying "Failed to synchronize with server". Fair enough, I usually forget to run gcloud auth login before attempting a sync anyways. Successfully authenticate in the browser window that opens using the same google account I've always used. Run task sync again, same generic error message "Failed to synchronize with server".

• I have not changed anything about my Taskwarrior config on this device or any other device since the initial sync setup
• Haven't modified anything in Google Cloud
• I only use Google Cloud to sync with Taskwarrior and have no other storage buckets, roles, or service accounts in this project other than the ones I made by following the instructions in man task-sync
• Highly doubt I've synced anywhere near enough to hit any quota limits in my free tier of Google Cloud; my task database is quite small (115 total tasks) and have synced less than a dozen times
• Verified that the JSON keys file is unmodified compared to when my syncing was last working, it's still in my task directory as expected, and the full absolute path + file name matches the value of my sync.gcp.credential_path taskrc property
• Verified that the correct project is selected via gcloud config get-value project
• Verified that the storage bucket still exists; I can still see all the task data in there:

Output of task diag:

task 3.0.0
   Platform: Linux

Compiler
    Version: 13.2.1 20230801
       Caps: +stdc +stdc_hosted +LP64 +c8 +i32 +l64 +vp64 +time_t64
 Compliance: C++17

Build Features
     Commit: 3e41fb604
      CMake: 3.29.1
    libuuid: libuuid + uuid_unparse_lower
 Build type:

Configuration
       File: /home/joseph/.config/task/taskrc (found), 1890 bytes, mode 100644
       Data: /home/joseph/.config/task (found), dir, mode 40755
    Locking: Enabled
         GC: Enabled
    $VISUAL: nvim
Hooks
     System: Enabled
   Location: /home/joseph/.config/task/hooks
     Active: on-add.990.annotate-jira-bot-links    (executable)
             on-modify.990.annotate-jira-bot-links (executable)
             on-modify.timewarrior                 (executable)
   Inactive:

Tests
   Terminal: 276x61
       Dups: Scanned 115 tasks for duplicate UUIDs:
             No duplicates found
 Broken ref: Scanned 115 tasks for broken references:
             No broken references found

At a loss at how to debug this; Taskwarrior only gives me a generic failure message when sync fails.

[djmitche]

This is a bit of a stretch, but can you try with 3.0.2? We fixed some handling of error messages, and hopefully that shows some more information.

That error comes from here, which is adding context to an underlying error. So it's a bummer that only the context, and not the underlying error, appears.

[jcoffa]

Authenticated with gcloud again and then attempted to task sync. Looks like I get the same thing:

$ task sync
Failed to synchronize with server

And here's diagnostics again:

$ task diag

task 3.0.2
   Platform: Linux

Compiler
    Version: 13.2.1 20230801
       Caps: +stdc +stdc_hosted +LP64 +c8 +i32 +l64 +vp64 +time_t64
 Compliance: C++17

Build Features
     Commit: bc86a1e53
      CMake: 3.29.2
    libuuid: libuuid + uuid_unparse_lower
 Build type:

Configuration
       File: /home/joseph/.config/task/taskrc (found), 1890 bytes, mode 100644
       Data: /home/joseph/.config/task (found), dir, mode 40755
    Locking: Enabled
         GC: Enabled
    $VISUAL: nvim
Hooks
     System: Enabled
   Location: /home/joseph/.config/task/hooks
     Active: on-add.990.annotate-jira-bot-links    (executable)
             on-modify.990.annotate-jira-bot-links (executable)
             on-modify.timewarrior                 (executable)
   Inactive:

Tests
   Terminal: 376x103
       Dups: Scanned 115 tasks for duplicate UUIDs:
             No duplicates found
 Broken ref: Scanned 115 tasks for broken references:
             No broken references found

[djmitche]

Are you able to recompile from source? If so, I can make a nice patch to hopefully get better debugging info. I filed #3411 to track the error message problem.

[jcoffa]

I am able to build from source on all my devices, yes.

Also had an important development in this issue. I did a normal update of system packages for all my devices and it fixed my ability to sync on one of them; specifically it's a normal Linux install of Arch Linux. Unsure exactly what caused that to start working, but my best guess is that I was using a pretty old version of gcloud as it looks like I hadn't updated that in almost a year.

Of the other two devices that still don't work, one is a Windows WSL instance of Arch Linux and the other is a completely separate device running a normal Linux install on EndeavourOS. I'm able to build taskwarrior on both of them fine, although when running build/src/task I get the error message Could not find file in CWD, directory of config file or search paths 'dark-256.theme'.. Not sure if I did something wrong while building it? I was just following the steps from the README.

Assuming we can resolve that somehow, then we can do the patch for more debugging information and I'll follow your instructions the same way for both devices if that would be helpful.

[djmitche]

The version of the gcloud CLI is unrelated to how Taskwarrior talk to Google Cloud (it uses a built-in library). So, it's a mystery why it started working!

Here's a patch which should hopefully include the error info:

diff --git taskchampion/taskchampion/src/replica.rs taskchampion/taskchampion/src/replica.rs
index bad6cceba..66d72c3c2 100644
--- taskchampion/taskchampion/src/replica.rs
+++ taskchampion/taskchampion/src/replica.rs
@@ -225,16 +225,17 @@ impl Replica {
     /// indicate it is urgent (snapshot urgency "high").  This allows time for other replicas to
     /// create a snapshot before this one does.
     ///
     /// Set this to true on systems more constrained in CPU, memory, or bandwidth than a typical desktop
     /// system
     pub fn sync(&mut self, server: &mut Box<dyn Server>, avoid_snapshots: bool) -> Result<()> {
         self.taskdb
             .sync(server, avoid_snapshots)
+            .map_err(|e| dbg!(e))
             .context("Failed to synchronize with server")?;
         self.rebuild_working_set(false)
             .context("Failed to rebuild working set after sync")?;
         Ok(())
     }
 
     /// Return undo local operations until the most recent UndoPoint, returning an empty Vec if there are no
     /// local operations to undo.

[jcoffa]

I rebuilt with the patch and still got that Could not find file in CWD, directory of config file or search paths 'dark-256.theme'. error. I was able to copy the dark theme into the current directory and that let it run. Not sure if this information matters.

Anyways here is the results after gcloud auth login successful:

$ ./task sync
[taskchampion/taskchampion/src/replica.rs:233:26] e = OutOfSync
Failed to synchronize with server

[djmitche]

Huh, that error only occurs when communicating with a taskchampion-sync-server instance, not when using gcloud. Did your config change from, or to, gcloud?

[jcoffa]

I set it up using gcloud immediately after upgrading to 3.0.0 and haven't changed any part of my config since then; my taskrc is the same on all my devices and they all have the json file containing the gcloud credentials as mentioned in man task-sync setup instructions. I didn't have any syncing set up before upgrading to 3.0.0.

My taskrc has the sync.gcp.bucket, sync.gcp.credential_path (pointing to that json file from gcloud as mentioned in the setup instructions), and sync.encryption_secret all set up and their values are identical across all my devices. There are no other sync.whatever properties in my taskrc besides those three.

[djmitche]

Oh, I'm sorry - I misread the source. This can happen for cloud syncs, too. This is one of those things that "shouldn't" happen, where the linear history of changes isn't linear anymore.

Is the screenshot above showing all of the objects in your bucket? If not, can you send that whole list? The values are just random UUIDs so don't reveal anything about your data.

Also, please show the output of sqlite3 .task/taskchampion.sqlite3 'select * from sync_meta'

[jcoffa]

Yes that's right; the image in the original issue message is every object in the bucket. It's still accurate as of today.

Also I use the XDG base directory system to organize my configs, which I believe began support in taskwarrior 2.6.0, so I modified your command a bit. Hope that's ok.

$ sqlite3 ~/.config/task/taskchampion.sqlite3 'select * from sync_meta'
base_version|ec3c99a1-95cd-4ab0-9f95-0c61cf249807

task diag correctly displays that my configuration can be found in ~/.config/task as well. Figured I should mention that just in case, to rule out any potential confusion!

Configuration
       File: /home/joseph/.config/task/taskrc (found), 1906 bytes, mode 100644
       Data: /home/joseph/.config/task (found), dir, mode 40755

[djmitche]

Thanks! So what I see in the bucket is two versions (v-...) and one snapshot (s-...). Versions must form a list, with each version named after its "parent" version. So:

• v-00000.. contains the first version (a32e4b..), with null (00000..) parent version
• v-a32e4b.. contains the second version, with the first version (a32e4b..) as parent version
• s-a32e4b.. contains a snapshot of the tasks at the first version.

Your local replica keeps track of the version on which its operations are based, in this case ec3c99a1... Sync begins by looking for any new versions on the server, by looking for a version with parent ec3c99a1... -- finding none. So it assumes that ec3c99a1... is the latest version. This is plausible from the screenshot -- the content of the latest file contains the actual latest version ID, but we can't see that in the screenshot.

Your local replica then packages up its local operations and tries to upload them as a new version at the end of the chain, with parent version ec3c99a1.. -- and the server rejects it for not having the latest version as its parent. Which means that latest file does not contain ec3c99a1..! So, we have three versions in play

Version	Parent	Notes
a32e4b	000000	In bucket
xxxxxx	a32e4b	In bucket
ec3c99	yyyyyy	Not in bucket

Here xxxxxx and yyyyyy are different, and latest in the bucket points to xxxxxx (well, in truth all we know is that it doesn't point to yyyyyy or a32e4b).

The first question is, how did ec3c99 get uploaded and then disappear? Taskwarrior only writes to the sync_meta table once it's successfully uploaded a version and updated the value of latest. And I think the answer is that Taskwarrior performs a periodic cleanup, deleting any versions that aren't on the list. So, it keeps xxxxxx and a32e4b, but deletes ec3c99.

As for why ec3c99 wasn't "on the list": that means that latest didn't point to that version when the cleanup was performed. In theory, that's not possible: that update to latest is "atomic" in the sense that it can't accidentally overwrite a value. So if your replica uploaded ec3c99 with parent a32e4b, then updated latest from a32e4b to ec3c99, that should have either succeeded or -- if some other replica updated latest at the same time, failed with an error causing your local replica to try again.

The object names in the bucket actually contain the version info, so we can at least learn xxxxxx just by making the Name column wider. Can you try that?

Also, if you click the "latest" object and download it, it should be a text file containing the latest version. What is that value?

[jcoffa]

Wow, what a cool explanation! It's interesting to learn the inner workings of how taskwarrior handles its syncing.

Anyways, here's a full view of the names of every object stored in the bucket (+ some extra columns just in case the timestamps end up mattering)

Here is the entire contents of the latest file I downloaded from the bucket:

a32e4bf2440e4d9bb86f5718db6ee103

[djmitche]

Huh, so the table is actually

Version	Parent	Notes
a32e4b	000000	In bucket, latest
ec3c99	a32e4b	In bucket

which means that the replica uploaded ec3c99, but somehow latest didn't get updated, yet it still stored its version in the sync_meta table.

Here's the code that handles updating latest:

taskwarrior/taskchampion/taskchampion/src/server/cloud/server.rs

Lines 371 to 386 in ef9613e

	// Try to compare-and-swap this value into LATEST
	let old_value = latest.map(version_to_bytes);
	let new_value = version_to_bytes(version_id);
	if !self
	.service
	.compare_and_swap(LATEST, old_value, new_value)?
	{
	// Delete the version data, since it was not latest.
	self.service.del(&new_name)?;
	let latest = self.get_latest()?;
	let latest = latest.unwrap_or(Uuid::nil());
	return Ok((
	AddVersionResult::ExpectedParentVersion(latest),
	self.snapshot_urgency()?,
	));
	}

So, if that compare_and_swap operation fails, it deletes the object from the bucket and returns AddVersionResult::ExpectedParentVersion with the value from latest. Here's the code that calls add_version:

taskwarrior/taskchampion/taskchampion/src/taskdb/sync.rs

Lines 82 to 113 in ef9613e

	let (res, snapshot_urgency) = server.add_version(base_version_id, history_segment)?;
	match res {
	AddVersionResult::Ok(new_version_id) => {
	info!("version {:?} received by server", new_version_id);
	txn.set_base_version(new_version_id)?;
	// make a snapshot if the server indicates it is urgent enough
	let base_urgency = if avoid_snapshots {
	SnapshotUrgency::High
	} else {
	SnapshotUrgency::Low
	};
	if snapshot_urgency >= base_urgency {
	let snapshot = snapshot::make_snapshot(txn)?;
	server.add_snapshot(new_version_id, snapshot)?;
	}
	break;
	}
	AddVersionResult::ExpectedParentVersion(parent_version_id) => {
	info!(
	"new version rejected; must be based on {:?}",
	parent_version_id
	);
	if let Some(requested) = requested_parent_version_id {
	if parent_version_id == requested {
	return Err(Error::OutOfSync);
	}
	}
	requested_parent_version_id = Some(parent_version_id);
	}
	}

That updates the base_version in sync_meta if it gets AddVersionResult::Ok, or tries again if it gets AddVersionResult::ExpectedParentVersion.

So presumably there was an attempt to upload new version ec3c99 which

• did upload the version object to the bucket
• did not successfully change the value of latest
• did not delete the version from the bucket
• did update base_version in sync_meta

I think what that means is that compare_and_swap failed to update latest but didn't return false. That's implemented here:

taskwarrior/taskchampion/taskchampion/src/server/cloud/gcp.rs

Lines 105 to 167 in ef9613e

	fn compare_and_swap(
	&mut self,
	name: &[u8],
	existing_value: Option<Vec<u8>>,
	new_value: Vec<u8>,
	) -> Result<bool> {
	let name = String::from_utf8(name.to_vec()).expect("non-UTF8 object name");
	let get_res = self
	.rt
	.block_on(self.client.get_object(&objects::get::GetObjectRequest {
	bucket: self.bucket.clone(),
	object: name.clone(),
	..Default::default()
	}));
	// Determine the object's generation. See https://cloud.google.com/storage/docs/metadata#generation-number
	let generation = if is_http_error(404, &get_res) {
	// If a value was expected, that expectation has not been met.
	if existing_value.is_some() {
	return Ok(false);
	}
	// Generation 0 indicates that the object does not yet exist.
	0
	} else {
	get_res?.generation
	};
	// If the file existed, then verify its contents.
	if generation > 0 {
	let data = self.rt.block_on(self.client.download_object(
	&objects::get::GetObjectRequest {
	bucket: self.bucket.clone(),
	object: name.clone(),
	// Fetch the same generation.
	generation: Some(generation),
	..Default::default()
	},
	&objects::download::Range::default(),
	))?;
	if Some(data) != existing_value {
	return Ok(false);
	}
	}
	// Finally, put the new value with a condition that the generation hasn't changed.
	let upload_type = objects::upload::UploadType::Simple(objects::upload::Media::new(name));
	let upload_res = self.rt.block_on(self.client.upload_object(
	&objects::upload::UploadObjectRequest {
	bucket: self.bucket.clone(),
	if_generation_match: Some(generation),
	..Default::default()
	},
	new_value.to_vec(),
	&upload_type,
	));
	if is_http_error(412, &upload_res) {
	// A 412 indicates the precondition was not satisfied: the given generation
	// is no longer the latest.
	Ok(false)
	} else {
	upload_res?;
	Ok(true)
	}
	}

and looking at all the places Ok(..) occurs there (the return values), those are all false before the call to upload_object, and then false if that failed with a 412 error. The upload_res? statement there should handle any other error as an actual error, propagated back and causing task sync to fail. Looking at the implementation of is_http_error suggests that even the 412 is represented as Result::Err so I think that's a valid assumption.

So, I'm stumped -- what have I missed?

Thanks for your patience, @jcoffa. The good news is, the fix here is pretty easy. Just put ec3c99a195cd4ab09f950c61cf249807 (with no newline!) in the latest file in your bucket and things should start working again.

[jcoffa]

I swapped out the value in latest as you said, deleted the one on gcloud, and uploaded the new copy. Re-auth with gcloud auth login and then a task sync. And something kind of strange happened!

That sync failed ("Failed to synchronize with server"), but it did change some files in the bucket! Here's a screenshot of what that looked like:

I then tried to use the local build that had the patch you sent, and then that one worked!

$ pwd
/home/joseph/taskwarrior/build/src
$ ./task sync
Syncing with GCP bucket <REDACTED>

After this successful sync the gcloud bucket appears completely unchanged. Another task sync with my system install on taskwarrior (i.e. not the local build) reported the same successful message.

Very weird. I'll try with my other devices and report back.

[djmitche]

It looks like it successfully uploaded a new version, and subsequent runs had nothing new to upload so didn't fail. Maybe try modifying a task and running ./task sync (with the local build) again?

[jcoffa]

Damn, got the same error again:

$ ./task add Become a sponsor of Taskwarrior because djmitche has been an absolute legend
Created task 21.

$ ./task sync
[taskchampion/taskchampion/src/replica.rs:233:26] e = OutOfSync
Failed to synchronize with server

gcloud bucket appears unchanged since the last screenshot (made sure to refresh too). I DID NOT end up trying with my other devices like I said I was going to at the end of my last comment, by the way.

[djmitche]

Has the value in latest been updated to 34e878? If not, then at least this is reproducible!

I'll be busy most of the day tomorrow, but will check back after that. It might be worth putting some logging (println!(..) should do) in taskwarrior/taskchampion/taskchampion/src/server/cloud/gcp.rs to see what's happening.

[jcoffa]

Value of latest downloaded from gcloud:

ec3c99a195cd4ab09f950c61cf249807

So looks like it is reproducible at least! I'll see if I can do some print debugging to figure out what's going on. I've never used rust before but I'll try my best!

No worries on any delays by the way, everyone's got a life to live and this is all practically volunteer work anyways! Syncing isn't a huge part of my daily workflow so I personally am not affected by this issue that much.

[jcoffa]

I believe I may have stumbled into something. TL;DR I got syncing working on my main device by manually updating latest a couple times and then adding a new permission to my taskwarrior syncing role on gcloud. There's kind of a lot that went on, including some error messages we've never seen before in this thread, so even though it's long I think the order of events and results might be important so I've included as much info as I could.

---

I'll do my best to explain what I did in the exact order I did them in:

1. I was poking around the local build, adding some print statements and stuff like you suggested
2. I modified the value in the latest file on gcloud like we did before. It used to be ec3c99a195cd4ab09f950c61cf249807 and then I changed it to 34e87843248f4d06ad48d402ad101a52 which is the most recent version out of all the files in my bucket
3. I re-build my local copy of taskwarrior which now has those print statements in it
4. I ran ./task sync using the local build, and here are some of the lines it spit out which I think were important:

After redefinition, name = "latest"
Generation not 0 indicates that the object exists
File existed; verifying its contents
upload_res = Err(Response(ErrorResponse { code: 403, errors: [ErrorResponseItem { domain: "global", location: None, location_type: None, message: "<EMAIL OF TW SYNC ROLE> does not have storage.objects.delete access to the Google Cloud Storage object.", reason: "forbidden" }], message: "<EMAIL OF TW SYNC ROLE> does not have storage.objects.delete access to the Google Cloud Storage object." }))

TL;DR is the message at the end: <EMAIL OF TW SYNC ROLE> does not have storage.objects.delete access to the Google Cloud Storage object.

I double-checked the instructions from man task-sync and referenced the role permissions with the one I set up for my bucket. There are 7 total permissions that the setup instructions mention, and none of them are storage.objects.delete (closest is storage.objects.update I guess?). I verified that all 7 roles in man task-sync are present in my taskwarrior role, and no unspecified permissions are applied (it's an exact match with what's in the man page). I added this 8th permission to the role and then attempted a task sync again.

And it worked!

5. I modified a random task so that I would have something to try syncing with
6. I run ./task sync again with the local copy

$ ./task sync
[taskchampion/taskchampion/src/replica.rs:233:26] e = OutOfSync
Failed to synchronize with server

Same issue as before; I'd like to highlight that I didn't manually swap out the value in latest before trying this 2nd sync. Also interesting that none of my print statements fired; it looks like the compare_and_swap function (the one you pointed me to in taskwarrior/taskchampion/taskchampion/src/server/cloud/gcp.rs) was never even called in this case.

7. I manually edit latest again and re-attempt a sync, which as expected is successful:

And here's my full print output just in case there's something in here that's even remotely helpful (although it's kind of a mess):

$ ./task sync

Start of function 'compare_and_swap' in gcp.rs, parameters:
        name = "latest"
        existing_value = "40270595af5e48acb2ed4ac2f529092a"
        new_value = "222b8a934e1f4ede92fed9e48217b786"

After redef, name = "latest"
Generation not 0 indicates that the object does exist
File existed; verifying its contents
Put the new value with a condition that the generation hasn't changed
upload_type = Simple(Media { name: "latest", content_type: "application/octet-stream", content_length: None })
upload_res = Ok(Object { self_link: "https://www.googleapis.com/storage/v1/b/<MY GCP BUCKET>/o/latest", media_link: "https://storage.googleapis.com/download/storage/v1/b/<MY GCP BUCKET>/o/latest?generation=1714703467735282&alt=media", content_encoding: None, content_disposition: None, cache_control: None, acl: None, content_language: None, metageneration: 1, time_deleted: None, content_type: Some("application/octet-stream"), size: 32, time_created: Some(2024-05-03 2:31:07.822 +00:00:00), crc32c: Some("x9Kpdg=="), md5_hash: Some("qE7MBXed1F+5gtolbWenXA=="), etag: "CPLR25i48IUDEAE=", updated: Some(2024-05-03 2:31:07.822 +00:00:00), storage_class: Some("STANDARD"), kms_key_name: None, time_storage_class_updated: Some(2024-05-03 2:31:07.822 +00:00:00), temporary_hold: None, retention_expiration_time: None, metadata: None, event_based_hold: None, name: "latest", id: "<MY GCP BUCKET>/latest/1714703467735282", bucket: "<MY GCP BUCKET>", generation: 1714703467735282, owner: None, customer_encryption: None, custom_time: None })
No HTTP error; returning OK(true)
Syncing with GCP bucket <MY GCP BUCKET>

(that last line "Syncing with GCP bucket" is simply the success message when taskwarrior itself performs a sync without error; that's not one of my print statements)

8. I modify a task and re-attempt a sync without manually modifying latest. This is also successful! It contains no noteworthy difference in output other than the new v- file so I won't bother putting another screenshot or print output here
9. I add a brand new task instead of modifying an existing one and re-attempt a sync without modifying latest. ALSO success!
10. I delete the new task I made in step 9 and sync without modifying latest. ANOTHER success!

Here is a screenshot of what my bucket looks like now:

---

So... things appear to be working now? At least for my main device I use taskwarrior on. I haven't attempted syncs on other devices as I figured I'd let you digest this absolute wall of text (sorry!) before trying to do anything else without guidance; for fear of messing something up and making it harder to debug.

[jcoffa]

Something I just thought of while sleeping on this issue and thinking about why latest wasn't getting updated and now it is all of a sudden. Could it be that, instead of modifying the contents of latest (using the storage.objects.update permission, perhaps) it is instead attempting to delete latest entirely and then upload a new latest that has the new/updated version ID in it? That could possibly explain why it started working after adding that storage.objects.delete permission.

[djmitche]

Nice work! I set up a service account to replicate this, without the storage.objects.delete permission, and sure enough it works for a few syncs but then stops when it comes time to overwrite latest.

However, I don't reproduce exactly the error you've seen -- a newer version in sync_meta than in latest. Instead, I see

⸩ task sync
Failed to synchronize with server: taskwarrior-testing@taskwarrior-testing.iam.gserviceaccount.com does not have storage.objects.delete access to the Google Cloud Storage object.

which before #3411 would have just said "Failed to synchronize with server". My bucket has

latest
salt
v-00000000000000000000000000000000-faeb7fdd9677421f8adbfae4f2565142 
v-faeb7fdd9677421f8adbfae4f2565142-fb1e1171d5f347d9a4ee05dd6860ba77 

so two versions: faeb7f followed by fb1e11. The latest file contains faeb7f.., so it wasn't updated. And select * from sync_meta gives the same, so it also wasn't updated. So, all of this failed in an expected way that wouldn't lead to OutOfSync. However, if I run task sync again:

⸩ task sync
Syncing with GCP bucket djmitche-taskwarrior-devel

It works?! And even more bizarre, sync_meta gets updated to fb1e11...

Ah, I see what's happening here! On the last (successful) run of task sync, the replica begins by trying to update to the latest version from the server. So, it looks for the child version of faeb7f and finds fb1e11, and applies that locally. But it shouldn't - that version is not on the chain from latest!

So, there are two bugs here now.

• get_child_version must check that the version it returns is reachable from latest taskchampion#387 - the child-version search must check against latest
• Include storage.objects.delete permission for GCP #3441 - updating the manpage to include the required permission.

Any chance you want to make a PR for the second one?

Thanks so much for your patience tracking this down. That's three bugs found and soon to be fixed in one issue!

[jcoffa]

Yeah I can make a PR for the 2nd one, it'd be my pleasure :)

And thank YOU for your patience as well! No chance could I have fixed my syncing issues by myself.