Synchronization support tomboy-ng <-> Nextcloud Notes

[monperrus]

Problem to solve: having a fully working setup with Linux desktop program / server side code / Android app for taking and synchronizing notes.

Tomdroid is dead, and one of the best Android app for self-hosted note taking today is Nextcloud + Nextcloud notes. So a potential solution is :

• Linux desktop program: tomboy-ng
• server side code: Nextcloud notes server with REST API
• Android app: Nextcloud notes app

Any hint on how to proceed would be appreciated.

[davidbannon]

Sigh, it has to be JSON doesn't it ? Hope you are better at JSON than me.

I'll have a read of that API but it looks like it might be a lot better documented than the Tomboy model was. Could be good ...

Davo

[monperrus]

Yes, this is JSON. I agree that the documentation of https://github.com/nextcloud/notes/blob/master/docs/api/v1.md is nice!

Looking forward to your thoughts.

[davidbannon]

OK, had a quick and very superficial read.

The nextcloud note attributes seem to map fairly well to tomboy, only exception is favourite, maybe thats "open on start" but I don't think so. So, we'd need to have some lookup table or modify the tomboy note format, both ugly.
The note ID is not going to be a 1:1 transfer, tomboy uses GUID and nextcloud has just a int. So, again, some mapping. Maybe thats version 2.0 ?
The Title and Timestamp looks easy.

The API looks like it it covers what we need, simple Gets and Puts.

The tomboy model assigns a Sync ID to make sure it is talking to the right repository, maybe we can overlay the username for that. You would assume a nextcloud would have only one note repository ?

I think I would take the same approach here that I took with Tomdroid, I would not consider Nextcloud as just another note in the sync model. Nextcloud does not, and cannot easily, keep the necessary metadata. So we need to keep it locally, per connection. We end up with one tomboy-ng node that syncs to Nextcloud in a one to one sync. That tomboy-ng node can of course also sync to other tomboy nodes using its normal sync model.

Just to be clear what I mean, so you can tell me if I am being silly -

Suppose I have a Linux laptop, a Linux desktop, occasionally reboot my laptop into Windows. And an old Mac I use to run some tests. All of these systems are part of the 'normal' tomboy file sync.

But on my Laptop I can also do a 1:1 sync to Nextcloud so that my Phone and Tablet can also see my notes.

I cannot do that 1:1 sync from any other system than the Laptop but because the Laptop auto syncs to my file repo and so do the other systems, its all in sync.

We would translate the Tomboy markup to md at sync, again, we'd loose some data but I think it would be acceptable.

What do you think of that model ?

I believe its pretty easy to implement.

Another approach might be to make a Nextcloud note that holds the meta data. As long as user did not mess with it, it would work, I think. Risky IMHO.

Davo

[davidbannon]

monperrus, have a had a better read and a think about the issues. A bit rambling but enough for you to see that there is issues we have to solve. One of the biggest problems with syncing is knowing which files have changed and which have not. Thats certainly the issue here.

If we download a list of notes on nextcloud, we get a noteID and a timestamp. Neither are particularly tomboy compatible, the noteID is an integer, the time stamp is a Unix one, so only one second resolution, Tomboy uses seven decimal places of a second. That's actually important, the chances of a note being edited at two places in exactly the same second is small, in the same 100 nS is negligible. But we will have to work to the second !

The Nextcloud noteID is an integer, presumably starting at 1 (or 0). So, if you have to merge two sets of notes, or move from one NextCloud to another, we cannot guarantee that the noteID is unique. Tomboy uses a GUID, unique (enough). Once a note has its GUID, Tomboy does not care about the actual format so we could, perhaps clear the last six digits and put the Nextcloud number in there, with some marker to show what we have done.

Usual Tomboy file name -

e8cb904d-2c4b-4ab7-8d93-3d353fcbdda0.note

A note derived from NextCloud, say, 177 -

e8cb904d-2c4b-4ab7-8d93-3d353-000177.note

That now allows us to recognise this note as one we should compare to the Nextcloud note number 177.

However, when I send a Tomboy created note up, say -

627e0bde-99ca-4377-9c73-92dcca942680.note

Next cloud will change its ID to, say, 178 and at the next sync, we have no way to tell we need to compare it to the local 627e0....
If the POST/Note API returned the newly assigned Nextcloud ID then we could maintain a map of the IDs used at either end. But we'd need a process to remove entries relating to deleted notes etc. Do-able but messy.

The Tomboy sync model stamps a SyncRevision number on every transaction and a list of all the current notes and their latest revision. That works if the "remote" is happy to maintain that data for us. I see no option to do that on NextCloud Notes.

In my Tomdroid sync, I rely a lot more on the LastChangeDate, I read the last change dates of all notes in Tomdroid and bring it back to tomboy-ng to compare. We can do that but only to one second resolution....

Deletions on NextCloud
(assuming we solve the above)
If we receive a list of notes from the Nextcloud server and we notice we have one locally that is no longer there on Nextcloud, we assume its been deleted up there and should be deleted here. Thats OK.

Deletions on Tomboy
Notes deleted locally that have been previously synced have their IDs recorded until the next sync, then the NextCloud is told to delete them. There is an API for that purpose.

Need for ServerID
The deletion models require a guarantee that we are talking to a system we have synced to before. Else we get a real mess on our hands. So, both the Tomboy and Tomdroid sync models stamp an ID when a new sync repo is established, it asks about that ID when making any subsequent connections, if they don't match, do not proceed ! That ID, or some sort of token, really needs to be stored at both ends !

Note Title is not useful in identifying notes, user's change the title all the time.

A lot of this could be solved if we could find a way to store some metadata on Nextcloud. Could we, for example, just have another document there, created by the sync process ? ( Or, better still, is there any possibility we can update that document when a note is changed/deleted on Nextcloud ? I understand, a lot harder.) But just a nextcloud doc we can upload and download outside of the note system is, perhaps, what Nextcloud is all about ?

Sorry to be so picking, I am concentrating on the problems so, once we have solved them, its full steam straight ahead !

Note: if I was to redesign a new Tomboy schema, I would include a new GUID or maybe a revision number at every edit, that way I'd always be able to tell if the remote one was the same as the local.

[monperrus]

Thanks a lot for the comprehensive analysis. The metadata problem is related to the file format. If we store the note in Markdown on Nextcloud, we can use a YAML frontpage to store the metadata.

[davidbannon]

OK monperrus, that sounds hopeful !

Are you suggesting you could return a reproducable UUID (that you have allocated in Nextcloud) when the GET /NOTES call is made ? If so, it would all become quite easy !

I am currently concentrating on establishing an Ubuntu PPA and possibly getting tomboy-ng into debian but when thats sorted, I will build a nextcloud set up and have a play ....

Davo

[monperrus]

I am currently concentrating on establishing an Ubuntu PPA and possibly getting tomboy-ng into debian

Super nice!

but when thats sorted, I will build a nextcloud set up and have a play ....

Excellent, looking forward to it.

[monperrus]

Are you suggesting you could return a reproducable UUID (that you have allocated in Nextcloud) when the GET /NOTES call is made ?

There is already one, the id field:

[
    {
        "id": 76,
        "modified": 1376753464,
        "title": "New note",
        "category": "sub-directory",
        "content": "New note\n and something more",
        "favorite": false
    }, // etc
]

Am I missing something?

[davidbannon]

"id": 76,

The problem is that the ID is not unique. If someone syncs to their Nextcloud and later re-installs nextcloud or syncs to another install for some reason (and people do do that all the time !) they come across that same number again, because its got the same ID, its assumed its the same note. So, it overwrites it. Every Nextcloud Notes install will have a number 1, a number 2 and so on.

Truth is, Nextcloud Notes needs to assign a unique ID to every note even if it does not sync with Tomboy !

Davo

[monperrus]

Do you mean not unique per user? per server?

[davidbannon]

I mean it needs to be unique.
Unique per server is not enough, suppose someone moves to another server that already has, perhaps one note.
Unique per user is not enough, what happens when two users swap notes? Users send notes between them, there are, perhaps ten Tomboy format notes in the tomboy-ng install kit.

If all you want to do is 'import' or 'export' from one platform to another, then, its easy, no need to worry about unique, but if we are going to track them over some sync, edit, sync, delete cycles, we have to be sure we are talking about the same note.

Davo

[monperrus]

I understand. What about putting the unique id as a metadata?

[davidbannon]

Nine days ago I said

"A lot of this could be solved if we could find a way to store some metadata on Nextcloud. Could we, for example, just have another document there, created by the sync process ? ( Or, better still, is there any possibility we can update that document when a note is changed/deleted on Nextcloud ? I understand, a lot harder.) But just a nextcloud doc we can upload and download outside of the note system is, perhaps, what Nextcloud is all about ?"

Davo

[monperrus]

Hi Davo,

The solution I proposed is to store the note in Markdown on Nextcloud, and to use a YAML front matter to store the metadata.

[davidbannon]

You did indeed. And I thought that was a great idea. But you followed it up by saying -

There is already one, the id field:
...
"id": 76,

Now, the way I understand it, everyone with more than 76 notes in their repository will have a note with ID = 76. So, IMHO, thats not unique. If you can replace the ID field with a UUID/GUID then it would be unique, that would be cool, we would be underway. But if the ID is allocated by ++(previous_ID) then we do not have a unique ID.

Just about every programing language these days has a library function to generate a UUID, its a reasonably clearly defined way to produce a (almost) guaranteed unique string.

I hope this is clear, we seem to be going around in circles here I am afraid.

Davo

[monperrus]

AFAIU, we can have two separate identifiers: * the "physical" id of the REST api, used in the URLs, it is not unique * the "logical" id of the note, generated on our own with a uuid library, stored in the front matter Makes more sense?

[davidbannon]

That sounds great ! I assume you can include the "logical id" in the initial REST response, along with the physical ID ? That would work. I don't necessarily need to use the logical ID when manipulating notes, as long as I have seen a confirmation that the mapping between them has not changed.

I'll need to do some thinking about how we handle deleted notes. When I see a LID (Logical ID in your terms) in Nextcloud, that I don't have locally, its either a new one that I have not seen before (action=download) or one I have seen before but decided to delete locally (action=remote delete). With my Tomdroid sync hack, I keep a one to one sync (and a list of deleted notes) from one instance of tomboy-ng to one Tomdroid phone. I think that may be better than making the Nextcloud Notes one participant in a Tomboy file sync but it will need some work to be sure.

Anyway, if we have a transport model worked out, a sync policy can be worked out too !

Davo

[monperrus]

I assume you can include the "logical id" in the initial REST response, along with the physical ID ? I'm not a Nextcloud developer, I don't know whether we would be successful in pushing this change. Is that required for our use case?

[davidbannon]

Yes, I think its necessary. Maybe we might get away with a having a (nextcloud?) document that maps UUID to NCNoteID but it would be fragile. Lots of things that could break it that we may not think of.

For example, and a test you can do perhaps ? Make a new note, its gets assigned, say, number 44, can you see its number ? Delete it immediately and create a another one. Does it get ID 44 or ID 45 ? If it gets 44, no, we cannot work like that. If it gets 45, maybe. I'll have to do some work to map out the risks.....

Davo

[monperrus]

Does it get ID 44 or ID 45 ?

I don't know.

If it gets 44, no, we cannot work like that

I am not sure, thanks to uuid metadata

• Make a new note with uuid metadata xyz, it gets assigned, say, number 44
• Delete it immediately
• Create a another one with uuid metadata abc. Even if it has URL ID 44 again, we know that the uuid is abc, so it is a different note. The UUID metadata would be the reference not the server id.

[davidbannon]

Another long answer I am afraid. I don't think its hopeless but there are some serious downsides.

I don't think you can associate a note with a particular UUID if we cannot add it to the note's own meta data. Maybe I am wrong ? You understand what happens inside Nextcloud far better than I do so, perhaps you have a plan that works ? Can you explain it please ?

In my mind, if the UUID is not part of a note's metadata then we have to have some sort of lookup table. -
...
ID=44 UUID=xyz
...

However, if the act of deleting that note and making a new one that uses the same ID does not automatically update the table, then we cannot tell it has happened, it looks like an edit. So, the process is perhaps -

• We trigger a sync between tomboy-ng and Nextcloud Notes
• We use the NCNote api call, get a list of notes including one with ID=44.
• We get the content that was in the first note to be called ID=44.
• We lookup our table, find ID=44 in there and its associated with UUID=xyz
• We then can update, one way or another, the Tomboy note 'xyz'. Good.
• Next, the Nextcloud user does as you say, deletes the note and create a new one, also assigned ID=44
• *** critical *** Does something update the lookup table here ? How or where is the new UUID of 'abc' you mention assigned ?
• We trigger a sync between tomboy-ng and Nextcloud Notes
• We use the NCNote api call, get a list of notes including one with ID=44.
• We lookup our table, find ID=44 in there and its associated with UUID=xyz (but its a new note, user deleted one 44)
• tomboy-ng now overwrites the first ID=44 note with the new content and keeps its old UUID. For all intents and purposes, its the same note, it just has new content and a new title.

Maybe thats an acceptable departure from the Tomboy model. It means for example I cannot backup the old, deleted content as per Tomboy practice. If there is a sync clash, in tomboy-ng I show a diff of the two notes, thats going to be a messy diff to read where everything has changed. If someone does a restore from snapshot, again unexpected results because we are reusing the original UUID, 'xzy'. Note creation date will be the time when the very first ID=44 was seen by tomboy-ng, that ID may well be reused many, many times with new notes but each one will still show the quite incorrect note creation date of the very first note to be assigned that ID.

I suppose we could take the view that a sync with NCNotes is not a normal Tomboy sync, as long as the end user understands there are times when unexpected results will occur.

I understand you don't think there is any prospect of getting the NCNotes people to add a UUID and/or a creation date ?

Davo

[monperrus]

Hi Davo,

Here is my understanding of the scenario (in bold my explanations)

• We trigger a sync between tomboy-ng and Nextcloud Notes
• We use the NCNote api call, get a list of notes including one with ID=44.
• We get the content that was in the first note to be called ID=44.
• We lookup our table, find ID=44 in there and its associated with UUID=xyz
• We then can update, one way or another, the Tomboy note 'xyz'. Good.
• Next, the Nextcloud user deletes the note and create a new one, also assigned ID=44
  • If the user creates the note on Nextcloud directly, there is no UUID. Because there is no UUID, tomboy-ng understands that 1) a new UUID must created 2) it must update its internal lookup table for ID=44
  • If the user creates the new note on tomboy-ng, tomboy-ng knows that how to update its internal lookup table for ID=44
• We trigger a sync between tomboy-ng and Nextcloud Notes
• We use the NCNote api call, get a list of notes including one with ID=44.
• Because there is no UUID in the metadata of this note, tomboy-ng understands that 1) a new UUID must created 2) it must update its internal lookup table for ID=44 with the newly created uuid

--Martin

[davidbannon]

Right, I think we are focusing in here.

Next, the Nextcloud user deletes the note and create a new one, also assigned ID=44

If the user creates the note on Nextcloud directly, there is no UUID. Because there is no UUID, tomboy-ng understands that 1) a new UUID must created 2) it must update its internal lookup table for ID=44

No, there is still an entry in the table for ID=44 : xyz, no one has deleted it. tomboy will not not be able to detect that the note has been deleted and reissued. So, will use the old UUID. (different result if user sync after every delete).

If the user creates the new note on tomboy-ng, tomboy-ng knows that how to update its internal lookup table for ID=44.

Yes, in that case, tomboy-ng will know to remove to old entry for 44 from the table. It will request a new note on NextCloud (hmm, does NextCloud tell us the ID ? Thats probably another problem, if it does not get told the ID of the new note, it cannot add an entry to the table.).

I am starting to think an export / import filter is practical but not real syncing. No prospect of interesting the NextCloud developers in adding a UUID to a note's metadata ? It would be useful for a lot more applications than tomboy-ng.

Sorry I am being so negative !

Davo

[davidbannon]

OK monperrus, I have built a nextcloud install (wow, finally found a use for Snaps!) and have some more data. Interesting.

Note IDs are not allocated sequentially, indeed they jump a lot more than one per note, I have run up some six notes and each one was (substantially) greater than the previous one. The sequence was -

136 155 (deleted) 193 212 (shutdown and restart) 339 Maybe its elapsed time based ?

So, if they never go back, that does make them kind of unique, doesn't it ?

Not unique across platforms but I guess if we warn users about that issue they can be alert to danger.

I would like a statement from the developers that the number will always keep growing, do you want to ask them or should I ?

Davo

[monperrus]

I have built a nextcloud install (wow, finally found a use for Snaps!) and have some more data. Interesting.

cool :)

Note IDs are not allocated sequentially, indeed they jump a lot more than one per note, Not unique across platforms but I guess if we warn users about that issue they can be alert to danger.

Agree, this is good enough IMHO

I would like a statement from the developers that the number will always keep growing

I've browsed the Nextcloud code.

AFAIU, the id comes from method getId from interface Node with the implementation of FileInfo.

This id is actually a database field called file_id, confirmed by https://github.com/nextcloud/notes/blob/017e825dafd4212c362197f7def97bf225f17997/lib/Db/MetaMapper.php#L42

It seems that this database field is created by $fileId = $builder->getLastInsertId(); see
https://github.com/nextcloud/server/blob/67ecdc969cd93442918180e179e1f05ed8618d92/lib/private/Files/Cache/Cache.php#L295

Per my understanding of the database code, this is is unique and most likely increases monotonically.

Good news!

[davidbannon]

Thought you might like an update. I have not dropped the ball....
I have working now a rough mockup of a tomboy-ng to NextCloud sync. While its working and I think, reasonably satisfactory, it does have some limitations that end users will need to work with. For the record -

• I have to be sure that I am talking to the same server each time we sync. (Otherwise, I see a missing note at the far end and delete it locally for example!) So, during the initial 'Join' process, I make a new note that contains a UUID. If the user renames that note, deletes it or alters the UUID line, bad things happen ! I'll put some clear advice in that same note.

• Because NextCloud does not assign a unique ID to a note, I have to keep a mapping of UUID to the current NextCloud assigned ID. That mapping becomes invalid if you move your notes to a new NextCloud provider or rebuild your own install for example. So, if you then disregard the advice you get and proceed to make a new 'Join' you get duplicates of all notes that exist at both ends. This problem can be managed by syncing before abandoning the old NextCloud server and ensuring there are no notes manually transferred from it to the new server. You allow tomboy-ng to populate NextCloud ....

• NextCloud does not support some of the Markup used in the Tomboy standard. So, if you use underline or highlight, or larger fonts mid paragraph, that markup (but not the text) will be lost. Small font is similar but as there is a well supported and reasonably unobtrusive markup for that (that Nextcloud does not render) I have left that in, if you don't alter it, it remains through a sync cycle. If you don't like it there, don't use it. _{Github supports it}

I have not yet addressed any SSL issues. Nor keeping the NextCloud password in a safe place, thats looking like a platform by platform problem, sigh ....

I am also a bit concerned by the extra bulk this is adding to tomboy-ng, there is quite a lot of extra code required, particularly for the TomboyXML to markup conversion. I really don't know how many users will find this a useful addition. One approach may be to make a separate, standalone app that does the sync, only the tomboy-ng AND NextCloud users would need it. It would have to talk to tomboy-ng when sync-ing, else we'd risk changing a note just as it was synced.

Hmm....

Davo

[monperrus]

I have working now a rough mockup of a tomboy-ng to NextCloud sync

Oh great! I volunteer to beta-test it!

I am also a bit concerned by the extra bulk this is adding to tomboy-ng,

I think it's a great idea to keep it here, in order to have easy deployment and dissemination in distros having tomboy-ng, esp. Debian

[davidbannon]

OK, I have a working system I think. However, I have not tested it against an SSL Nextcloud server. Seems I cannot get SSL onto my Multipass Nextcloud (because it does not have a proper FQHN and the commercial provider I signed up for a free account with, QLoad (chosen at random), does not have the notes application installed and does not let me install a new application of my choosing. Is that normal ?

So, can you recommend a more suited provider ? I'm reluctant to go for a paid one, have no need for it. Or do you have a system available to try out against ? If so, let me know what OS you want the tomboy-ng client to run on (and assure me you understand about backups).

David

[sliwowitz]

Or do you have a system available to try out against ?

I have my own SSL server, I can set up a nextcloud account there for you to test if you want.

[monperrus]

It works!

First feedback: if the password field contain spaces or newlines at the beginning or at the end, it would be great to trim them.

[davidbannon]

First feedback: if the password field contain spaces or newlines at the beginning or at the end, it would be great to trim them.

I am not sure about that. On some systems, spaces in passwords are permitted, but probably not at beginning or end. Easy enough to do but I wonder how an errant space has got in there in the first place. I guess we could have a 'show password' option.

[monperrus]

When I first connected, I had an error "authentication failed", it took me 5 minutes to realize I copy-pasted the password with a newline.

[davidbannon]

Yeah, that would do it ! And most people will be copy and pasting credentials from some sort of password manager I guess ....

So, strip any leading or trailing white spaces or newline chars ? Probably ...

Davo

[monperrus]

Here are the results of my first tests, that's really cool

What works:

• create a note on desktop, sync, see it on server OK
• modify a note on desktop, sync, see it on server OK
• modify a note content on server, sync, see it on desktop OK
• create a note on server, sync, see it on desktop OK (updated on Oct 14 2020)

What does not work:

• rename a note (ie modify a note title) on server, sync, see it on desktop

Thanks Davo!

[davidbannon]

Interesting !
I was aware of the first issue (but forgot to follow up on it ). I suspect that when you rename a note in NextCloud, its last change date is not updated, therefore the sync engine does not know it needs to be updated. But I have not yet confirmed that. Will need to.

The second one has me a little unsure what you mean. You "create a note on a server, sync, see it on desktop" - "the note does not appear on desktop". Er, if you see it on the desktop, it must have appeared on the desktop ? Can you please clarify ? You make a new note using NextCloud, sync, and you don't then see it in tomboy-ng ? Thats something I thought I had tested pretty thoroughly but maybe I missed something. Was, I must admit, done in something of a rush....

monperrus, I will try and chase the first problem down, and reproduce the second one if I can but I will be pushed for time over the next few weeks. So, may take a bit of time.

Davo

[monperrus]

Hi Davo,

"create a note on server, sync, see it on desktop " is the test scenario. "the note does not appear on desktop" is the test result.

--Martin

[davidbannon]

OK, finally got a bit of time !

• Firstly - renaming a note on the server does not update its last modified time. Thats a bug I will report nextcloud/notes page.
• Secondly - creating a new note on server and syncing does work for me. I am not sure what we could be doing differently so, could you please test again ? Could there be issues with the version of Nextcloud/notes we are using ?
• Third - I am seeing a small memory leak, seems to be a GTK2 issue, will need further investigation.
• Fourth - rather strangely, after a couple of weeks away, when I fired up my test rig, it told me I needed to download fresh copies of all notes. But I had not changed them since last time I synced. I suspect this may be a daylight saving issue, we we moved to DST during that off time. Thats a worry, hard to debug ...

Davo

[davidbannon]

Item one above - nextcloud/notes#621

[davidbannon]

Item three above - I cannot replicate those GTK warnings nor the memory leak. 24 bytes, too small to be something I allocated, have to assume its some GTK thing, will watch for it but assume its not an issue.

Davo

[monperrus]

Thanks for the update. Tried again test create a note on server, sync, see it on desktop, it works. Updated my test results above accordingly

[davidbannon]

Thanks for testing. It would be nicer if we could find a reason why it failed the first time. I don't like problems that just 'go away'. They usually come back !

There is/was a problem where a note would go wrong if it was in a category that included, in its name, a html illegal character, specifically '&', '<' and '>'. Fixed here so be careful until next beta releases....

Davo

[davidbannon]

I can now replicate both the GTK warnings and the memory leak. To see either you have to compile with GTK (slaps forehead). The GTK warning are a bug in Lazarus Trunk, I use trunk for dev work but not release. Have reported it and its been fixed already. The memory leak is more mystifying. Its not repeatable, seems random. Never present in Qt5 version so I still suspect its a GTK issue. But until I can make it repeatable, no hope of getting anything I can fix or report.

[davidbannon]

So, it seems that the NextCloud Notes people consider that changing a note's title is not sufficient to update its last change date. While I disagree, there is little I can do to about it. I guess I will have to assume that this was an experiment that, like many experiments, that did not work !

I don't think I can justify merging this code with the mainstream one. Sorry !

Davo

[monperrus]

Hi Davo, I understand that you would close this issue for now? Thanks! --Martin

[davidbannon]

Hmm, not sure, I cannot merge the code while its incomplete, but maybe we leave this issue and see if NextCloud people change their mind ?

I could also, possibly, download all the note titles and see if there has been a change, its pretty time consuming, especially if you have a large note collection, I have many users with lots and lots of notes !

The NextCloud Notes API is not really very suited to this job I am afraid. So I wonder if we would see many people actually wanting to do this sync anyway.

Davo

[monperrus]

I understand, thanks a lot for your effort!

[davidbannon]

OK, I have had an answer from korelstar, it appears I have made a mistake in using the Modified parameter. I need to re do the whole model using a parameter to GET, pruneBefore. That will require bit of work and may not happen immediately. But it does sound like it can be done.

Davo

[monperrus]

Nice!

[davidbannon]

No, sorry, the pruneBefore model has problems too. nextcloud/notes#627
Here is a report.

There seems to be, potentially, two ways to determine what NextCloud notes need to be synced. The more (Tomboy) conventional way is to ask NextCloud for a list of all notes and noting each one's Modified time. However, some activity on the NextCloud system, such as renaming a note, does not cause the Modified time to be updated.

The alternative approach, recommended by the NextCloud Notes people, is to set a pruneBefore parameter when asking for a list of NextCloud notes, the returned list highlights those notes changed between that pruneBefore date and 'now' . However, a note, uploaded as part of a sync event, ends up with a timestamp (for pruneBefore purposes) after the official time for that sync event. So, is reported as changed at the next sync.

So, using the first model, we could tell end users that they cannot change NextCloud's Note's title in NextCloud, doing so without also making some arbitrary change to the note's content will leave the note with its original name. Apart from the name itself, no data would be lost.

In the second model, the user may be confronted by a 'Clash Screen' asking them to choose between two changed versions of the note, at a time when they are (rightly) sure they have not changed the NextCloud version. Tempting to run an automatic immediate second sync, to get a new LastSyncTime. However, if the user is currently editing the note, a real possibility if it was just in need of sync, then again, a 'Clash Screen' would be triggered.

The first is arguably preferable in that it is quite predictable and minimises data loss. The second model is less predictable, depending on when a user next edits an uploaded note and could possibly result in substantial data loss.

I believe any application using the NextCloud Notes API risks the problems outlined above. Would love to be proven wrong !

Davo

[monperrus]

Thanks for the analysis, it seems that the Notes API is not the best backend for what we want to achieve.

[davidbannon]

I don't think its a bad API, it just has a bug ! Could be fixed by either making things like name changes updating 'modified' time or providing a means to set an uploaded note to a specific timestamp (as seen by pruneBefore).
Or we learn to accept those limitations, personally, I don't think so ...
Lets hope someone has a better way that I did not spot !
Davo

[davidbannon]

nextcloud/notes#692 indicates that the issue I was concerned about may have been fixed. I have not looked at it closely but hopeful. I imagine my Nextcloud branch will too far back to safely merge so will have to a bit of work to find out, so don't expect a quick turnaround.

Davo

[monperrus]

thanks for the update, looking forward to it!

[davidbannon]

Monperrus, I have released a version of tomboy-ng that syncs with github, I knwo thats definitly not what you were asking for here but I think its probably a better solution for the average user, installing your own vrsion of nextcloud is not for the faint harted !

So, assuming you don't mind, I al closing this ticket.

Thanks for your help and input, rather ironically, I did get to use some of the Nextcloud code I wrote in my github solution !

Davo

[monperrus]

ack, thanks!

[monperrus]

Hi Davo,
The one thing which I really miss is an Android app for my notes.
If we're able to get the Nextcloud Notes bridge working, then we would reuse the Nextcloud Notes app for free.

[davidbannon]

Ah, Martin, another message to a closed ticket ? There does not seem any setting that would allow the author to reopen a closed note and I think thats a pity. Reopened this one but its an unreliable way to work.....

Look, I still have the code I wrote for my first attempt, the API it hits from my end has not really changed, maybe I could look at it again. But its a big job, needs a lot of time and, from memory, the notes app did not appear in the online, free, nextcloud servers. And, honestly I hear nothing about NextCloud since we discussed it last time, is it going up, or down ?

I was involved in the Grid Computing movement (that morphed into Cloud) long, long ago and NextCloud seems a lot bigger then ....

Davo