synchronization: add support for automated synchronization with frequency

[monperrus]

In the synchronization setting screen, it would be great to be able to set up automated synchronization with frequency:

• hourly
• daily
• weekly
• monthly

Thanks!

[davidbannon]

Yes, I agree, good idea. I'll look at that at some stage.
(Sorry for slow response, have been traveling and seriously out of contract).

Davo

[davidbannon]

OK, thinking out loud here.
Drop the Sync Enabled box and instead offer -
Sync : [] Off [] Hourly [] Daily [] Weekly

Will require replacing existing settings in config file, FileSyncEnabled and GitSyncEnabled (both boolean) with new ones, perhaps string, off, hour, day, month. Old setting must remain to ensure backward compatibility. Day and Month will also require saving the time and date of last sync. Will need to dwell on this, hate changing config meanings ....

But overall, sounds reasonable.

No, 'Off' does not mean the same as not auto. If not auto, user may still trigger a sync manually.

David

[monperrus]

Sync : [] Off [] Hourly [] Daily [] Weekly

Loving it

[davidbannon]

This seems OK, superficially tested. Clearer code, slightly simpler user interface, good !
Dropped checkboxes 'enabled' and 'auto'. Combo box, 'manual', 'hourly' etc.

[monperrus]

cool, looking forward to testing it.

[davidbannon]

OK, there is a beta with 0.36e in the file name, here -
https://github.com/tomboy-notes/tomboy-ng/releases/tag/v0.36c

I don't remember what platform you use and there are a lot of files in that directory. Look for the one with 'e' in its version number.

If you are using Linux and Qt5, you must get a new libqt5pas library, see link in that page.

Davo

[monperrus]

installed 0.36e. I can see the new sync option, thanks! however, it's only available for local filesystem sync, but it's disabled for Github sync (which is my use case).

[davidbannon]

Oh, that is not intentional ! I'll look into it ......
Fortunately, my debug code is still in there.
Davo

[davidbannon]

OK, seems to work for me. You realise you set the frequency for each sort of sync individually ?
Can you please see if any messages are being dropped to console ? I wasted an hour or so tracing a bug here that was being caused by what I discovered to be a very slow (ie timeout) DNS. You should see an error message if that happening but I am pretty sure its a local to our network problem rather than github.

What OS do you use ? (if Linux, gtk2 or Qt5 ?) - I could send you a debug binary that might be helpful.

Davo

[monperrus]

I use Linux ia64. Probably the gtk2 build given the output of ldd.

[davidbannon]

OK, GTK2, 64 bit linux binary attached.
In your existing tomboy-ng, make sure File Sync is set to manual, github to hourly, exit.
Then just untar the attached, drop the binary in, eg, your home directory. Make sure your real tomboy-ng is NOT running, start this one from the command line. It will pick up your notes and config so behaves just like your existing one except that drops messages to console and it's sync cycle is running 100x faster. So, we get several cycles in 4 or 5 minutes. Thats all I need, take a screen shot or a screen scrap and copy to here. Pretty sure nothing of private nature should be shown but do check before posting !

Davo

tomboy-ng.tar.gz

[davidbannon]

OK, this issue resolved in https://github.com/tomboy-notes/tomboy-ng/wiki/Download_Release

So closing this issue, thanks for your report.

Davo

[monperrus]

great, thanks Davo

[monperrus]

Hi Davo,
FYI, in settings, I set and see "Github sync". But with right click, synchronize, it says cannot synchronize to /tmp as if it considers a local folder for synchronization.

[davidbannon]

Hi Martin, sorry, this ticket was closed so I did not notice you message until just now. I have reopened it.

Did you setup your tomboy-ng repo on github ? Or are you telling me it went ahead and tried to sync without a proper config ? That would be messy indeed.

Davo

[monperrus]

Tried again. The actual problem is that I cannot complete the first sync. It ends with
"List index (3) out of bounds."

[davidbannon]

Sorry, your really need to need to spell out exactly what you are doing/trying to do I am afraid. Without that information, I really am pretty useless. Just to focus down a little -

Can you please tell me what version of tomboy-ng you are using ? On what platform ? And, platform specific, do you know the version of openssl you are using ? That has become quite critical now that many platforms are using openssl 3.0 and some versions of tomboy-ng are not happy about that.

You are trying to do a github sync ? Assuming yes, you have added a valid security token to your tomboy-ng config ? And how far did you get with that sync ? An initial sync is a two stage thing, it does a dummy run to see if everything is OK and then asks you to confirm and save. Did you get that far ?

Does the private github repository (TB_notes) now exist in your github account ? You can only see it when logged into github. Does it contain an index and some notes ?

---

Background info.
I am particularly concerned about an openssl issue I just found out about in the last couple of weeks. But I don't think that is your problem because the error "should" be well trapped and you get a sensible error message to the effect that it cannot initialize openssl. Whereas the error you see about a list index could come from anywhere and is that dreaded un-handled error IMHO. Some background -
I believe, but am not certain that tomboy-ng compiled with fpc322 will not work with on 'some' platforms with openssl 3.0. In Linux, its easily fixed, install libssl-dev. I don't know of any easy fix on Windows (but generally, Windows lags way behind with things like openessl anyway. I have not had time to test on the Mac.

Davo

[monperrus]

Can you please tell me what version of tomboy-ng you are using ? On what platform ?

0.39 on Linux

And, platform specific, do you know the version of openssl you are using ?

3.0.2-0ubuntu1.15

You are trying to do a github sync ?

yes

you have added a valid security token to your tomboy-ng config ?

yes

And how far did you get with that sync ?

Starts, "Save & Sync", detects clashes, ask for what to do, say "Use local for all", then fails with "List index (3) out of bounds", does not complete sync.

[davidbannon]

Martin, I am currently traveling and cannot setup a test rig. But will make a few suggestions.

Firstly, I still have some concerns about version 0.39 of tomboy-ng and the openssl you are using. I believe you "should" get a clear openssl error message if that the problem (and I'd be conducting tests in that direction if I could).

I do have a beta test version out at present that will be much happier with your openssl, maybe worth looking at.

I don't, without some testing, understand what the list index issue is all about, I cannot see how you'd get there if you do have openssl issues. I understand you would not be able to connect at all.

Can you please look, in your github repository and see if - Does the private github repository (TB_notes) now exist in your github account ? You can only see it when logged into github. Does it contain an index and some notes ?

From your description, it sounds to me as if you did not, at any stage, get to the "Save and Sync" stage ? Crash while still reading your local repo (and presumably NO remote repo) ? Thats important info for me, sounds just a bit OpenSSL again. But maybe, github's extra level of security, they recently added two factor authentication, you may need to do that two factor thing before proceeding ?? I will have already done so because I commit to my github account all the time . So I would not see this problem. Maybe ....

This extra level of github security could turn our to be the problem now I think about it, may be a serious problem...

Davo

[monperrus]

Does the private github repository (TB_notes) now exist in your github account ?

yes

Does it contain an index and some notes ?

yes

it sounds to me as if you did not, at any stage, get to the "Save and Sync" stage ?

maybe partially since the repo exists and contain some content thanks!

[davidbannon]

Sorry for slow response, I have been travelling.

I have confirmed that the existing tomboy-ng v0.39 will not work with openssl 3.0 so it would seem likely this is your problem. If you do have some files in your github notes repo, could that repo have been made before you updated your desktop OS ? In particular, that would have got you a new openssl.

The beta I have just pushed to https://github.com/tomboy-notes/tomboy-ng/releases/tag/v0.39a solves this openssl problem. (note github tag says v0.39a but content is v0.39c, released 18th April).

Sorry about this. Problem is that the current formal FPC release does not support openssl 3.0, I have built these with a (very well tested) beta compiler. Sigh ...

Davo

[monperrus]

I downloaded the last version. Same problem. I'm sure OpenSSL works because some notes are pushed.

Looking at .config/tomboy-ng/tomboy-ng.cfg

[SyncSettings]
Autosync=false
SyncOption=AlwaysAsk
SyncType=file
SyncNextCloudRepo=
SyncRepo=/tmp/
AutosyncGit=false
FileSyncEnabled=false
GitSyncEnabled=false
SyncRepoGithub=
GHPassword=<valid key>
GHUserName=monperrus
SyncTimingFile=daily
SyncTimingGithub=manual

What should be SyncType for Github? What's the format of SyncRepoGithub value?

[davidbannon]

Sorry Martin, been distracted.

From memory, SyncType is not used with GitHub sync. Looking at my config, whats important appears to be -

SyncRepoGithub=https://github.com/davidbannon/
GHPassword=
GHUserName=davidbannon
AutosyncGit=true
GitSyncEnabled=true

It also observes the common ones about timing Autosync etc.

So, your GitSyncRepo appears to be blank (unless you blanked it before posting ?). If its really blank, no sync is going to happen. Your GitSync is also disabled. That says to be it was once configured, is not usable now.

I'm guessing you have tried to re-configure with the "change sync repository" button.

Its (intended to be) always safe to resync or reconfigure sync with tomboy-ng, unlike Tomboy where you would end up with a heap of duplicated notes. If you have not tried to resync, please do so.

Next step might be to blow away the repo, allow -ng to create a new one, again, should be safe.

Why is going on ? Beats me. One thing that keeps nagging me is the possibility you had a working sync, one way or another, your openssl was updated, bad things happened. That bad thing is fixed in that beta you are now using, don't go back to the old one. I accept the old one would not be able to sync, however, surprised that what ever went wrong left it unrecoverable. If I get some time, I'll try and simulate that situation on a test rig and see what happens.

Davo

[monperrus]

SyncRepoGithub=https://github.com/davidbannon/
GHPassword=
GHUserName=davidbannon
AutosyncGit=true
GitSyncEnabled=true
SyncRepo=tb_notes

Thanks, aligned accordingly. I understand that SyncRepoGithub is concatenated with SyncRepo.

More debug info:

• I rightlick on the tomboy tray, then synchronize
• in the console, I see syncgui.pas, ManualSync(), line:352 : Test Transport Failed.

[davidbannon]

OK, thats useful, when that console msg appears, I expect you to be doing a Manual Sync and you should see a (hopefully) more useful message in the tomboy-ng window that is open at that time. Is that the message you mentioned on the 17th March ? eg "List index (3) out of bounds." ?

If so, firstly, thats not one of my messages, that in itself is interesting. It appears you are calling my function TSync.TestConnection(): TSyncAvailable; and getting one of the TSyncAvailable enumerated types back. All the "error" conditions should give a further error message and you are obviously not seeing that.

So, I suspect you are getting that error from Transport.TestTransport(), "Transport", in this case being the github transport class. Its the only one that can return an error without me translating a low level error into a more meaningful one. (Because, I suspect, I did not expect, or could not trigger, an error of that nature.)

This method makes initial contact with Github, gets back and decodes some JSON, makes some decisions about if its safe to proceed. The JSON decoding and returned data all use lists and that likely whats going wrong.

Please give me a couple of days (travelling again, sorry !) to put together a binary that will generate some much more explicit error message (useful to have in the code anyway now you have demonstrated its need).

Davo

[davidbannon]

Martin, I have now added some better error checking to the github sync, will make you up a tarball if you tell me if you use the Qt5, Qt6 or gtk2 version of tomboy-ng please ?

So far, this "better error checking" has revealed, for me, an occasional timeout when talking to github's servers. I am unsure if it has been happening in the past, it could be due to my current poor network connection (my phone left sitting on an upturned bucket on a small ridge near our caravan). But it is happening. I have not, at this stage, built in some retries following a timeout, I would prefer to try and catch the problem "red handed" with your very tolerant help !

Sadly, a closer examination of the code has not revealed any list likely to be the cause of the exception you see. And I still cannot replicate your problem !

Davo

[monperrus]

I'm using the gtk version. Thanks Davo.

[davidbannon]

Ah, well timed ! We are in town with a decent connection !

Attached a zip containing "tomboy-ng-text", unzip, don't bother to install, just run, from command line, must use eg

$> ./tomboy-ng-test --debug-sync > log.txt

It will generate a lot of noise during a sync, including listing names of notes so check content before posting !

Davo
tomboy-ng-test.zip

[monperrus]

Here is the result of Click on tomboy icon in system tray >> Synchronise

TFormSync.FormShow 
Remote address is tb_notes/
Local Config /home/martin/.config/tomboy-ng/
Notes dir /home/martin/.local/share/tomboy-ng/
Reading local mainfest /home/martin/.config/tomboy-ng/manifest.xml
TSync.TestConnection() : failed Transport.TestTransport, SyncNoRemoteDir : Perhaps sync device or shared drive is not mounted ?
syncgui.pas, ManualSync(), line:354 : Test Transport Failed, SyncNoRemoteDir : Perhaps sync device or shared drive is not mounted ?
Failed testConnection
TFormSync.FormShow 
TGithubSync.SetTransport - called
Remote address is 
Local Config /home/martin/.config/tomboy-ng/SyncGithub/
Notes dir /home/martin/.local/share/tomboy-ng/
ReadLocalManifest set an empty LocalLastSyncDateSt, probably local manifest does not exist.
syncgui.pas, ManualSync(), line:354 : Test Transport Failed, SyncNoLocal : We dont have a local manifest, only an error if config thinks there should be one
Failed testConnection

with

[SyncSettings]
Autosync=false
SyncOption=AlwaysAsk
SyncType=file
SyncNextCloudRepo=
SyncRepo=tb_notes
AutosyncGit=true
FileSyncEnabled=false
GitSyncEnabled=true
SyncRepoGithub=https://github.com/monperrus/
GHPassword=XXXXXX
GHUserName=monperrus
SyncTimingFile=daily
SyncTimingGithub=manual

thanks!

[davidbannon]

OK, key here is this line -

TSync.TestConnection() : failed Transport.TestTransport, SyncNoRemoteDir : Perhaps sync device or shared drive is not mounted ?

I understand you do, definitly, have a 'tb-notes' repo in github account. The above problem then could be caused by two things :

1. The github token does not allow reading of the remote (that is on github) manifest or serverID file.
2. The above files do not exist.

Easy check, in your browser, login to github, go to your the tb_notes repository, look in the Meta directory. You should see two files, serverid and manifest.json. serverid has a single line with an GUID, manifest has a some json listing the notes in the repo. If files are present and not empty, then I strongly suspect your github token does not have "repoository scope".

If either file is not present, then we have to assume that 'something' has broken during a sync and its not recoverable. I am not sure just what that could be, I do a "journal like" write to github and all my tests, including breaking in the middle of a sync, were recoverable on the next run.

Anyway, if its broken, and you don't have notes on github that are not present on your local PC, I suggest you remove the github tb_notes repo and start again. Again, my test indicate that should work, -ng will detect no repo present at the other end, will offer to create one for you and then do a full sync.

Davo

[monperrus]

You should see two files, serverid and manifest.json. serverid has a single line with an GUID, manifest has a some json listing the notes in the repo.

That's the problem. The repo has serverid but not manifest.json.

'something' has broken during a sync and its not recoverable

OK, I'll try again from an empty branch.

[monperrus]

Tried from empty branch.

First upload fails with List index (3) out of bounds., and no manifest.json on the repo, and only 2/3 of the notes uploaded.

Last log lines:

ID=9dd17741- 0    UploadNew        software takes command (book)  sha=
          CDate=2019-04-15T18:49:55.4176840+02:00 LCDate=2019-04-15T18:54:37.2765520+02:00
Downloaded notes.
DoDeletes Count = 1108
Doing uploads and Remote ServerRev is -1

That's consistent, I was never able to complete the initial sync.

[davidbannon]

Hmm, sounds like it chocked on one particular note. But then proceeded, so the sync completed (skipping that note) and would have written a manifest. But evidence is that did happen.

OK. Lets assume it did not proceed. That makes sense. It sounds like you have a note, named "9dd17741-something.note" that cannot, for some reason be uploaded to github. You need to workout which note that is -

$> la -l ~/.local/share/tomboy-ng/9dd17741 {and press tab for file name completion}

then, grep that file for the title line -

$> grep title {whatever that full file name was}

I would really like to look at that that note to determine what the problem is, if its not confidential, you might consider sending it to my private email dbannon at internode dot on dot net - I totally understand if you do not want to, totally !

As part of the github upload process, the note is converted to (github flavour) markdown, its possible that something in the note is not being so converted. You could try opening the note (in tomby-ng) and exporting as markdown. If that generates an error, that might give us a hint about the problem.

Opening and making a small change will trigger a rewrite of the note, that might well make it acceptable. The note is some five years old, tomboy-ng was pretty buggy code back then....

Otherwise, look through the note, look for something unexpected, utf8 characters were a problem early on, see something that does not render as you expect ?

I will be home in a few days with better access to debugging platforms.

Davo

[monperrus]

sounds like it chocked on one particular note.

OK, we'll have to find this one and I can probably send it to you over email. For this, we need to add one log line about the note being processed and sent to Github.

[davidbannon]

Martin, I have mislead you, I am afraid I did not look at that log you sent anywhere near carefully enough. The note ID I mentioned is almost certainly not the problem one I am afraid.

Looking again, now I am home, at your log, we seem to build a list to up load OK, being a new repository, no downloads or deletes. But I am still convienced its one particular note that is casusing the problem, we just need to workout which one. So, I have now added a fair bit of reporting in the Uploading code, it should identify the ID of the note and, perhaps confirm its a convert to markdown issue.

So, sorry to do this, but attached is another binary, please run it, if I am right in my guess, the useful information will be at the end of the log.

Sorry about how long this has taken, I have been distracted by some family business interstate.

Davo
tomboy-ng-test.zip

[davidbannon]

OK, better debugging still. I have disabled the listing of all present notes, far too much info there, but not useful in current context. I have also added some more, hopefully targeted debugging.

For the record, I have seen, occasionally and not reproducible, an error relating to httpclient not being able to resolve api.github.com - I don't think that is the underlying problem, it only occurs when offering a token to scan the private directory. I think its just possible I am hitting github's rate limit but seems unlikely (5K per hour ?). Anyway, better info about that effect built in now. Interestingly, I only have seen this when using my phone for network services.

Also, better info about a particular note being a problem, thats where I think your problem lies. Logs now give note title and area where it goes wrong. But, a worry, the markup code should, already, make a song and dance if something bad happens !

I have now run, extensively, my test rig, several hundred notes, cannot generate the error you see.

Attached, is a mk 2 test binary, same thing, clean out the github repo, run the binary with --debug-sync. When it goes wrong, there will be a block of 20 or 30 line of debug, possibly title of a troublesome note ?

Davo

tomboy-ng-test2a.zip

[monperrus]

Used the debug more to identify the faulty notes.

You were right, two notes were crashing tomboy-ng while syncing.

I attach them for reproducing and fixing the bug.
df4241ff-34c3-4429-9ed7-86115101211b.note.txt
48f35170-144e-4c38-8d4b-f0cfc392d24a.note.txt

thanks!

[davidbannon]

Yep, I can replicate your crash. Best news !

Neither should be accepted as valid notes but apparently do get through. So, I'll ensure, firstly that the markdown converter gives a sensible error message and then that tomboy-ng flags them as invalid when you open them.

Thanks for your tolerance here, I consider robustness very important !

Davo

[davidbannon]

Progress, just committed b5a51e8
Martin, your first attached note, one starting with df42 is OK, has no visible content but does have the required xml and a blank lines as content.
The second one, starting with 48f35 is the real winner here ! It is valid xml but not valid "tomboy xml". It contains a a line
<note-content version="0.1" />
That is good xml but cannot belong in a tomboy note, I am quite sure tomboy-ng did not create it and reasonably sure Tomboy did not. So, very strange. Anyway, its now detected and flagged as a bad note, you would be encouraged to either try and re-save it or, better in this case, delete. Importantly, sync ignores it.

I'll be doing some cross platform testing then release this as first v0.39e beta and then as v0.40, hopefully not too long afterwards.

Thanks for your help here, having a note that passes the standard xml tests but fails tomboy was a) unexpected and b) immensely useful for me to test robustness, now significantly better !

Davo

[monperrus]

great, looking forward to v0.40!