Some data never gets uploaded if initial sync is interrupted

[CapoChino]

This has been happening to me as long as I've used Simperium. I kept hoping it would go away or I was doing something wrong. But it's here still.

As you know from #208 and #274 , when one of my users uploads their data to Simperium for the first time, it may take 8 hours for a moderately-sized database. It's unlikely that my users will allow the app to run long enough to upload all data. If the user closes the app and returns later, Simperium is supposed to pick up where it left off and continue uploading data. However, often times, I've found that the upload is never resumed, no matter how long or how many more times my app runs.

This seems like a bug. Or maybe there's something I should do upon launch to check for leftover data or kickstart the sync? I have a database in this state now. I could send you the account login, sqlite database, and SPDictionary-* and SPMutableSet-* files if that would help.

Thank you!

[jleandroperez]

Hello there @CapoChino,

Hope you're doing great!. I got your metadata-files (thank you for that!), however, it'd help quite a lot if we could debug using your app (so the entire scenario is entirely accurate).

If possible, please, try doing the following (clean start, before you begin the data upload):

1. Add a breakpoint here. This line is expected NOT to be executed. If so, this indicates there is a backend glitch, and a change-set got discarded.

Please, begin your data upload. Whenever you're ready to reproduce the Changes not posted glitch, stop running your app, add the following breakpoints, and when ready, hit Run:

1. Breakpoint Here and make sure that the second if clause is executed (this is just a sanity check!).
2. Breakpoint Here and make sure that * onlyQueuedChanges* is false, at least the first time this gets executed!.
3. Breakpoint Here and once LLDB breaks there -for the first time-, please, execute the following commands

p (int)processor.numChangesPending
p (int)processor.numKeysForObjectsWithMoreChanges

The first print (numChangesPending) should carry the number of pending changes to be uploaded, while the second one should indicate how many objects got new changes that are not already tracked by the 'changesPending' collection.

If both prints indicate zero changes, then we'll need to track why they got discarded without being applied.

Thanks in advance!!

P.s.: This bug might be related to #281

[CapoChino]

Indeed #281 does share a similar property.

Thank you for the detailed debugging instructions. I followed them to interesting result. Data upload happens normally. I placed the breakpoints and they were hit periodically as the data was uploaded. Then I stopped the app by pressing the 'Home' button in the simulator, then killing the app in the debugger. When I re-ran the app, none of the breakpoints were hit. I enabled verbose logging and ran again. Nothing extra was printed, and still no breakpoints hit, and no more objects uploaded to the cloud.

Here's the output:
2014-06-03 13:08:10.092 MyApp[68574:907] Reachability Flag Status: -R -----l- networkStatusForFlags
2014-06-03 13:08:10.097 MyApp[68574:1103] Simperium loaded 6 entity definitions
2014-06-03 13:08:11.043 MyApp[68574:1b03] Simperium starting...
2014-06-03 13:08:11.105 MyApp[68574:907] Simperium managing 122 ... object instances
2014-06-03 13:08:11.106 MyApp[68574:907] Simperium managing 8 ... object instances
2014-06-03 13:08:11.153 MyApp[68574:907] Simperium managing 852 ... object instances
2014-06-03 13:08:11.160 MyApp[68574:907] Simperium managing 852 ... object instances
2014-06-03 13:08:40.950 MyApp[68574:907] Simperium managing 136180 ... object instances
2014-06-03 13:08:40.988 MyApp[68574:907] Simperium managing 8 ... object instances

Looks like it might be a simple problem of simperium not fully starting up. Thanks for looking at this.

[jleandroperez]

@CapoChino many thanks for helping us debug this glitch!

Would you please confirm that after stopping + relaunching... this breakpoint is not getting hit?.

If possible, please, add the following breakpoints, and check if they get hit after relaunching:

1. Breakpoint Here and make sure startWithAppId method gets called.
2. Breakpoint Here and verify that the [self.authenticator authenticateIfNecessary] line gets called
3. Breakpoint Here. This should get called right away with the user's token
4. Breakpoint Here and make sure the start method is called for every bucket.

This sequence is valid, assuming that you're starting Simperium like this:

simperium = [[Simperium alloc] initWithModel:model context:context coordinator:coordinator];

[simperium authenticateWithAppID:identifier APIKey:yourKey rootViewController:rootViewController];

Thanks in advance!

[CapoChino]

Oh shoot. I'm sorry, I made a mistake a wasted some of your time. I'm sorry about that. I set self.simperium.authenticationEnabled = NO because I don't want to prompt the user to create an account when they are first using the app. The first time I was explicitly signing in, triggering the auth. Then on the subsequent runs, no auth was happening, so most simperium machinery wasn't started. That explains no breakpoints getting hit. When I commented out the disable auth line, the upload does resume upon app launch. I confirmed this twice.

What is the correct way to prevent prompting users to create an account, yet attempt to log in the user if they already have an account? Should I check SPUsername in NSUserDefaults and only disable auth if it doesn't exist? Furthermore, even my solution of disabling auth isn't supported because it seems you guys made that property private at some point. I think it used to be public and that it's still documented. Any suggestions on the best way to accomplish what I'm trying to do?

Lastly, I'd like to keep this issue open. As I mentioned, I've noticed this issue since I first started using Simperium, and my auth disable change is much more recent. I'm concerned there still may be a sporadic issue. I'll try to repro with these breakpoints in place and will keep you posted.

Thanks again for your time, and I'm sorry for the false alarm due to not authorizing the user!

[jleandroperez]

@CapoChino no worries!. The standard behavior (right now) would be:

• Whenever you're ready initialize Simperium, you'd need to just call authenticateWithAppID:APIKey:rootViewController:.
• If the user had its credentials stored, no UI will be shown. However, if there were no credentials stored, the Login/Signup UI will be presented.
• By setting the authenticationOptional flag to YES, you'd be allowing the user to dismiss the Login/Signup viewController (if he decides to do so).

I've just added a getter in SPAuthenticator class (needsAuthentication), which will return YES if no credentials are stored in the keychain (if this method returns YES, then the Login/Signup UI will be shown).

You can check that by hitting simperium.authenticator.needsAuthentication.

Regarding the Sync'ing issues, we've recently fixed a potential candidate that would match your description: #289

Hope this helps!

[jleandroperez]

@CapoChino may i ask you if you're still having issues?.

Thanks!

[CapoChino]

Hi @jleandroperez,

I'm not having this exact issue anymore, so we can close this issue. However, I am still having an issue similar to this one. Instead of the upload sometimes stalling, the download sometimes stalls. I end up with a partial local database about 1 in 5 times on the initial sync down to a fresh install. I'll try to create a test case and file a separate issue for that.

Thanks!

[jleandroperez]

@CapoChino thanks for your feedback!. May i ask you to please check if your SPReachability looks exactly this way?. We've recently tracked (and fixed!) a stallment bug caused by Reachability reporting an invalid networking status.

Thank you!

[CapoChino]

Yes, I've got that Reachability fix. I should clarify that the problem is actually with resolving relationships. All objects get downloaded, but not all the relationships are set; some remain nil in the local db.

The issue happens with master and develop branches.

[jleandroperez]

@CapoChino ahhhhh i see. I believe you're facing this issue.

I've implemented an experimental fix in this pull request, which does the trick and fixes the issue, but might be a bit over-engineered.

We'll be fixing that one very soon, thanks for sharing the details!

[CapoChino]

Ahh, that seems to be exactly it. I glanced over open issues but somehow I missed that one! Thank you.

[jleandroperez]

@CapoChino cool, thanks!. Closing this issue in favor of #250