Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

YapDatabaseCloudKit with a large amount of data. #212

Open
SofteqDG opened this issue Sep 2, 2015 · 4 comments · May be fixed by #512
Open

YapDatabaseCloudKit with a large amount of data. #212

SofteqDG opened this issue Sep 2, 2015 · 4 comments · May be fixed by #512

Comments

@SofteqDG
Copy link
Contributor

SofteqDG commented Sep 2, 2015

I want to sync all my contacts (~900 contacts) from address book to cloudkit. I have a separate connection for synchronization between address book and my database where enumeration of contacts is performed. All saved contacts will be passed in YapDatabaseCloudKitRecordBlock when readWriteTransaction will be finished. YapDatabaseCloudKit sends all saved contacts to CloudKit. But operation fails with "Invalid Arguments" (12/1020); "Your request contains more than the maximum number of items in a single request (400)" error.

How i can limit a number of items in single request?
I didn't found setting for that.

@SofteqDG SofteqDG closed this as completed Sep 6, 2015
@ainopara
Copy link
Contributor

ainopara commented Sep 7, 2015

Why close the issue?
I met the similar situation. When I migrate data from other database, I can limit items in a single request by divide those setObject: forKey: into different transaction.
But there is still some situation that cloudkit will automatically try to upload all entity in my database to iCloud and it will exceed the limit when entity count in my database is more than 400.
I tried to fix this problem by modify

- (void)queueOperationForChangeSet:(YDBCKChangeSet *)changeSet

in YapDatabaseCloudKit.h to divide a changeSet to multiple modifyRecordsOperation.
it works but leave some awkward issue that the masterOperationQueue is not the queue YapDatabaseCloudKit used to count for inFlight and suspendCount. and operation that not finished successfully will not be recorded.
So I think a build in limit to changeSet is necessary.

@SofteqDG
Copy link
Contributor Author

SofteqDG commented Sep 7, 2015

I think that we have several ways to fix this issue:

  1. Rewrite your logic. Don't make more that 400 changes (insertion, deletion, modification) in one transaction.
  2. Add a flush method to YapDatabaseReadWriteTransaction like the flush method in YapDatabaseRelationshipTransaction. In this case developer can flush all pending changes when he need it.
  3. Somehow split YDBCKChangeSet.

1st way is simple and doesn't require changes in YapDatabase framework. I have already rewrote my logic. I'm saving not more than 350 changes in one transaction and all is working fine.

3rd way is very dangerous because of CKReference. According to Apple's documentation:

When saving a record that contains a CKReference object whose action is CKReferenceActionDeleteSelf, the server must know about the target of the reference. Attempts to save this record without first saving its target record results in an error. The solution is to save the target record first or to save both records using the same CKModifyRecordsOperation object.

YapDatabase knows nothing about stored data. Your objects can be sent in different operations what will cause an error.

2nd way is a great way. Developers will have a more control. But developer should be aware of situation described above.

What do you think, @robbiehanson?

@ainopara
Copy link
Contributor

ainopara commented Sep 7, 2015

1st way is the method I used right now. But there is a situation it can not cover.
When registering the cloudKit extension,

- (BOOL)createIfNeeded

is called and it will check if database hasOldClassVersion.
if hasOldClassVersion is NO, cloudKit extension will call

- (BOOL)populateTables

to enumerate allowedCollections and add CKRecord generated by the entity in those collections (it can easily exceed the 400 limit) to the dirtyMappingTableInfoDict and the dirtyRecordTableInfoDict.
Then in

- (void)flushPendingChangesToExtensionTables

they will transform to ONE YDBCKChangeSet in a pendingQueue and be merged to masterQueue.

In my case, the YDBCKChangeSet has 7000+ recordsToSave. And It seems there is no way to handle the CKError 12/1020 outside the framework. So I think a limit in YDBCKChangeSet is necessary to avoid all potential mistake that will generate a big change set.

@robbiehanson robbiehanson reopened this Sep 9, 2015
@robbiehanson
Copy link
Contributor

Add a flush method [...] like the flush method in YapDatabaseRelationshipTransaction.
In this case developer can flush all pending changes when he need it.

Adding a flush method to YapDatabaseCloudKitTransaction is a pretty good idea. Even if it doesn't solve every scenario, I can still see it being a useful option in some scenarios.

When registering the cloudKit extension [...] cloudKit extension will call - (BOOL)populateTables

Excellent point !

From my current understanding, it seems that we have 2 possibilities:

A. Modify YapDatabaseCloudKit in such a way as to eliminate the populateTables / repopulateTables code. In order to accomplish this, it would have to be replaced using a more manual technique. I don't know what such an API would look like at this point. But I would imagine it would end up causing a lot more work on the side of the developer (you). And would also suffer from being significantly different from other extensions (in this does-not-auto-update manner).

B. Fix the internals of YapDatabaseCloudKit to automatically split a ChangeSet into a maximum of 400 items. And do so in a safe way that allows us to take into consideration CKReferences.

I believe the best option is B. I just need a bit of time to think up exactly how I want to implement the "dependency graph & storage".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants