Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Batch processing and Unprocessed records clarification #236

Open
jamesgibbons92 opened this issue May 4, 2023 · 0 comments
Open

Batch processing and Unprocessed records clarification #236

jamesgibbons92 opened this issue May 4, 2023 · 0 comments

Comments

@jamesgibbons92
Copy link

From the docs:

Elements of the unprocessed array are unlike results received from a query. Instead of containing all the attributes of a record, an unprocessed record only includes the composite attributes defined in the Table Index. This is in keeping with DynamoDB’s practice of returning only Keys in the case of unprocessed records. For convenience, ElectroDB will return these keys as composite attributes, but you can pass the [execution option](https://electrodb.dev/en/queries/batch-get/#execution-options) {unprocessed:"raw"} override this behavior and return the Keys as they came from DynamoDB.

When using the default options, I am not seeing this behavior when inspecting unprocessed results, the records contain all the attributes of the item passed into the batch put call, not just the composite attributes.

Ideally I want to just pass the unprocessed items back into entity.put() to re-process with some back-off retry logic, which is a pattern we use with the native dynamodb client as recommended by their documentation "the UnprocessedItems value is in the same form as RequestItems, so you can provide this value directly to a subsequent BatchWriteItem operation"
https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchWriteItem.html#API_BatchWriteItem_ResponseSyntax

This is not so straightforward with ElectroDB as the response type is AllTableIndexCompositeAttributes<>[], currently I have to do some filtering on the original items to filter out the unprocessed items based on the comp attributes.

I've also seen the 'raw' processing option but the type is still AllTableIndexCompositeAttributes<>[].

So to summarize my points:

  1. Shouldn't the default behavior and type of unprocessed records be of PutItem type? Which would be in keeping with the response type of the DynamoDB client. This would allow user to pass the items back into another put() to retry.
  2. If the type must be AllTableIndexCompositeAttributes then it seems there is a bug as the records contain more than the composite attributes.
  3. Can the unprocessed response type correctly reflect the response options, when using 'items' or 'raw'.

Many thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant