DynamoDBContext.FromScanAsync scans more than the provided limit before returning a result

[JManou]

Describe the bug

I have the following code that should return the first 30 items, but it takes a few minutes to return the result.
Is there another way to specify the max items?

var search = DbContext!.FromScanAsync<DynamoPropertyRecord>(
            new ScanOperationConfig()
            {
                Limit = 30,
                AttributesToGet = new List<string>(){ "ShortId", "Content" },
                Select = SelectValues.SpecificAttributes
            },
            _config);

        var results = await search.GetRemainingAsync();

Expected Behavior

The result should be returned and the scan stopped once the limit is reached

Current Behavior

It looks like it scan all table and returns the last 30 items. Looking at the code of the DocumentSearch, as long as there is a LastEvaluatedKey the scan will continue. Shouldn't the GetRemainingAsync method take into account the configure limit ? It doesn't seem intuitive to use GetNextSetAsync

Reproduction Steps

var search = DbContext!.FromScanAsync<DynamoPropertyRecord>(
            new ScanOperationConfig()
            {
                Limit = 30,
                AttributesToGet = new List<string>(){ "ShortId", "Content" },
                Select = SelectValues.SpecificAttributes
            },
            _config);

        var results = await search.GetRemainingAsync();

Possible Solution

No response

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

AWSSDK.DynamoDBv2 3.7.201.12

Targeted .NET Platform

.NET 7

Operating System and version

Windows 11

[ashishdhingra]

@JManou Good afternoon.

Looking at the DynamoDB API Reference:

• Scan where it mentions that The Scan operation returns one or more items and item attributes by accessing every item in a table or a secondary index. To have DynamoDB return fewer items, you can provide a FilterExpression operation.. Please confirm if you have tried using FilterExpression property of ScanOperationConfig.

• Limit parameter mentions that it specifies the maximum number of items to evaluate (not necessarily the number of matching items).

Are you noticing different behavior using other SDK(s) or AWS CLI?

Thanks,
Ashish

[JManou]

@ashishdhingra when using the aws cli I can specify --max-items to get a limited number of elements without doing any filtering. Is it possible to achieve the same behavior using the SDK ?

[ashishdhingra]

@ashishdhingra when using the aws cli I can specify --max-items to get a limited number of elements without doing any filtering. Is it possible to achieve the same behavior using the SDK ?

@JManou Thanks for your response. Kindly advise on the following:

• How you came into conclusion that AWS SDK for .NET if not stopping after evaluating the number of items specified by Limit property.
• Do you see the delay when executing the DynamoDBContext.FromScanAsync() first time, or, delay is noticed every time this call is made?
• Have you tried using AmazonDynamoDBClient.Scan (ScanRequest)? Is the behavior different?

Thanks,
Ashish

[JManou]

@ashishdhingra

• I debugged the code and I have seen that, until the NextKey is null or empty the scan continue (cf

  aws-sdk-net/sdk/src/Services/DynamoDBv2/Custom/DocumentModel/Search.cs

  Line 356 in 8c88092

  if (NextKey == null || NextKey.Count == 0)

  )
• Yes every time

From what I understand from the doc limit is used for batching/pagination scan operation. Maybe a new attribute could be introduced in ScanOperationConfig to support --max-items as in the cli
https://docs.aws.amazon.com/cli/latest/reference/dynamodb/scan.html