Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance: MockedItemBenchmark.ReadItemExists;[Type=Stream] takes longer to run as expect in Preview mode. #4438

Open
philipthomas-MSFT opened this issue Apr 18, 2024 · 0 comments
Assignees

Comments

@philipthomas-MSFT
Copy link
Contributor

philipthomas-MSFT commented Apr 18, 2024

Description

Release pipeline is showing a larger number than expected for preview on MockedItemBenchmark.ReadItemExists;[Type=Stream]. Should be 26018.25, but 27381 (5.24%)

To Reproduce
Run command-line locally. Observe and compare results from preview and non-preview. If the numbers are off, then we must determine why there is a gap.

Command-line

Release (non preview)
dotnet.exe run --project Microsoft.Azure.Cosmos.Performance.Tests.csproj --configuration Release --no-restore --framework net6.0 --allCategories=GateBenchmarkReadItemExists -- -j Short -m --BaselineValidation
Results
Current benchmark results: {
  "MockedItemBenchmark.ReadItemExists;[Type=OfT]": "34527.75 (2.67%)",
  "MockedItemBenchmark.ReadItemExists;[Type=OfTCustom]": "34536.25 (2.68%)",
  "MockedItemBenchmark.ReadItemExists;[Type=OfTWithClientTelemetryEnabled]": "34527.75 (2.68%)",
  "MockedItemBenchmark.ReadItemExists;[Type=OfTWithDiagnosticsToString]": "56717.25 (3.04%)",
  "MockedItemBenchmark.ReadItemExists;[Type=Stream]": "26766 (2.87%)"
}
Release (preview)
dotnet.exe run --project Microsoft.Azure.Cosmos.Performance.Tests.csproj --configuration Release --no-restore --framework net6.0 --allCategories=GateBenchmarkReadItemExists -- -j Short -m --BaselineValidation /p:IsPreview=true
Results
Current benchmark results: {
  "MockedItemBenchmark.ReadItemExists;[Type=OfT]": "34529.25 (2.67%)",
  "MockedItemBenchmark.ReadItemExists;[Type=OfTCustom]": "34534 (2.67%)",
  "MockedItemBenchmark.ReadItemExists;[Type=OfTWithClientTelemetryEnabled]": "34543.25 (2.72%)",
  "MockedItemBenchmark.ReadItemExists;[Type=OfTWithDiagnosticsToString]": "56784.75 (3.16%)",
  "MockedItemBenchmark.ReadItemExists;[Type=Stream]": "26765.5 (2.87%)"
}

Expected behavior
Both preview and non-preview tests for ReadItemExists, stream, should have the same performance metrics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Blocked
Development

No branches or pull requests

2 participants