Skip to content

Commit

Permalink
Update benchmarks with latest results. (#2016)
Browse files Browse the repository at this point in the history
* Update benchmarks with latest results.

* Address PR feedback
  • Loading branch information
kant2002 committed Dec 18, 2023
1 parent 3c3509f commit 4160c9f
Show file tree
Hide file tree
Showing 6 changed files with 99 additions and 49 deletions.
4 changes: 2 additions & 2 deletions Directory.Packages.props
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@
<PackageVersion Include="Iesi.Collections" Version="4.0.5" />
<PackageVersion Include="linq2db.SqlServer" Version="3.7.0" />
<PackageVersion Include="Microsoft.Data.SqlClient" Version="5.1.1" />
<PackageVersion Include="Microsoft.Data.Sqlite" Version="7.0.10" />
<PackageVersion Include="Microsoft.EntityFrameworkCore.SqlServer" Version="5.0.0" />
<PackageVersion Include="Microsoft.Data.Sqlite" Version="8.0.0" />
<PackageVersion Include="Microsoft.EntityFrameworkCore.SqlServer" Version="8.0.0" />
<PackageVersion Include="Microsoft.NET.Test.Sdk" Version="17.7.1" />
<PackageVersion Include="Mighty" Version="3.2.0" />
<PackageVersion Include="MySqlConnector" Version="2.2.7" />
Expand Down
104 changes: 59 additions & 45 deletions Readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,56 +156,70 @@ A key feature of Dapper is performance. The following metrics show how long it t

The benchmarks can be found in [Dapper.Tests.Performance](https://github.com/DapperLib/Dapper/tree/main/benchmarks/Dapper.Tests.Performance) (contributions welcome!) and can be run via:
```bash
dotnet run --project .\benchmarks\Dapper.Tests.Performance\ -c Release -f netcoreapp3.1 -- -f * --join
dotnet run --project .\benchmarks\Dapper.Tests.Performance\ -c Release -f net8.0 -- -f * --join
```
Output from the latest run is:
``` ini
BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.208 (2004/?/20H1)
Intel Core i7-7700HQ CPU 2.80GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=3.1.201
[Host] : .NET Core 3.1.3 (CoreCLR 4.700.20.11803, CoreFX 4.700.20.12001), X64 RyuJIT
ShortRun : .NET Core 3.1.3 (CoreCLR 4.700.20.11803, CoreFX 4.700.20.12001), X64 RyuJIT
BenchmarkDotNet v0.13.7, Windows 10 (10.0.19045.3693/22H2/2022Update)
Intel Core i7-3630QM CPU 2.40GHz (Ivy Bridge), 1 CPU, 8 logical and 4 physical cores
.NET SDK 8.0.100
[Host] : .NET 8.0.0 (8.0.23.53103), X64 RyuJIT AVX
ShortRun : .NET 8.0.0 (8.0.23.53103), X64 RyuJIT AVX

```
| ORM | Method | Return | Mean | StdDev | Error | Gen 0 | Gen 1 | Gen 2 | Allocated |
|--------------- |------------------------------ |-------- |----------:|----------:|----------:|--------:|-------:|-------:|----------:|
| Belgrade | ExecuteReader | Post | 94.46 μs | 8.115 μs | 12.268 μs | 1.7500 | 0.5000 | - | 8.42 KB |
| Hand Coded | DataTable | dynamic | 105.43 μs | 0.998 μs | 1.508 μs | 3.0000 | - | - | 9.37 KB |
| Hand Coded | SqlCommand | Post | 106.58 μs | 1.191 μs | 1.801 μs | 1.5000 | 0.7500 | 0.1250 | 7.42 KB |
| Dapper | QueryFirstOrDefault&lt;dynamic&gt; | dynamic | 119.52 μs | 1.320 μs | 2.219 μs | 3.6250 | - | - | 11.39 KB |
| Dapper | &#39;Query&lt;dynamic&gt; (buffered)&#39; | dynamic | 119.93 μs | 1.943 μs | 2.937 μs | 2.3750 | 1.0000 | 0.2500 | 11.73 KB |
| Massive | &#39;Query (dynamic)&#39; | dynamic | 120.31 μs | 1.340 μs | 2.252 μs | 2.2500 | 1.0000 | 0.1250 | 12.07 KB |
| Dapper | QueryFirstOrDefault&lt;T&gt; | Post | 121.57 μs | 1.564 μs | 2.364 μs | 1.7500 | 0.7500 | - | 11.35 KB |
| Dapper | &#39;Query&lt;T&gt; (buffered)&#39; | Post | 121.67 μs | 2.913 μs | 4.403 μs | 1.8750 | 0.8750 | - | 11.65 KB |
| PetaPoco | &#39;Fetch&lt;T&gt; (Fast)&#39; | Post | 124.91 μs | 4.015 μs | 6.747 μs | 2.0000 | 1.0000 | - | 11.5 KB |
| Mighty | Query&lt;T&gt; | Post | 125.23 μs | 2.932 μs | 4.433 μs | 2.2500 | 1.0000 | - | 12.6 KB |
| LINQ to DB | Query&lt;T&gt; | Post | 125.76 μs | 2.038 μs | 3.081 μs | 2.2500 | 0.7500 | 0.2500 | 10.62 KB |
| PetaPoco | Fetch&lt;T&gt; | Post | 127.48 μs | 4.283 μs | 6.475 μs | 2.0000 | 1.0000 | - | 12.18 KB |
| LINQ to DB | &#39;First (Compiled)&#39; | Post | 128.89 μs | 2.627 μs | 3.971 μs | 2.5000 | 0.7500 | - | 10.92 KB |
| Mighty | Query&lt;dynamic&gt; | dynamic | 129.20 μs | 2.577 μs | 3.896 μs | 2.0000 | 1.0000 | - | 12.43 KB |
| Mighty | SingleFromQuery&lt;T&gt; | Post | 129.41 μs | 2.094 μs | 3.166 μs | 2.2500 | 1.0000 | - | 12.6 KB |
| Mighty | SingleFromQuery&lt;dynamic&gt; | dynamic | 130.59 μs | 2.432 μs | 3.677 μs | 2.0000 | 1.0000 | - | 12.43 KB |
| Dapper | &#39;Contrib Get&lt;T&gt;&#39; | Post | 134.74 μs | 1.816 μs | 2.746 μs | 2.5000 | 1.0000 | 0.2500 | 12.29 KB |
| ServiceStack | SingleById&lt;T&gt; | Post | 135.01 μs | 1.213 μs | 2.320 μs | 3.0000 | 1.0000 | 0.2500 | 15.27 KB |
| LINQ to DB | First | Post | 151.87 μs | 3.826 μs | 5.784 μs | 3.0000 | 1.0000 | 0.2500 | 13.97 KB |
| EF 6 | SqlQuery | Post | 171.00 μs | 1.460 μs | 2.791 μs | 3.7500 | 1.0000 | - | 23.67 KB |
| DevExpress.XPO | GetObjectByKey&lt;T&gt; | Post | 172.36 μs | 3.758 μs | 5.681 μs | 5.5000 | 1.2500 | - | 29.06 KB |
| Dapper | &#39;Query&lt;T&gt; (unbuffered)&#39; | Post | 174.40 μs | 3.296 μs | 4.983 μs | 2.0000 | 1.0000 | - | 11.77 KB |
| Dapper | &#39;Query&lt;dynamic&gt; (unbuffered)&#39; | dynamic | 174.45 μs | 1.988 μs | 3.340 μs | 2.0000 | 1.0000 | - | 11.81 KB |
| DevExpress.XPO | FindObject&lt;T&gt; | Post | 181.76 μs | 5.554 μs | 9.333 μs | 8.0000 | - | - | 27.15 KB |
| DevExpress.XPO | Query&lt;T&gt; | Post | 189.81 μs | 4.187 μs | 8.004 μs | 10.0000 | - | - | 31.61 KB |
| EF Core | &#39;First (Compiled)&#39; | Post | 199.72 μs | 3.983 μs | 7.616 μs | 4.5000 | - | - | 13.8 KB |
| NHibernate | Get&lt;T&gt; | Post | 248.71 μs | 6.604 μs | 11.098 μs | 5.0000 | 1.0000 | - | 29.79 KB |
| EF Core | First | Post | 253.20 μs | 3.033 μs | 5.097 μs | 5.5000 | - | - | 17.7 KB |
| NHibernate | HQL | Post | 258.70 μs | 11.716 μs | 17.712 μs | 5.0000 | 1.0000 | - | 32.1 KB |
| EF Core | SqlQuery | Post | 268.89 μs | 19.349 μs | 32.516 μs | 6.0000 | - | - | 18.5 KB |
| EF 6 | First | Post | 278.46 μs | 12.094 μs | 18.284 μs | 13.5000 | - | - | 44.18 KB |
| EF Core | &#39;First (No Tracking)&#39; | Post | 280.88 μs | 8.192 μs | 13.765 μs | 3.0000 | 0.5000 | - | 19.38 KB |
| NHibernate | Criteria | Post | 304.90 μs | 2.232 μs | 4.267 μs | 11.0000 | 1.0000 | - | 60.29 KB |
| EF 6 | &#39;First (No Tracking)&#39; | Post | 316.55 μs | 7.667 μs | 11.592 μs | 8.5000 | 1.0000 | - | 50.95 KB |
| NHibernate | SQL | Post | 335.41 μs | 3.111 μs | 4.703 μs | 19.0000 | 1.0000 | - | 78.86 KB |
| NHibernate | LINQ | Post | 807.79 μs | 27.207 μs | 45.719 μs | 8.0000 | 2.0000 | - | 53.65 KB |

| ORM | Method | Return | Mean | StdDev | Error | Gen0 | Gen1 | Gen2 | Allocated |
|-------------------- |------------------------------- |------------- |----------:|----------:|----------:|--------:|-------:|-------:|----------:|
| Dapper cache impact | ExecuteParameters_Cache | Void | 96.75 us | 0.668 us | 1.010 us | 0.6250 | - | - | 2184 B |
| Dapper cache impact | QueryFirstParameters_Cache | Void | 96.86 us | 0.493 us | 0.746 us | 0.8750 | - | - | 2824 B |
| Hand Coded | SqlCommand | Post | 119.70 us | 0.706 us | 1.067 us | 1.3750 | 1.0000 | 0.1250 | 7584 B |
| Hand Coded | DataTable | dynamic | 126.64 us | 1.239 us | 1.873 us | 3.0000 | - | - | 9576 B |
| SqlMarshal | SqlCommand | Post | 132.36 us | 1.008 us | 1.523 us | 2.0000 | 1.0000 | 0.2500 | 11529 B |
| Dapper | QueryFirstOrDefault<T> | Post | 133.73 us | 1.301 us | 2.186 us | 1.7500 | 1.5000 | - | 11608 B |
| Mighty | Query<dynamic> | dynamic | 133.92 us | 1.075 us | 1.806 us | 2.0000 | 1.7500 | - | 12710 B |
| LINQ to DB | Query<T> | Post | 134.24 us | 1.068 us | 1.614 us | 1.7500 | 1.2500 | - | 10904 B |
| RepoDB | ExecuteQuery<T> | Post | 135.83 us | 1.839 us | 3.091 us | 1.7500 | 1.5000 | - | 11649 B |
| Dapper | 'Query<T> (buffered)' | Post | 136.14 us | 1.755 us | 2.653 us | 2.0000 | 1.5000 | - | 11888 B |
| Mighty | Query<T> | Post | 137.96 us | 1.485 us | 2.244 us | 2.2500 | 1.2500 | - | 12201 B |
| Dapper | QueryFirstOrDefault<dynamic> | dynamic | 139.04 us | 1.507 us | 2.279 us | 3.5000 | - | - | 11648 B |
| Mighty | SingleFromQuery<dynamic> | dynamic | 139.74 us | 2.521 us | 3.811 us | 2.0000 | 1.7500 | - | 12710 B |
| Dapper | 'Query<dynamic> (buffered)' | dynamic | 140.13 us | 1.382 us | 2.090 us | 2.0000 | 1.5000 | - | 11968 B |
| ServiceStack | SingleById<T> | Post | 140.76 us | 1.147 us | 2.192 us | 2.5000 | 1.2500 | 0.2500 | 15248 B |
| Dapper | 'Contrib Get<T>' | Post | 141.09 us | 1.394 us | 2.108 us | 2.0000 | 1.5000 | - | 12440 B |
| Mighty | SingleFromQuery<T> | Post | 141.17 us | 1.941 us | 2.935 us | 1.7500 | 1.5000 | - | 12201 B |
| Massive | 'Query (dynamic)' | dynamic | 142.01 us | 4.957 us | 7.494 us | 2.0000 | 1.5000 | - | 12342 B |
| LINQ to DB | 'First (Compiled)' | Post | 144.59 us | 1.295 us | 1.958 us | 1.7500 | 1.5000 | - | 12128 B |
| RepoDB | QueryField<T> | Post | 148.31 us | 1.742 us | 2.633 us | 2.0000 | 1.5000 | 0.5000 | 13938 B |
| Norm | 'Read<> (tuples)' | ValueTuple`8 | 148.58 us | 2.172 us | 3.283 us | 2.0000 | 1.7500 | - | 12745 B |
| Norm | 'Read<()> (named tuples)' | ValueTuple`8 | 150.60 us | 0.658 us | 1.106 us | 2.2500 | 2.0000 | 1.2500 | 14562 B |
| RepoDB | Query<T> | Post | 152.34 us | 2.164 us | 3.271 us | 2.2500 | 1.5000 | 0.2500 | 14106 B |
| RepoDB | QueryDynamic<T> | Post | 154.15 us | 4.108 us | 6.210 us | 2.2500 | 1.7500 | 0.5000 | 13930 B |
| RepoDB | QueryWhere<T> | Post | 155.90 us | 1.953 us | 3.282 us | 2.5000 | 0.5000 | - | 14858 B |
| Dapper cache impact | ExecuteNoParameters_NoCache | Void | 162.35 us | 1.584 us | 2.394 us | - | - | - | 760 B |
| Dapper cache impact | ExecuteNoParameters_Cache | Void | 162.42 us | 2.740 us | 4.142 us | - | - | - | 760 B |
| Dapper cache impact | QueryFirstNoParameters_Cache | Void | 164.35 us | 1.206 us | 1.824 us | 0.2500 | - | - | 1520 B |
| DevExpress.XPO | FindObject<T> | Post | 165.87 us | 1.012 us | 1.934 us | 8.5000 | - | - | 28099 B |
| Dapper cache impact | QueryFirstNoParameters_NoCache | Void | 173.87 us | 1.178 us | 1.781 us | 0.5000 | - | - | 1576 B |
| LINQ to DB | First | Post | 175.21 us | 2.292 us | 3.851 us | 2.0000 | 0.5000 | - | 14041 B |
| EF 6 | SqlQuery | Post | 175.36 us | 2.259 us | 3.415 us | 4.0000 | 0.7500 | - | 24209 B |
| Norm | 'Read<> (class)' | Post | 186.37 us | 1.305 us | 2.496 us | 3.0000 | 0.5000 | - | 17579 B |
| DevExpress.XPO | GetObjectByKey<T> | Post | 186.78 us | 3.407 us | 5.151 us | 4.5000 | 1.0000 | - | 30114 B |
| Dapper | 'Query<dynamic> (unbuffered)' | dynamic | 194.62 us | 1.335 us | 2.019 us | 1.7500 | 1.5000 | - | 12048 B |
| Dapper | 'Query<T> (unbuffered)' | Post | 195.01 us | 0.888 us | 1.343 us | 2.0000 | 1.5000 | - | 12008 B |
| DevExpress.XPO | Query<T> | Post | 199.46 us | 5.500 us | 9.243 us | 10.0000 | - | - | 32083 B |
| Belgrade | FirstOrDefault | Task`1 | 228.70 us | 2.181 us | 3.665 us | 4.5000 | 0.5000 | - | 20555 B |
| EF Core | 'First (Compiled)' | Post | 265.45 us | 17.745 us | 26.828 us | 2.0000 | - | - | 7521 B |
| NHibernate | Get<T> | Post | 276.02 us | 8.029 us | 12.139 us | 6.5000 | 1.0000 | - | 29885 B |
| NHibernate | HQL | Post | 277.74 us | 13.032 us | 19.703 us | 8.0000 | 1.0000 | - | 31886 B |
| NHibernate | Criteria | Post | 300.22 us | 14.908 us | 28.504 us | 13.0000 | 1.0000 | - | 57562 B |
| EF 6 | First | Post | 310.55 us | 27.254 us | 45.799 us | 13.0000 | - | - | 43309 B |
| EF Core | First | Post | 317.12 us | 1.354 us | 2.046 us | 3.5000 | - | - | 11306 B |
| EF Core | SqlQuery | Post | 322.34 us | 23.990 us | 40.314 us | 5.0000 | - | - | 18195 B |
| NHibernate | SQL | Post | 325.54 us | 3.937 us | 7.527 us | 22.0000 | 1.0000 | - | 80007 B |
| EF 6 | 'First (No Tracking)' | Post | 331.14 us | 27.760 us | 46.649 us | 12.0000 | 1.0000 | - | 50237 B |
| EF Core | 'First (No Tracking)' | Post | 337.82 us | 27.814 us | 46.740 us | 3.0000 | 1.0000 | - | 17986 B |
| NHibernate | LINQ | Post | 604.74 us | 5.549 us | 10.610 us | 10.0000 | - | - | 46061 B |
| Dapper cache impact | ExecuteParameters_NoCache | Void | 623.42 us | 3.978 us | 6.684 us | 3.0000 | 2.0000 | - | 10001 B |
| Dapper cache impact | QueryFirstParameters_NoCache | Void | 630.77 us | 3.027 us | 4.576 us | 3.0000 | 2.0000 | - | 10640 B |

Feel free to submit patches that include other ORMs - when running benchmarks, be sure to compile in Release and not attach a debugger (<kbd>Ctrl</kbd>+<kbd>F5</kbd>).

Expand Down
23 changes: 23 additions & 0 deletions benchmarks/Dapper.Tests.Performance/Benchmarks.PetaPoco.cs
Original file line number Diff line number Diff line change
@@ -1,10 +1,32 @@
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Jobs;
using PetaPoco;
using System.ComponentModel;
using System.Linq;

namespace Dapper.Tests.Performance
{
#if !NET5_0_OR_GREATER
/*
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
---> System.InvalidProgramException: Common Language Runtime detected an invalid program.
at System.Reflection.Emit.DynamicMethod.CreateDelegate(Type delegateType, Object target)
at PetaPoco.Database.PocoData.GetFactory[T](String key, Boolean ForceDateTimesToUtc, IDataReader r) in /_/benchmarks/Dapper.Tests.Performance/PetaPoco/PetaPoco.cs:line 1127
at PetaPoco.Database.Fetch[T](String sql, Object[] args) in /_/benchmarks/Dapper.Tests.Performance/PetaPoco/PetaPoco.cs:line 458
at Dapper.Tests.Performance.PetaPocoBenchmarks.FetchFast() in /_/benchmarks/Dapper.Tests.Performance/Benchmarks.PetaPoco.cs:line 38
at BenchmarkDotNet.Autogenerated.Runnable_42.WorkloadActionUnroll(Int64 invokeCount) in /_/benchmarks/Dapper.Tests.Performance/bin/Release/net8.0/5e0d07b1-6b4c-4578-a0eb-d46563cab999/5e0d07b1-6b4c-4578-a0eb-d46563cab999.notcs:line 49834
at BenchmarkDotNet.Engines.Engine.RunIteration(IterationData data)
at BenchmarkDotNet.Engines.EngineFactory.Jit(Engine engine, Int32 jitIndex, Int32 invokeCount, Int32 unrollFactor)
at BenchmarkDotNet.Engines.EngineFactory.CreateReadyToRun(EngineParameters engineParameters)
at BenchmarkDotNet.Autogenerated.Runnable_42.Run(IHost host, String benchmarkName) in /_/benchmarks/Dapper.Tests.Performance/bin/Release/net8.0/5e0d07b1-6b4c-4578-a0eb-d46563cab999/5e0d07b1-6b4c-4578-a0eb-d46563cab999.notcs:line 49238
at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)
--- End of inner exception stack trace ---
at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)
at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[] args) in /_/benchmarks/Dapper.Tests.Performance/bin/Release/net8.0/5e0d07b1-6b4c-4578-a0eb-d46563cab999/5e0d07b1-6b4c-4578-a0eb-d46563cab999.notcs:line 57
*/
[Description("PetaPoco")]
public class PetaPocoBenchmarks : BenchmarkBase
{
Expand Down Expand Up @@ -38,4 +60,5 @@ public Post FetchFast()
return _dbFast.Fetch<Post>("SELECT * from Posts where Id=@0", i).First();
}
}
#endif
}
13 changes: 13 additions & 0 deletions benchmarks/Dapper.Tests.Performance/Benchmarks.RepoDB.cs
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@
using System.Linq;
using BenchmarkDotNet.Attributes;
using RepoDb;
using RepoDb.DbHelpers;
using RepoDb.DbSettings;
using RepoDb.StatementBuilders;

namespace Dapper.Tests.Performance
{
Expand All @@ -13,6 +16,16 @@ public void Setup()
{
BaseSetup();
GlobalConfiguration.Setup().UseSqlServer();

// We need this since benchmarks using System.Data.SqlClient
var dbSetting = new SqlServerDbSetting();
DbSettingMapper
.Add<System.Data.SqlClient.SqlConnection>(dbSetting, true);
DbHelperMapper
.Add<System.Data.SqlClient.SqlConnection>(new SqlServerDbHelper(), true);
StatementBuilderMapper
.Add<System.Data.SqlClient.SqlConnection>(new SqlServerStatementBuilder(dbSetting), true);

ClassMapper.Add<Post>("Posts");
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<AssemblyName>Dapper.Tests.Performance</AssemblyName>
<Description>Dapper Core Performance Suite</Description>
<OutputType>Exe</OutputType>
<TargetFrameworks>net462;net5.0</TargetFrameworks>
<TargetFrameworks>net462;net8.0</TargetFrameworks>
<IsTestProject>false</IsTestProject>
<NoWarn>$(NoWarn);IDE0063;IDE0034;IDE0059;IDE0060</NoWarn>
</PropertyGroup>
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/Dapper.Tests.Performance/app.config
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<connectionStrings>
<add name="Main" connectionString="Data Source=.;Initial Catalog=tempdb;Integrated Security=True" providerName="System.Data.SqlClient" />
<add name="Main" connectionString="Data Source=.;Initial Catalog=tempdb;Integrated Security=True;TrustServerCertificate=True" providerName="System.Data.SqlClient" />
</connectionStrings>
</configuration>

0 comments on commit 4160c9f

Please sign in to comment.