Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update benchmarks with latest results. #2016

Merged
merged 2 commits into from
Dec 18, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions Directory.Packages.props
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@
<PackageVersion Include="Iesi.Collections" Version="4.0.5" />
<PackageVersion Include="linq2db.SqlServer" Version="3.7.0" />
<PackageVersion Include="Microsoft.Data.SqlClient" Version="5.1.1" />
<PackageVersion Include="Microsoft.Data.Sqlite" Version="7.0.10" />
<PackageVersion Include="Microsoft.EntityFrameworkCore.SqlServer" Version="5.0.0" />
<PackageVersion Include="Microsoft.Data.Sqlite" Version="8.0.0" />
<PackageVersion Include="Microsoft.EntityFrameworkCore.SqlServer" Version="8.0.0" />
<PackageVersion Include="Microsoft.NET.Test.Sdk" Version="17.7.1" />
<PackageVersion Include="Mighty" Version="3.2.0" />
<PackageVersion Include="MySqlConnector" Version="2.2.7" />
Expand Down
104 changes: 59 additions & 45 deletions Readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,56 +156,70 @@ A key feature of Dapper is performance. The following metrics show how long it t

The benchmarks can be found in [Dapper.Tests.Performance](https://github.com/DapperLib/Dapper/tree/main/benchmarks/Dapper.Tests.Performance) (contributions welcome!) and can be run via:
```bash
dotnet run --project .\benchmarks\Dapper.Tests.Performance\ -c Release -f netcoreapp3.1 -- -f * --join
dotnet run --project .\benchmarks\Dapper.Tests.Performance\ -c Release -f net8.0 -- -f * --join
```
Output from the latest run is:
``` ini
BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.208 (2004/?/20H1)
Intel Core i7-7700HQ CPU 2.80GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=3.1.201
[Host] : .NET Core 3.1.3 (CoreCLR 4.700.20.11803, CoreFX 4.700.20.12001), X64 RyuJIT
ShortRun : .NET Core 3.1.3 (CoreCLR 4.700.20.11803, CoreFX 4.700.20.12001), X64 RyuJIT
BenchmarkDotNet v0.13.7, Windows 10 (10.0.19045.3693/22H2/2022Update)
Intel Core i7-3630QM CPU 2.40GHz (Ivy Bridge), 1 CPU, 8 logical and 4 physical cores
.NET SDK 8.0.100
[Host] : .NET 8.0.0 (8.0.23.53103), X64 RyuJIT AVX
ShortRun : .NET 8.0.0 (8.0.23.53103), X64 RyuJIT AVX

```
| ORM | Method | Return | Mean | StdDev | Error | Gen 0 | Gen 1 | Gen 2 | Allocated |
|--------------- |------------------------------ |-------- |----------:|----------:|----------:|--------:|-------:|-------:|----------:|
| Belgrade | ExecuteReader | Post | 94.46 μs | 8.115 μs | 12.268 μs | 1.7500 | 0.5000 | - | 8.42 KB |
| Hand Coded | DataTable | dynamic | 105.43 μs | 0.998 μs | 1.508 μs | 3.0000 | - | - | 9.37 KB |
| Hand Coded | SqlCommand | Post | 106.58 μs | 1.191 μs | 1.801 μs | 1.5000 | 0.7500 | 0.1250 | 7.42 KB |
| Dapper | QueryFirstOrDefault&lt;dynamic&gt; | dynamic | 119.52 μs | 1.320 μs | 2.219 μs | 3.6250 | - | - | 11.39 KB |
| Dapper | &#39;Query&lt;dynamic&gt; (buffered)&#39; | dynamic | 119.93 μs | 1.943 μs | 2.937 μs | 2.3750 | 1.0000 | 0.2500 | 11.73 KB |
| Massive | &#39;Query (dynamic)&#39; | dynamic | 120.31 μs | 1.340 μs | 2.252 μs | 2.2500 | 1.0000 | 0.1250 | 12.07 KB |
| Dapper | QueryFirstOrDefault&lt;T&gt; | Post | 121.57 μs | 1.564 μs | 2.364 μs | 1.7500 | 0.7500 | - | 11.35 KB |
| Dapper | &#39;Query&lt;T&gt; (buffered)&#39; | Post | 121.67 μs | 2.913 μs | 4.403 μs | 1.8750 | 0.8750 | - | 11.65 KB |
| PetaPoco | &#39;Fetch&lt;T&gt; (Fast)&#39; | Post | 124.91 μs | 4.015 μs | 6.747 μs | 2.0000 | 1.0000 | - | 11.5 KB |
| Mighty | Query&lt;T&gt; | Post | 125.23 μs | 2.932 μs | 4.433 μs | 2.2500 | 1.0000 | - | 12.6 KB |
| LINQ to DB | Query&lt;T&gt; | Post | 125.76 μs | 2.038 μs | 3.081 μs | 2.2500 | 0.7500 | 0.2500 | 10.62 KB |
| PetaPoco | Fetch&lt;T&gt; | Post | 127.48 μs | 4.283 μs | 6.475 μs | 2.0000 | 1.0000 | - | 12.18 KB |
| LINQ to DB | &#39;First (Compiled)&#39; | Post | 128.89 μs | 2.627 μs | 3.971 μs | 2.5000 | 0.7500 | - | 10.92 KB |
| Mighty | Query&lt;dynamic&gt; | dynamic | 129.20 μs | 2.577 μs | 3.896 μs | 2.0000 | 1.0000 | - | 12.43 KB |
| Mighty | SingleFromQuery&lt;T&gt; | Post | 129.41 μs | 2.094 μs | 3.166 μs | 2.2500 | 1.0000 | - | 12.6 KB |
| Mighty | SingleFromQuery&lt;dynamic&gt; | dynamic | 130.59 μs | 2.432 μs | 3.677 μs | 2.0000 | 1.0000 | - | 12.43 KB |
| Dapper | &#39;Contrib Get&lt;T&gt;&#39; | Post | 134.74 μs | 1.816 μs | 2.746 μs | 2.5000 | 1.0000 | 0.2500 | 12.29 KB |
| ServiceStack | SingleById&lt;T&gt; | Post | 135.01 μs | 1.213 μs | 2.320 μs | 3.0000 | 1.0000 | 0.2500 | 15.27 KB |
| LINQ to DB | First | Post | 151.87 μs | 3.826 μs | 5.784 μs | 3.0000 | 1.0000 | 0.2500 | 13.97 KB |
| EF 6 | SqlQuery | Post | 171.00 μs | 1.460 μs | 2.791 μs | 3.7500 | 1.0000 | - | 23.67 KB |
| DevExpress.XPO | GetObjectByKey&lt;T&gt; | Post | 172.36 μs | 3.758 μs | 5.681 μs | 5.5000 | 1.2500 | - | 29.06 KB |
| Dapper | &#39;Query&lt;T&gt; (unbuffered)&#39; | Post | 174.40 μs | 3.296 μs | 4.983 μs | 2.0000 | 1.0000 | - | 11.77 KB |
| Dapper | &#39;Query&lt;dynamic&gt; (unbuffered)&#39; | dynamic | 174.45 μs | 1.988 μs | 3.340 μs | 2.0000 | 1.0000 | - | 11.81 KB |
| DevExpress.XPO | FindObject&lt;T&gt; | Post | 181.76 μs | 5.554 μs | 9.333 μs | 8.0000 | - | - | 27.15 KB |
| DevExpress.XPO | Query&lt;T&gt; | Post | 189.81 μs | 4.187 μs | 8.004 μs | 10.0000 | - | - | 31.61 KB |
| EF Core | &#39;First (Compiled)&#39; | Post | 199.72 μs | 3.983 μs | 7.616 μs | 4.5000 | - | - | 13.8 KB |
| NHibernate | Get&lt;T&gt; | Post | 248.71 μs | 6.604 μs | 11.098 μs | 5.0000 | 1.0000 | - | 29.79 KB |
| EF Core | First | Post | 253.20 μs | 3.033 μs | 5.097 μs | 5.5000 | - | - | 17.7 KB |
| NHibernate | HQL | Post | 258.70 μs | 11.716 μs | 17.712 μs | 5.0000 | 1.0000 | - | 32.1 KB |
| EF Core | SqlQuery | Post | 268.89 μs | 19.349 μs | 32.516 μs | 6.0000 | - | - | 18.5 KB |
| EF 6 | First | Post | 278.46 μs | 12.094 μs | 18.284 μs | 13.5000 | - | - | 44.18 KB |
| EF Core | &#39;First (No Tracking)&#39; | Post | 280.88 μs | 8.192 μs | 13.765 μs | 3.0000 | 0.5000 | - | 19.38 KB |
| NHibernate | Criteria | Post | 304.90 μs | 2.232 μs | 4.267 μs | 11.0000 | 1.0000 | - | 60.29 KB |
| EF 6 | &#39;First (No Tracking)&#39; | Post | 316.55 μs | 7.667 μs | 11.592 μs | 8.5000 | 1.0000 | - | 50.95 KB |
| NHibernate | SQL | Post | 335.41 μs | 3.111 μs | 4.703 μs | 19.0000 | 1.0000 | - | 78.86 KB |
| NHibernate | LINQ | Post | 807.79 μs | 27.207 μs | 45.719 μs | 8.0000 | 2.0000 | - | 53.65 KB |

| ORM | Method | Return | Mean | StdDev | Error | Gen0 | Gen1 | Gen2 | Allocated |
|-------------------- |------------------------------- |------------- |----------:|----------:|----------:|--------:|-------:|-------:|----------:|
| Dapper cache impact | ExecuteParameters_Cache | Void | 96.75 us | 0.668 us | 1.010 us | 0.6250 | - | - | 2184 B |
| Dapper cache impact | QueryFirstParameters_Cache | Void | 96.86 us | 0.493 us | 0.746 us | 0.8750 | - | - | 2824 B |
| Hand Coded | SqlCommand | Post | 119.70 us | 0.706 us | 1.067 us | 1.3750 | 1.0000 | 0.1250 | 7584 B |
| Hand Coded | DataTable | dynamic | 126.64 us | 1.239 us | 1.873 us | 3.0000 | - | - | 9576 B |
| SqlMarshal | SqlCommand | Post | 132.36 us | 1.008 us | 1.523 us | 2.0000 | 1.0000 | 0.2500 | 11529 B |
| Dapper | QueryFirstOrDefault<T> | Post | 133.73 us | 1.301 us | 2.186 us | 1.7500 | 1.5000 | - | 11608 B |
| Mighty | Query<dynamic> | dynamic | 133.92 us | 1.075 us | 1.806 us | 2.0000 | 1.7500 | - | 12710 B |
| LINQ to DB | Query<T> | Post | 134.24 us | 1.068 us | 1.614 us | 1.7500 | 1.2500 | - | 10904 B |
| RepoDB | ExecuteQuery<T> | Post | 135.83 us | 1.839 us | 3.091 us | 1.7500 | 1.5000 | - | 11649 B |
| Dapper | 'Query<T> (buffered)' | Post | 136.14 us | 1.755 us | 2.653 us | 2.0000 | 1.5000 | - | 11888 B |
| Mighty | Query<T> | Post | 137.96 us | 1.485 us | 2.244 us | 2.2500 | 1.2500 | - | 12201 B |
| Dapper | QueryFirstOrDefault<dynamic> | dynamic | 139.04 us | 1.507 us | 2.279 us | 3.5000 | - | - | 11648 B |
| Mighty | SingleFromQuery<dynamic> | dynamic | 139.74 us | 2.521 us | 3.811 us | 2.0000 | 1.7500 | - | 12710 B |
| Dapper | 'Query<dynamic> (buffered)' | dynamic | 140.13 us | 1.382 us | 2.090 us | 2.0000 | 1.5000 | - | 11968 B |
| ServiceStack | SingleById<T> | Post | 140.76 us | 1.147 us | 2.192 us | 2.5000 | 1.2500 | 0.2500 | 15248 B |
| Dapper | 'Contrib Get<T>' | Post | 141.09 us | 1.394 us | 2.108 us | 2.0000 | 1.5000 | - | 12440 B |
| Mighty | SingleFromQuery<T> | Post | 141.17 us | 1.941 us | 2.935 us | 1.7500 | 1.5000 | - | 12201 B |
| Massive | 'Query (dynamic)' | dynamic | 142.01 us | 4.957 us | 7.494 us | 2.0000 | 1.5000 | - | 12342 B |
| LINQ to DB | 'First (Compiled)' | Post | 144.59 us | 1.295 us | 1.958 us | 1.7500 | 1.5000 | - | 12128 B |
| RepoDB | QueryField<T> | Post | 148.31 us | 1.742 us | 2.633 us | 2.0000 | 1.5000 | 0.5000 | 13938 B |
| Norm | 'Read<> (tuples)' | ValueTuple`8 | 148.58 us | 2.172 us | 3.283 us | 2.0000 | 1.7500 | - | 12745 B |
| Norm | 'Read<()> (named tuples)' | ValueTuple`8 | 150.60 us | 0.658 us | 1.106 us | 2.2500 | 2.0000 | 1.2500 | 14562 B |
| RepoDB | Query<T> | Post | 152.34 us | 2.164 us | 3.271 us | 2.2500 | 1.5000 | 0.2500 | 14106 B |
| RepoDB | QueryDynamic<T> | Post | 154.15 us | 4.108 us | 6.210 us | 2.2500 | 1.7500 | 0.5000 | 13930 B |
| RepoDB | QueryWhere<T> | Post | 155.90 us | 1.953 us | 3.282 us | 2.5000 | 0.5000 | - | 14858 B |
| Dapper cache impact | ExecuteNoParameters_NoCache | Void | 162.35 us | 1.584 us | 2.394 us | - | - | - | 760 B |
| Dapper cache impact | ExecuteNoParameters_Cache | Void | 162.42 us | 2.740 us | 4.142 us | - | - | - | 760 B |
| Dapper cache impact | QueryFirstNoParameters_Cache | Void | 164.35 us | 1.206 us | 1.824 us | 0.2500 | - | - | 1520 B |
| DevExpress.XPO | FindObject<T> | Post | 165.87 us | 1.012 us | 1.934 us | 8.5000 | - | - | 28099 B |
| Dapper cache impact | QueryFirstNoParameters_NoCache | Void | 173.87 us | 1.178 us | 1.781 us | 0.5000 | - | - | 1576 B |
| LINQ to DB | First | Post | 175.21 us | 2.292 us | 3.851 us | 2.0000 | 0.5000 | - | 14041 B |
| EF 6 | SqlQuery | Post | 175.36 us | 2.259 us | 3.415 us | 4.0000 | 0.7500 | - | 24209 B |
| Norm | 'Read<> (class)' | Post | 186.37 us | 1.305 us | 2.496 us | 3.0000 | 0.5000 | - | 17579 B |
| DevExpress.XPO | GetObjectByKey<T> | Post | 186.78 us | 3.407 us | 5.151 us | 4.5000 | 1.0000 | - | 30114 B |
| Dapper | 'Query<dynamic> (unbuffered)' | dynamic | 194.62 us | 1.335 us | 2.019 us | 1.7500 | 1.5000 | - | 12048 B |
| Dapper | 'Query<T> (unbuffered)' | Post | 195.01 us | 0.888 us | 1.343 us | 2.0000 | 1.5000 | - | 12008 B |
| DevExpress.XPO | Query<T> | Post | 199.46 us | 5.500 us | 9.243 us | 10.0000 | - | - | 32083 B |
| Belgrade | FirstOrDefault | Task`1 | 228.70 us | 2.181 us | 3.665 us | 4.5000 | 0.5000 | - | 20555 B |
| EF Core | 'First (Compiled)' | Post | 265.45 us | 17.745 us | 26.828 us | 2.0000 | - | - | 7521 B |
| NHibernate | Get<T> | Post | 276.02 us | 8.029 us | 12.139 us | 6.5000 | 1.0000 | - | 29885 B |
| NHibernate | HQL | Post | 277.74 us | 13.032 us | 19.703 us | 8.0000 | 1.0000 | - | 31886 B |
| NHibernate | Criteria | Post | 300.22 us | 14.908 us | 28.504 us | 13.0000 | 1.0000 | - | 57562 B |
| EF 6 | First | Post | 310.55 us | 27.254 us | 45.799 us | 13.0000 | - | - | 43309 B |
| EF Core | First | Post | 317.12 us | 1.354 us | 2.046 us | 3.5000 | - | - | 11306 B |
| EF Core | SqlQuery | Post | 322.34 us | 23.990 us | 40.314 us | 5.0000 | - | - | 18195 B |
| NHibernate | SQL | Post | 325.54 us | 3.937 us | 7.527 us | 22.0000 | 1.0000 | - | 80007 B |
| EF 6 | 'First (No Tracking)' | Post | 331.14 us | 27.760 us | 46.649 us | 12.0000 | 1.0000 | - | 50237 B |
| EF Core | 'First (No Tracking)' | Post | 337.82 us | 27.814 us | 46.740 us | 3.0000 | 1.0000 | - | 17986 B |
| NHibernate | LINQ | Post | 604.74 us | 5.549 us | 10.610 us | 10.0000 | - | - | 46061 B |
| Dapper cache impact | ExecuteParameters_NoCache | Void | 623.42 us | 3.978 us | 6.684 us | 3.0000 | 2.0000 | - | 10001 B |
| Dapper cache impact | QueryFirstParameters_NoCache | Void | 630.77 us | 3.027 us | 4.576 us | 3.0000 | 2.0000 | - | 10640 B |

Feel free to submit patches that include other ORMs - when running benchmarks, be sure to compile in Release and not attach a debugger (<kbd>Ctrl</kbd>+<kbd>F5</kbd>).

Expand Down
23 changes: 23 additions & 0 deletions benchmarks/Dapper.Tests.Performance/Benchmarks.PetaPoco.cs
Original file line number Diff line number Diff line change
@@ -1,10 +1,32 @@
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Jobs;
using PetaPoco;
using System.ComponentModel;
using System.Linq;

namespace Dapper.Tests.Performance
{
#if !NET5_0_OR_GREATER
/*
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
---> System.InvalidProgramException: Common Language Runtime detected an invalid program.
at System.Reflection.Emit.DynamicMethod.CreateDelegate(Type delegateType, Object target)
at PetaPoco.Database.PocoData.GetFactory[T](String key, Boolean ForceDateTimesToUtc, IDataReader r) in /_/benchmarks/Dapper.Tests.Performance/PetaPoco/PetaPoco.cs:line 1127
at PetaPoco.Database.Fetch[T](String sql, Object[] args) in /_/benchmarks/Dapper.Tests.Performance/PetaPoco/PetaPoco.cs:line 458
at Dapper.Tests.Performance.PetaPocoBenchmarks.FetchFast() in /_/benchmarks/Dapper.Tests.Performance/Benchmarks.PetaPoco.cs:line 38
at BenchmarkDotNet.Autogenerated.Runnable_42.WorkloadActionUnroll(Int64 invokeCount) in /_/benchmarks/Dapper.Tests.Performance/bin/Release/net8.0/5e0d07b1-6b4c-4578-a0eb-d46563cab999/5e0d07b1-6b4c-4578-a0eb-d46563cab999.notcs:line 49834
at BenchmarkDotNet.Engines.Engine.RunIteration(IterationData data)
at BenchmarkDotNet.Engines.EngineFactory.Jit(Engine engine, Int32 jitIndex, Int32 invokeCount, Int32 unrollFactor)
at BenchmarkDotNet.Engines.EngineFactory.CreateReadyToRun(EngineParameters engineParameters)
at BenchmarkDotNet.Autogenerated.Runnable_42.Run(IHost host, String benchmarkName) in /_/benchmarks/Dapper.Tests.Performance/bin/Release/net8.0/5e0d07b1-6b4c-4578-a0eb-d46563cab999/5e0d07b1-6b4c-4578-a0eb-d46563cab999.notcs:line 49238
at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)
--- End of inner exception stack trace ---
at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)
at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[] args) in /_/benchmarks/Dapper.Tests.Performance/bin/Release/net8.0/5e0d07b1-6b4c-4578-a0eb-d46563cab999/5e0d07b1-6b4c-4578-a0eb-d46563cab999.notcs:line 57
*/
[Description("PetaPoco")]
public class PetaPocoBenchmarks : BenchmarkBase
{
Expand Down Expand Up @@ -38,4 +60,5 @@ public Post FetchFast()
return _dbFast.Fetch<Post>("SELECT * from Posts where Id=@0", i).First();
}
}
#endif
}
13 changes: 13 additions & 0 deletions benchmarks/Dapper.Tests.Performance/Benchmarks.RepoDB.cs
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@
using System.Linq;
using BenchmarkDotNet.Attributes;
using RepoDb;
using RepoDb.DbHelpers;
using RepoDb.DbSettings;
using RepoDb.StatementBuilders;

namespace Dapper.Tests.Performance
{
Expand All @@ -13,6 +16,16 @@ public void Setup()
{
BaseSetup();
GlobalConfiguration.Setup().UseSqlServer();

// We need this since benchmarks using System.Data.SqlClient
var dbSetting = new SqlServerDbSetting();
DbSettingMapper
.Add<System.Data.SqlClient.SqlConnection>(dbSetting, true);
DbHelperMapper
.Add<System.Data.SqlClient.SqlConnection>(new SqlServerDbHelper(), true);
StatementBuilderMapper
.Add<System.Data.SqlClient.SqlConnection>(new SqlServerStatementBuilder(dbSetting), true);

ClassMapper.Add<Post>("Posts");
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<AssemblyName>Dapper.Tests.Performance</AssemblyName>
<Description>Dapper Core Performance Suite</Description>
<OutputType>Exe</OutputType>
<TargetFrameworks>net462;net5.0</TargetFrameworks>
<TargetFrameworks>net462;net8.0</TargetFrameworks>
<IsTestProject>false</IsTestProject>
<NoWarn>$(NoWarn);IDE0063;IDE0034;IDE0059;IDE0060</NoWarn>
</PropertyGroup>
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/Dapper.Tests.Performance/app.config
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<connectionStrings>
<add name="Main" connectionString="Data Source=.;Initial Catalog=tempdb;Integrated Security=True" providerName="System.Data.SqlClient" />
<add name="Main" connectionString="Data Source=.;Initial Catalog=tempdb;Integrated Security=True;TrustServerCertificate=True" providerName="System.Data.SqlClient" />
</connectionStrings>
</configuration>