Skip to content

Commit c1a4839

Browse files
committed
Refine .NET query benchmarks and optimize dynamic streaming
1 parent 808828f commit c1a4839

10 files changed

Lines changed: 431 additions & 111 deletions

File tree

README.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,9 +26,9 @@ Keep `MiniExcel-Rust` and the [.NET MiniExcel repository](https://github.com/min
2626
pwsh ./scripts/compare-dotnet-v1-rust.ps1 -DotNetRepository D:\git\MiniExcel
2727
```
2828

29-
The script resolves and archives `v1.x-maintenance` from the local .NET repository without changing its checkout. It compares .NET v1 `MiniExcel.Query(..., useHeaderRow: false)` with Rust `MiniExcel::query` against the same 100,000-row, 10-column XLSX workbook. Release builds are warmed up, then run in alternating order for five independent iterations with three full query passes per process. The harness verifies matching row counts and reports median elapsed time, throughput, and sampled peak working set. Process startup and .NET JIT time are included; save performance is not.
29+
The script resolves and archives `v1.x-maintenance` from the local .NET repository without changing its checkout. It compares .NET v1 `MiniExcel.Query(..., useHeaderRow: false)` with Rust `MiniExcel::query` against the same 100,000-row, 10-column XLSX workbook. It reports separate cold first-call and warmed steady-state results, runs five fresh processes in alternating order, and verifies matching row and cell counts. Query timing excludes process launch; peak working set covers the complete process. Save performance is not included.
3030

31-
The machine-readable report is written to `target/benchmarks/dotnet-v1-vs-rust.json`. Use `-Passes`, `-Iterations`, `-Workbook`, `-DotNetRevision`, or `-OutputJson` to change the workload. Results vary by environment, so compare only values produced on the same machine.
31+
The machine-readable report is written to `target/benchmarks/dotnet-v1-vs-rust.json`. Use `-Scenario`, `-Passes`, `-WarmupPasses`, `-Iterations`, `-Workbook`, `-DotNetRevision`, or `-OutputJson` to change the workload. Results vary by environment, so compare only values produced on the same machine.
3232

3333
See [the benchmark methodology and recorded result](docs/dotnet-v1-query-benchmark.md).
3434

README.zh-CN.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,9 +26,9 @@ MiniExcel 最低支持 Rust 1.85.0。
2626
pwsh ./scripts/compare-dotnet-v1-rust.ps1 -DotNetRepository D:\git\MiniExcel
2727
```
2828

29-
脚本会从本地 .NET 仓库解析并归档 `v1.x-maintenance`,不会切换其当前 checkout。测试使用同一份 100,000 行、10 列 XLSX 工作簿,比较 .NET v1 `MiniExcel.Query(..., useHeaderRow: false)` 与 Rust `MiniExcel::query`Release 构建预热后,每个进程完整查询三遍,共执行五轮,并逐轮交替运行顺序。脚本会校验读取行数一致,报告耗时中位数、吞吐量和采样峰值工作集。计时包含进程启动及 .NET JIT,不包含 Save 性能。
29+
脚本会从本地 .NET 仓库解析并归档 `v1.x-maintenance`,不会切换其当前 checkout。测试使用同一份 100,000 行、10 列 XLSX 工作簿,比较 .NET v1 `MiniExcel.Query(..., useHeaderRow: false)` 与 Rust `MiniExcel::query`测试分别报告首次冷态调用与进程内预热后的稳态结果,使用五个新进程并交替运行顺序,同时校验行数和单元格数一致。Query 计时不包含进程启动,峰值工作集覆盖完整进程;测试不包含 Save 性能。
3030

31-
机器可读报告写入 `target/benchmarks/dotnet-v1-vs-rust.json`。可通过 `-Passes``-Iterations``-Workbook``-DotNetRevision``-OutputJson` 调整测试。结果受运行环境影响,只应比较同一台机器产生的数据。
31+
机器可读报告写入 `target/benchmarks/dotnet-v1-vs-rust.json`。可通过 `-Scenario``-Passes``-WarmupPasses``-Iterations``-Workbook``-DotNetRevision``-OutputJson` 调整测试。结果受运行环境影响,只应比较同一台机器产生的数据。
3232

3333
参见[压力测试方法与已记录结果](docs/dotnet-v1-query-benchmark.zh-CN.md)
3434

Lines changed: 42 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,29 +1,57 @@
11
using MiniExcelLibs;
2+
using System.Diagnostics;
3+
using System.Text.Json;
24

3-
if (args.Length is < 1 or > 2)
5+
if (args.Length is < 1 or > 3)
46
{
5-
Console.Error.WriteLine("Usage: DotNetV1Query <xlsx-path> [passes]");
7+
Console.Error.WriteLine("Usage: DotNetV1Query <xlsx-path> [measured-passes] [warmup-passes]");
68
return 2;
79
}
810

9-
if (args.Length == 2 && (!int.TryParse(args[1], out var parsedPasses) || parsedPasses < 1))
11+
if (args.Length >= 2 && (!int.TryParse(args[1], out var measuredPasses) || measuredPasses < 1))
1012
{
11-
Console.Error.WriteLine("passes must be a positive integer");
13+
Console.Error.WriteLine("measured-passes must be a positive integer");
14+
return 2;
15+
}
16+
17+
if (args.Length == 3 && (!int.TryParse(args[2], out var warmupPasses) || warmupPasses < 0))
18+
{
19+
Console.Error.WriteLine("warmup-passes must be a non-negative integer");
1220
return 2;
1321
}
1422

1523
var path = Path.GetFullPath(args[0]);
16-
var passes = args.Length == 2 ? int.Parse(args[1]) : 1;
17-
long rowCount = 0;
24+
var measured = args.Length >= 2 ? int.Parse(args[1]) : 1;
25+
var warmup = args.Length == 3 ? int.Parse(args[2]) : 0;
26+
27+
RunQuery(path, warmup);
28+
GC.Collect();
29+
GC.WaitForPendingFinalizers();
30+
GC.Collect();
1831

19-
for (var pass = 0; pass < passes; pass++)
32+
var stopwatch = Stopwatch.StartNew();
33+
var (rows, cells) = RunQuery(path, measured);
34+
stopwatch.Stop();
35+
36+
Console.WriteLine(JsonSerializer.Serialize(new
37+
{
38+
Rows = rows,
39+
Cells = cells,
40+
QueryElapsedMs = stopwatch.Elapsed.TotalMilliseconds
41+
}));
42+
return 0;
43+
44+
static (long Rows, long Cells) RunQuery(string path, int passes)
2045
{
21-
foreach (var row in MiniExcel.Query(path, useHeaderRow: false))
46+
long rows = 0;
47+
long cells = 0;
48+
for (var pass = 0; pass < passes; pass++)
2249
{
23-
_ = row;
24-
rowCount++;
50+
foreach (object row in MiniExcel.Query(path, useHeaderRow: false))
51+
{
52+
rows++;
53+
cells += ((IDictionary<string, object>)row).Count;
54+
}
2555
}
26-
}
27-
28-
Console.WriteLine(rowCount);
29-
return 0;
56+
return (rows, cells);
57+
}

docs/dotnet-v1-query-benchmark.md

Lines changed: 62 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,21 @@ This benchmark compares dynamic, headerless XLSX streaming over the same workboo
1111

1212
Both runners enumerate every returned row without retaining the complete worksheet. Save performance, typed mapping, formulas, and other APIs are outside this comparison.
1313

14-
The harness starts a fresh process for each measured iteration. Each process reads the workbook three times, so process startup and .NET JIT are included but amortized across 300,000 rows. A separate warm-up process runs first to populate operating-system file caches. The five measured iterations alternate runtime order. Peak working set is sampled approximately every 10 ms.
14+
## Fairness Controls
15+
16+
- Both runners use Release builds, the same workbook, and equivalent public dynamic Query APIs.
17+
- Both runners count rows and cells. A result is rejected unless both counts match in every iteration.
18+
- An untimed preflight process for each runtime populates operating-system file caches.
19+
- Query time is measured inside each runner, excluding process launch and result serialization.
20+
- Five measured iterations use fresh processes and alternate runtime order. The median is reported.
21+
- No custom PGO, native CPU target, CPU affinity, or runtime tuning is applied.
22+
23+
Two scenarios keep startup and sustained throughput separate:
24+
25+
- **Cold:** one measured Query with no in-process warm-up. It includes first-call library initialization and .NET JIT, but not process launch. This is not a cold-disk test because the preflight has warmed the operating-system cache.
26+
- **Steady:** one complete Query warm-up inside each process, followed by three measured Queries. The warm-up is outside Query timing. The .NET runner performs a full garbage collection after warm-up and before timing so warm-up garbage is not charged to the measured queries. This measures sustained in-process throughput after .NET JIT and runtime caches are warm.
27+
28+
Peak working set is sampled approximately every 10 ms over the whole process. It therefore includes runtime startup and, for the steady scenario, the warm-up pass. Total process elapsed time is retained as a secondary metric but is not used to calculate Query throughput.
1529

1630
## Environment
1731

@@ -23,33 +37,62 @@ The following result was captured on 2026-08-26:
2337
| Processor | AMD Ryzen 5 5600X 6-Core Processor, 12 logical processors |
2438
| Workbook | 100,000 rows x 10 columns, 3,563,449 bytes |
2539
| Workbook SHA-256 | `5F0997993785630C7307811387A1F6D1B07534D0A88D922B377A64E472583ED5` |
26-
| Passes per process | 3 |
40+
| Cold scenario | 0 warm-up passes, 1 measured pass |
41+
| Steady scenario | 1 warm-up pass, 3 measured passes |
2742
| Measured iterations | 5 |
2843
| .NET SDK | 10.0.103 |
2944
| .NET MiniExcel | 1.46.1, commit `8b6feb87cfd00d0802de91bfca5616ec2dd744b7` |
3045
| Rust toolchain | rustc 1.85.0 (`4d91de4e4`, 2025-02-17) |
31-
| MiniExcel Rust base revision | `36d2d27f8c078181ae46c383e47321dd3a8256bc` |
46+
| MiniExcel Rust base revision | `808828f3ce892dad8b00bda1cee370fae6451e1c` |
47+
48+
## Summary
3249

33-
## Results
50+
Both implementations returned exactly 100,000 rows and 1,000,000 cells per pass in every measured process.
3451

35-
Both implementations returned exactly 300,000 rows in every measured process.
52+
| Scenario | Runtime | Median Query time | Throughput | Median process time | Median peak working set | Maximum peak working set |
53+
| --- | --- | ---: | ---: | ---: | ---: | ---: |
54+
| Cold | .NET v1 | 1,890.76 ms | 52,889 rows/s | 1,963.50 ms | 67.22 MB | 68.12 MB |
55+
| Cold | Rust | 1,455.79 ms | 68,691 rows/s | 1,473.66 ms | 9.71 MB | 9.77 MB |
56+
| Steady | .NET v1 | 2,585.18 ms | 116,046 rows/s | 4,499.20 ms | 78.75 MB | 82.07 MB |
57+
| Steady | Rust | 4,209.62 ms | 71,265 rows/s | 5,728.73 ms | 9.91 MB | 9.94 MB |
3658

37-
| Runtime | Median elapsed | Throughput | Median peak working set | Maximum peak working set |
38-
| --- | ---: | ---: | ---: | ---: |
39-
| .NET v1 | 3,369.14 ms | 89,043 rows/s | 79.91 MB | 81.49 MB |
40-
| Rust | 3,955.76 ms | 75,839 rows/s | 9.86 MB | 9.91 MB |
59+
For the first Query in a fresh process, Rust delivered 1.30x the .NET v1 throughput, completed in 23.0% less Query time, and used 85.6% less peak working set.
60+
61+
After an in-process warm-up, Rust delivered 0.61x the .NET v1 throughput and took 62.8% more Query time. Its median peak working set remained 87.4% lower, at 9.91 MB versus 78.75 MB.
62+
63+
The result is therefore workload-dependent: Rust has lower first-call latency and substantially lower memory use here, while .NET v1 has higher sustained throughput after JIT warm-up.
64+
65+
## Cold Results
4166

4267
| Iteration | .NET v1 elapsed | .NET v1 peak | Rust elapsed | Rust peak |
4368
| ---: | ---: | ---: | ---: | ---: |
44-
| 1 | 3,513.19 ms | 81.49 MB | 3,948.10 ms | 9.79 MB |
45-
| 2 | 3,294.73 ms | 79.91 MB | 4,139.42 ms | 9.84 MB |
46-
| 3 | 3,249.22 ms | 79.73 MB | 3,939.59 ms | 9.90 MB |
47-
| 4 | 3,369.14 ms | 79.99 MB | 3,962.12 ms | 9.91 MB |
48-
| 5 | 3,513.73 ms | 79.85 MB | 3,955.76 ms | 9.86 MB |
69+
| 1 | 2,063.54 ms | 65.97 MB | 1,571.51 ms | 9.66 MB |
70+
| 2 | 1,950.17 ms | 67.22 MB | 1,397.59 ms | 9.75 MB |
71+
| 3 | 1,890.76 ms | 68.12 MB | 1,461.67 ms | 9.77 MB |
72+
| 4 | 1,816.06 ms | 67.63 MB | 1,394.83 ms | 9.71 MB |
73+
| 5 | 1,793.67 ms | 66.70 MB | 1,455.79 ms | 9.67 MB |
4974

50-
For this workload, Rust took 17.4% longer and delivered 14.8% lower throughput than .NET v1. Its median peak working set was 87.7% lower: 9.86 MB versus 79.91 MB, or about one eighth of the .NET v1 footprint.
75+
## Steady Results
5176

52-
These numbers describe one workbook and one machine, not a general performance guarantee. In particular, the process-level timing includes runtime startup, and the sampled working set includes runtime overhead. Use the harness on representative workbooks before drawing application-specific conclusions.
77+
Each timed value below covers three complete Query passes after one untimed warm-up pass.
78+
79+
| Iteration | .NET v1 elapsed | .NET v1 peak | Rust elapsed | Rust peak |
80+
| ---: | ---: | ---: | ---: | ---: |
81+
| 1 | 2,401.10 ms | 79.22 MB | 4,209.62 ms | 9.92 MB |
82+
| 2 | 2,602.98 ms | 78.44 MB | 4,128.13 ms | 9.91 MB |
83+
| 3 | 2,554.63 ms | 82.07 MB | 4,357.14 ms | 9.89 MB |
84+
| 4 | 2,766.50 ms | 78.75 MB | 4,205.49 ms | 9.94 MB |
85+
| 5 | 2,585.18 ms | 78.27 MB | 4,254.68 ms | 9.80 MB |
86+
87+
## Interpretation
88+
89+
The benchmark workbook contains 1,000,000 shared-string cells, 100,000 unique strings, and no merged ranges. Its worksheet XML expands to 39,167,847 bytes.
90+
91+
Rust currently performs a complete worksheet scan before the emitting pass, even though this workbook has a valid `<dimension>` and merged-cell filling is disabled. It also clones each shared-string value into an intermediate `Data::String` and then again into the public `CellValue::String`, clones dynamic column names for every row, and transfers each parsed row through a bounded synchronous channel. These choices preserve bounded memory and iterator ownership but add steady-state work.
92+
93+
.NET v1 can stop its preliminary extent check as soon as it reads `<dimension ref="A1:J100000">`. Its JIT cost makes the first Query slower, but later Query passes benefit from already compiled and dynamically optimized hot paths. This explains why Rust leads in the cold scenario while .NET v1 leads in the steady scenario.
94+
95+
These numbers describe one workbook and one machine, not a general performance guarantee. The harness does not pin CPU affinity or suppress all operating-system noise; the median limits the effect of outliers. Use representative workbooks before drawing application-specific conclusions.
5396

5497
## Reproduce
5598

@@ -59,4 +102,6 @@ Keep the Rust and .NET repositories in sibling directories, then run from the Ru
59102
pwsh ./scripts/compare-dotnet-v1-rust.ps1 -DotNetRepository D:\git\MiniExcel
60103
```
61104

62-
The script reads `v1.x-maintenance` from the local .NET Git repository without changing its checkout. It builds both runners in Release mode, verifies matching row counts, prints the comparison, and writes the full machine-readable report to `target/benchmarks/dotnet-v1-vs-rust.json`.
105+
The script reads `v1.x-maintenance` from the local .NET Git repository without changing its checkout. It builds both runners in Release mode, verifies matching row and cell counts, prints both scenarios, and writes the full machine-readable report to `target/benchmarks/dotnet-v1-vs-rust.json`.
106+
107+
Use `-Scenario Cold` or `-Scenario Steady` to run one scenario. `-Passes` and `-WarmupPasses` configure the steady scenario; `-Iterations` controls the number of fresh measured processes.

0 commit comments

Comments
 (0)