Skip to content

Commit b0d164d

Browse files
pkwarrenpkwarren
committed
Create a small JMH benchmark harness
In order to verify performance fixes (like #451 and #454), it would be helpful to be able to write a JMH benchmark and quickly measure the impact. This adds a benchmark module to the Gradle build. CI only verifies the build of the benchmarks. Added a README showing how to add benchmarks and measure improvements locally. Instead of outputting JSON or other format, created a jq script to roughly print out output similar to benchstat for Go.
1 parent ebfc081 commit b0d164d

10 files changed

Lines changed: 470 additions & 0 deletions

File tree

‎benchmarks/README.md‎

Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
# Benchmarks
2+
3+
JMH microbenchmarks for protovalidate-java.
4+
Used locally to quantify performance changes.
5+
Not executed in CI; `./gradlew build` only verifies that benchmark code compiles.
6+
7+
## Prerequisites
8+
9+
- JDK 21
10+
- `buf` CLI (installed automatically by Gradle)
11+
- `jq` and `column` (preinstalled on macOS)
12+
13+
## Running benchmarks
14+
15+
Run all benchmarks:
16+
17+
```
18+
./gradlew :benchmarks:jmh
19+
```
20+
21+
Filter to a subset via `-Pbench` (accepts a regex over method names):
22+
23+
```
24+
./gradlew :benchmarks:jmh -Pbench=validateSimple # one method
25+
./gradlew :benchmarks:jmh -Pbench='compile.*' # prefix match
26+
./gradlew :benchmarks:jmh -Pbench='validate.*' # all steady-state
27+
```
28+
29+
Results land in `build/results/jmh/results.json`.
30+
31+
## Comparing before and after a change
32+
33+
Typical A/B workflow:
34+
35+
```
36+
# 1. run baseline on the current tree and save it
37+
./gradlew :benchmarks:jmh -Pbench='compile.*' :benchmarks:jmhSaveBaseline
38+
39+
# 2. apply your change (edit code, or gh pr checkout <N>)
40+
41+
# 3. re-run and diff against the saved baseline
42+
./gradlew :benchmarks:jmh -Pbench='compile.*' :benchmarks:jmhCompare
43+
```
44+
45+
Output:
46+
47+
```
48+
benchmark metric before after delta
49+
compileValidatorForRepeated time 4696209.43 ns/op 1064942.21 ns/op -77.3%
50+
compileValidatorForRepeated alloc 12950196.95 B/op 3262651.61 B/op -74.8%
51+
```
52+
53+
`jmhSaveBaseline` copies the current `results.json` to `results-before.json`.
54+
`jmhCompare` diffs `results-before.json` against `results.json` by default.
55+
Pass explicit paths with `-Pbefore=<path> -Pafter=<path>`.
56+
57+
## Adding a new benchmark
58+
59+
Benchmarks live in `src/jmh/java/...` and target proto messages in `src/jmh/proto/...`.
60+
61+
### 1. Define (or reuse) a proto message
62+
63+
Edit `src/jmh/proto/bench/v1/bench.proto` to add a message that exercises the code path you want to measure.
64+
`buf generate` runs automatically before `compileJmhJava`, so no separate codegen step is needed.
65+
66+
### 2. Add a `@Benchmark` method
67+
68+
Edit `src/jmh/java/build/buf/protovalidate/benchmarks/ValidationBenchmark.java`.
69+
Put one-time state (validator, messages) in `@Setup` and the measured work in the `@Benchmark` method.
70+
71+
Steady-state (hot-path) pattern:
72+
73+
```java
74+
@Benchmark
75+
public void validateMyMessage(Blackhole bh) throws ValidationException {
76+
bh.consume(validator.validate(myMessage));
77+
}
78+
```
79+
80+
Cold/compile-path pattern (each iteration builds a fresh validator):
81+
82+
```java
83+
@Benchmark
84+
@OutputTimeUnit(TimeUnit.MILLISECONDS)
85+
public void compileValidatorForMyMessage(Blackhole bh) throws CompilationException {
86+
Validator v = ValidatorFactory.newBuilder()
87+
.buildWithDescriptors(Collections.singletonList(MyMessage.getDescriptor()), false);
88+
bh.consume(v);
89+
}
90+
```
91+
92+
Choose based on what the change you want to measure actually touches.
93+
`EvaluatorBuilder` caches compiled evaluators per descriptor, so after the first `validate()` call, further calls skip compilation.
94+
If your fix is in the compile path (e.g. `RuleCache`, `DescriptorCacheBuilder`), a steady-state benchmark will not show the effect because `@Setup` absorbs it.
95+
96+
## Configuration
97+
98+
`build.gradle.kts` holds the JMH plugin config.
99+
Defaults are tuned for fast local iteration (~30s per benchmark):
100+
101+
- 3 warmup iterations of 2s each
102+
- 5 measurement iterations of 2s each
103+
- 2 forks
104+
- Average-time mode, nanoseconds
105+
- GC profiler on (`gc.alloc.rate.norm` for per-op allocations)
106+
107+
For higher-confidence numbers (tighter confidence intervals, useful for deltas under ~10%), bump `fork`, `warmup`, and `timeOnIteration` in the `jmh {}` block.
108+
Expect ~5 min per benchmark at `fork=5, warmup=5s, timeOnIteration=5s`.
109+
110+
## Metrics
111+
112+
Each benchmark emits:
113+
114+
- **Primary:** average time per `@Benchmark` invocation (`ns/op` by default).
115+
- **Secondary (GC profiler):**
116+
- `gc.alloc.rate.norm` - bytes allocated per op; deterministic, used by `jmhCompare`.
117+
- `gc.alloc.rate` - allocation rate in MB/sec; varies with CPU.
118+
- `gc.count` / `gc.time` - GC activity during the run.
119+
120+
For allocation flame graphs, uncomment the `async` profiler line in `build.gradle.kts`.
121+
Requires `async-profiler` installed locally.

‎benchmarks/buf.gen.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
version: v2
2+
plugins:
3+
- remote: buf.build/protocolbuffers/java:$protocJavaPluginVersion
4+
out: build/generated/sources/bufgen
5+
inputs:
6+
- directory: src/jmh/proto

‎benchmarks/buf.lock‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# Generated by buf. DO NOT EDIT.
2+
version: v2
3+
deps:
4+
- name: buf.build/bufbuild/protovalidate
5+
commit: 50325440f8f24053b047484a6bf60b76
6+
digest: b5:74cb6f5c0853c3c10aafc701614194bbd63326bdb8ef4068214454b8894b03ba4113e04b3a33a8321cdf05336e37db4dc14a5e2495db8462566914f36086ba31

‎benchmarks/buf.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
version: v2
2+
modules:
3+
- path: src/jmh/proto
4+
deps:
5+
- buf.build/bufbuild/protovalidate

‎benchmarks/build.gradle.kts‎

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
plugins {
2+
java
3+
alias(libs.plugins.osdetector)
4+
id("me.champeau.jmh") version "0.7.2"
5+
}
6+
7+
// JMH can use modern bytecode; benchmarks aren't shipped.
8+
java {
9+
sourceCompatibility = JavaVersion.VERSION_21
10+
targetCompatibility = JavaVersion.VERSION_21
11+
}
12+
13+
val buf: Configuration by configurations.creating
14+
15+
tasks.register("configureBuf") {
16+
description = "Installs the Buf CLI."
17+
File(buf.asPath).setExecutable(true)
18+
}
19+
20+
tasks.register<Copy>("filterBufGenYaml") {
21+
from(files("buf.gen.yaml"))
22+
includeEmptyDirs = false
23+
into(layout.buildDirectory.dir("buf-gen-templates"))
24+
expand("protocJavaPluginVersion" to "v${libs.versions.protobuf.get().substringAfter('.')}")
25+
filteringCharset = "UTF-8"
26+
}
27+
28+
tasks.register<Exec>("generateBenchmarkSources") {
29+
dependsOn("configureBuf", "filterBufGenYaml")
30+
description = "Generates Java sources for benchmark protos via buf generate."
31+
val template = layout.buildDirectory.file("buf-gen-templates/buf.gen.yaml")
32+
inputs.files(buf)
33+
inputs.dir("src/jmh/proto")
34+
inputs.file("buf.yaml")
35+
inputs.file(template)
36+
outputs.dir(layout.buildDirectory.dir("generated/sources/bufgen"))
37+
commandLine(buf.asPath, "generate", "--template", template.get().asFile.absolutePath)
38+
}
39+
40+
sourceSets {
41+
named("jmh") {
42+
java {
43+
srcDir(layout.buildDirectory.dir("generated/sources/bufgen"))
44+
}
45+
}
46+
}
47+
48+
tasks.matching { it.name == "compileJmhJava" }.configureEach {
49+
dependsOn("generateBenchmarkSources")
50+
}
51+
52+
// Ensure `./gradlew build` (and `make build`) compiles the JMH sources so CI
53+
// catches breakages in benchmark code. Execution remains gated behind the
54+
// explicit `:benchmarks:jmh` task.
55+
tasks.named("build") {
56+
dependsOn("compileJmhJava")
57+
}
58+
59+
dependencies {
60+
jmhImplementation(project(":"))
61+
jmhImplementation(libs.protobuf.java)
62+
buf("build.buf:buf:${libs.versions.buf.get()}:${osdetector.classifier}@exe")
63+
}
64+
65+
// Benchmarks produce fresh timing data each run; disable Gradle's up-to-date
66+
// check so the task always executes (otherwise -Pbench changes are ignored).
67+
tasks.named("jmh") {
68+
outputs.upToDateWhen { false }
69+
}
70+
71+
jmh {
72+
// Defaults tuned for fast local A/B runs (~90s total).
73+
// For higher-confidence numbers bump iteration time and fork count.
74+
warmupIterations.set(3)
75+
warmup.set("2s")
76+
iterations.set(5)
77+
timeOnIteration.set("2s")
78+
fork.set(2)
79+
timeUnit.set("ns")
80+
benchmarkMode.set(listOf("avgt"))
81+
resultFormat.set("JSON")
82+
// GC profiler reports bytes allocated per op (gc.alloc.rate.norm), which
83+
// jmhCompare can diff alongside timing. ~5-10% overhead on timings.
84+
profilers.set(listOf("gc"))
85+
// For allocation flame graphs (requires async-profiler installed locally):
86+
// profilers.set(listOf("async:event=alloc;output=flamegraph;dir=build/reports/jmh/async"))
87+
88+
// Filter to a subset of benchmarks via `-Pbench=<regex>`. Example:
89+
// ./gradlew :benchmarks:jmh -Pbench=validateSimple
90+
// ./gradlew :benchmarks:jmh -Pbench='compile.*'
91+
project.findProperty("bench")?.toString()?.let {
92+
includes.set(listOf(it))
93+
}
94+
}
95+
96+
val jmhResults = layout.buildDirectory.file("results/jmh/results.json")
97+
val jmhBaseline = layout.buildDirectory.file("results/jmh/results-before.json")
98+
99+
// Saves the latest JMH results.json as the baseline for jmhCompare.
100+
//
101+
// Usage:
102+
// ./gradlew :benchmarks:jmh :benchmarks:jmhSaveBaseline
103+
// # apply change...
104+
// ./gradlew :benchmarks:jmh :benchmarks:jmhCompare
105+
tasks.register<Copy>("jmhSaveBaseline") {
106+
description = "Copies the latest JMH results.json to results-before.json as the baseline."
107+
from(jmhResults)
108+
into(jmhResults.get().asFile.parentFile)
109+
rename { "results-before.json" }
110+
mustRunAfter("jmh")
111+
}
112+
113+
// Diffs two JMH results.json files as a concise benchstat-style table.
114+
// Defaults to comparing results-before.json (written by jmhSaveBaseline)
115+
// against the latest results.json.
116+
//
117+
// Override paths:
118+
// ./gradlew :benchmarks:jmhCompare -Pbefore=a.json -Pafter=b.json
119+
tasks.register<Exec>("jmhCompare") {
120+
description = "Diffs two JMH result JSON files as a concise table."
121+
val before = project.findProperty("before")?.toString()
122+
?: jmhBaseline.get().asFile.absolutePath
123+
val after = project.findProperty("after")?.toString()
124+
?: jmhResults.get().asFile.absolutePath
125+
val jqScript = file("jmh-compare.jq").absolutePath
126+
commandLine(
127+
"bash", "-c",
128+
"jq --slurp --raw-output --from-file \"\$1\" \"\$2\" \"\$3\" | column -t -s \$'\\t'",
129+
"jmh-compare", // $0
130+
jqScript, // $1
131+
before, // $2
132+
after, // $3
133+
)
134+
}

‎benchmarks/jmh-compare.jq‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
def pct(a; b):
2+
if a == null or b == null or b == 0 then "~"
3+
else (((a - b) / b * 100) * 10 | round / 10) as $d
4+
| if $d > 0 then "+\($d)%" elif $d == 0 then "~" else "\($d)%" end
5+
end;
6+
def num(x):
7+
if x == null then "-"
8+
else (x * 100 | round / 100 | tostring)
9+
end;
10+
11+
def extract: map({
12+
key: (.benchmark | split(".") | last),
13+
time: .primaryMetric.score,
14+
time_unit: .primaryMetric.scoreUnit,
15+
alloc: (.secondaryMetrics["·gc.alloc.rate.norm"].score // null)
16+
});
17+
18+
(.[0] | extract) as $b
19+
| (.[1] | extract) as $a
20+
| (["benchmark", "metric", "before", "after", "delta"] | @tsv),
21+
($b[] | . as $bi
22+
| ($a[] | select(.key == $bi.key)) as $ai
23+
| ([$bi.key, "time", "\(num($bi.time)) \($bi.time_unit)", "\(num($ai.time)) \($ai.time_unit)", pct($ai.time; $bi.time)] | @tsv),
24+
([$bi.key, "alloc", "\(num($bi.alloc)) B/op", "\(num($ai.alloc)) B/op", pct($ai.alloc; $bi.alloc)] | @tsv))

0 commit comments

Comments
 (0)