Skip to content

fix: clarify benchmark regression report verdicts - #2394

Merged
zeitlinger merged 3 commits into
prometheus:mainfrom
zeitlinger:agent/benchmark-report-verdict
Aug 19, 2026
Merged

zeitlinger merged 3 commits into
prometheus:mainfrom
zeitlinger:agent/benchmark-report-verdict

Conversation

@zeitlinger

Copy link
Copy Markdown
Member

Summary

  • Add an explicit base-vs-head regression verdict using comparable JMH metadata, confidence-interval overlap, and a practical 5% threshold.
  • Remove cross-method Nx slower rankings from the PR-head tables.
  • List head-only benchmarks separately as descriptive results with no regression verdict.

Motivation

The #2329 benchmark report described HistogramBenchmark.openTelemetryExponential as “20x slower” only because it was ranked against a different benchmark method in the same run. Its actual base/head delta was -5.7% with overlapping confidence intervals (within noise). This change makes that distinction explicit and prevents the within-run ranking from being mistaken for a regression.

Tests

  • python3 .mise/tasks/test_generate-benchmark-summary.py
  • python3 .mise/tasks/test_update-benchmarks.py
  • ruff check and ruff format --check on the changed Python files
  • mise run lint (full lint is blocked only by the pre-existing README Slack link returning HTTP 403; the pre-push scoped lint passed)

Related: #2329.

Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>
Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>
Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>
@zeitlinger
zeitlinger marked this pull request as ready for review August 19, 2026 09:27
@zeitlinger
zeitlinger merged commit e5fa067 into prometheus:main Aug 19, 2026
18 checks passed
@zeitlinger
zeitlinger deleted the agent/benchmark-report-verdict branch August 19, 2026 11:01
zeitlinger pushed a commit that referenced this pull request Sep 16, 2026
🤖 I have created a release *beep* *boop*
---


##
[1.9.0](v1.8.0...v1.9.0)
(2026-09-16)


### Features

* support metric name filtering in OpenTelemetry exporter
([#2344](#2344))
([9b0ede8](9b0ede8))


### Bug Fixes

* avoid protobuf debug reflection in native images
([#2251](#2251))
([7f899e7](7f899e7))
* bound HTTPServer request resources
([#2333](#2333))
([33ec556](33ec556))
* bound observation buffering during collection
([#2336](#2336))
([43788f5](43788f5))
* bound scrape query parameters
([#2334](#2334))
([27e1912](27e1912))
* **ci:** skip benchmark report for skipped runs
([#2422](#2422))
([40eddb0](40eddb0))
* clarify benchmark regression report verdicts
([#2394](#2394))
([e5fa067](e5fa067))
* **deps:** update dependency com.google.guava:guava to v33.7.0-jre
([#2387](#2387))
([bf0db49](bf0db49))
* **deps:** update dependency io.dropwizard.metrics:metrics-core to
v4.2.40 ([#2432](#2432))
([dd88326](dd88326))
* **deps:** update dependency io.dropwizard.metrics5:metrics-core to
v5.0.8 ([#2433](#2433))
([42f3c8a](42f3c8a))
* **deps:** update dependency
io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom-alpha
to v2.29.0-alpha
([#2235](#2235))
([cf9f702](cf9f702))
* **deps:** update dependency
io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom-alpha
to v2.30.0-alpha
([#2328](#2328))
([1ca2716](1ca2716))
* **deps:** update dependency
io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom-alpha
to v2.30.0-alpha
([#2330](#2330))
([07623c1](07623c1))
* **deps:** update dependency
io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom-alpha
to v2.31.0-alpha
([#2401](#2401))
([6c26619](6c26619))
* **deps:** update dependency
io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom-alpha
to v2.31.0-alpha
([#2402](#2402))
([ac0d68a](ac0d68a))
* **deps:** update dependency
io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom-alpha
to v2.31.1-alpha
([#2409](#2409))
([5eea652](5eea652))
* **deps:** update dependency
io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom-alpha
to v2.31.1-alpha
([#2410](#2410))
([0bcef89](0bcef89))
* **deps:** update dependency org.apache.tomcat.embed:tomcat-embed-core
to v11.0.23
([#2241](#2241))
([a017f80](a017f80))
* **deps:** update dependency org.apache.tomcat.embed:tomcat-embed-core
to v11.0.24
([#2294](#2294))
([63967bd](63967bd))
* **deps:** update dependency org.apache.tomcat.embed:tomcat-embed-core
to v11.0.25
([#2389](#2389))
([92f8344](92f8344))
* **deps:** update dependency org.apache.tomcat.embed:tomcat-embed-core
to v11.0.26
([#2477](#2477))
([05146c0](05146c0))
* **deps:** update dependency
org.springframework.boot:spring-boot-starter-parent to v4.1.1
([#2399](#2399))
([a0b0880](a0b0880))
* **deps:** update jetty monorepo to v12.1.11
([#2279](#2279))
([4dc54da](4dc54da))
* **deps:** update jetty monorepo to v12.1.12
([#2371](#2371))
([08967e0](08967e0))
* **deps:** update jetty monorepo to v12.1.13
([#2459](#2459))
([b217f05](b217f05))
* **deps:** update junit-framework monorepo to v6.1.2
([#2300](#2300))
([5966d1d](5966d1d))
* **deps:** update junit-framework monorepo to v6.1.3
([#2374](#2374))
([d1ade52](d1ade52))
* **deps:** update otel.instrumentation.version
([#2236](#2236))
([158230d](158230d))
* **deps:** update protobuf
([#2400](#2400))
([e2db1ed](e2db1ed))
* **deps:** update protobuf
([#2438](#2438))
([8ad6fa8](8ad6fa8))
* **deps:** update protobuf to v4.35.1
([#2221](#2221))
([cf17073](cf17073))
* disable micrometer compat build cache
([#2457](#2457))
([6a40eda](6a40eda))
* drop +Inf bound from OpenTelemetry classic histogram boundaries
([#2458](#2458))
([a3bce9a](a3bce9a))
* **exposition:** export internal package for OSGi resolution
([#2415](#2415))
([28b503d](28b503d))
* **httpserver:** make scrape error responses secure and configurable
([f6d9df5](f6d9df5))
* include counter names in negative value errors
([#2315](#2315))
([ea8f935](ea8f935))
* include license files in release source jars
([#2250](#2250))
([08cf925](08cf925)),
closes [#2216](#2216)
* keep late observations out of subsequent collection buffers
([#2471](#2471))
([d78b149](d78b149))
* keep PR title check required after rebases
([#2414](#2414))
([e3d4c3b](e3d4c3b))
* prevent buffer stripe index overflow
([#2331](#2331))
([b6cd000](b6cd000))
* redact invalid configuration values
([#2335](#2335))
([7e7e533](7e7e533))
* show uncertainty in benchmark comparisons
([#2476](#2476))
([398d087](398d087))
* stabilize OpenTelemetry exporter builder API
([#2257](#2257))
([09e6e2d](09e6e2d))
* Summary quantiles collapsing for targeted quantiles with 2*epsilon
&gt;= 1-quantile
([#2396](#2396))
([9c7479f](9c7479f))
* update component-prefixed action tags
([#2419](#2419))
([4acf481](4acf481))


### Performance Improvements

* skip snapshot rebuild in mergeDuplicates when names are unique
([#2441](#2441))
([8722230](8722230))


### Documentation

* add API design guideline to contributing docs
([#2350](#2350))
([23ae29a](23ae29a))
* document scrape query limits in request API
([#2391](#2391))
([ba8f5eb](ba8f5eb))
* document semantic PR title guidance
([#2318](#2318))
([5e813a0](5e813a0))

---
> [!IMPORTANT]
> Close and reopen this PR to trigger CI checks.

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants