Developer and CI utility scripts for the ml-cpp project.

Fetches step-level timing data from Buildkite and indexes it into the buildkite-build-timings Elasticsearch index for anomaly detection and trend analysis.

# Backfill last 30 builds from all nightly pipelines
python3 dev-tools/ingest_build_timings.py --backfill 30

# Ingest a specific build
python3 dev-tools/ingest_build_timings.py --pipeline ml-cpp-snapshot-builds --build 5819

# Dry run (print without indexing)
python3 dev-tools/ingest_build_timings.py --pipeline ml-cpp-snapshot-builds --latest --dry-run

Each document records: pipeline, build number, branch, step key, platform, step type, duration, agent wait time, and state.

Environment variables:

• BUILDKITE_TOKEN or BUILDKITE_API_READ_TOKEN — Buildkite API read token
• ES_ENDPOINT — Elasticsearch endpoint (or ~/.elastic/serverless-endpoint)
• ES_API_KEY — Elasticsearch API key (or ~/.elastic/serverless-api-key)

In CI, credentials are injected from Vault by the post-checkout hook.

Compares step durations from a PR build against a 30-day rolling baseline from nightly builds. Flags steps whose duration exceeds the baseline mean + 2σ.

# Check a specific build
python3 dev-tools/check_build_regression.py --pipeline ml-cpp-pr-builds --build 1234

# In CI (uses BUILDKITE_PIPELINE_SLUG and BUILDKITE_BUILD_NUMBER)
python3 dev-tools/check_build_regression.py --annotate

Exits with code 1 if regressions are detected. With --annotate, posts results as a Buildkite annotation.

Environment variables: same as ingest_build_timings.py.

Runs Elasticsearch Java integration tests against locally-built ml-cpp artifacts. Used by the CI pipeline's Java IT steps.

Arguments: $1 = parent directory for ES clone, $2 = path to local Ivy repo.

Includes Gradle build cache restore/upload via GCS when GRADLE_BUILD_CACHE_GCS_BUCKET is set.

Downloads and configures sccache with a GCS backend for CI builds. See SCCACHE_SETUP.md for full setup details.

Configures sccache for local development (without GCS).

Gradle init script that enables the local build cache for Java integration test builds. Injected via --init-script in run_es_tests.sh.

Docker wrappers for Linux builds, tests, and clang-format style checking.

Run clang-format and style validation.

Strips debug symbols from built binaries.

Measure build times with various configurations.

Windows-specific dependency management.

Generates release notes from git history.

Copies secrets between Vault paths.