experimental/air: parallel gzip for the plain_tar snapshot packer by ben-hansen-db · Pull Request #6571 · databricks/cli

ben-hansen-db · 2026-09-08T14:36:49Z

Summary

The air run plain_tar snapshot path (dirty working tree / no git ref) packages the code_source by shelling out to tar -czf. tar's built-in gzip is single-threaded, so for a large tree it dominates packaging latency.

This pipes tar -cf - (uncompressed) through klauspost/pgzip (MIT), a parallel drop-in for gzip that spreads compression across cores and still emits an ordinary gzip stream.

Level is DefaultCompression — the same level as the old tar -czf, not BestSpeed. The archive is re-uploaded on every run, so its size matters. So this is a pure latency win with no upload-size regression.
Compressing outside tar means no archive path is passed to tar, which lets us drop the Windows colon-in-path workaround the -f <path> form required.

Results (local packaging: git walk + tar + gzip; warm page cache)

Target	before (`tar -czf`, serial gz6)	after (`tar -cf -` \| pgzip, gz6)
large folder (7.4k files)	2,456 ms	716 ms (~3.4×)
gzip step alone, 470 MiB tar	7,940 ms	436 ms (~18×)

Same compression level, so the tarball is the same size as before (~24 MB for large folder). Level chosen from the curve on a 476 MiB tar: L1→L6 costs +150 ms for ~17% fewer bytes (97→80 MB); L6→L9 doubles time for ~1%, so 6 is the knee.

Validation

Output verified: gzip -t clean, entry count matches the file list, extracts correctly.
Existing snapshot_package_test.go unit tests pass; the internal/build license test passes for the new pgzip dep (// MIT in go.mod + NOTICE entry).

vinchenzo-db

lgtm

vinchenzo-db · 2026-09-08T17:50:24Z

@@ -9,6 +9,8 @@ import (
 	"os/exec"
 	"path/filepath"
 	"strings"
+


is Go semantics/lint to have this empty space? if not then rm

need to keep it according to claude:

that blank line is the standard import grouping (stdlib vs third-party) that goimports/gofumpt enforce, and this repo runs both as formatters. Without it, gofmt sorts github.com/klauspost/pgzip alphabetically into the stdlib block (between fmt and os) and goimports re-adds the separator, so ./task fmt would put it right back.

vinchenzo-db · 2026-09-08T17:51:41Z

+	// level 9 buys almost nothing beyond that for ~2x the time. Compressing outside tar
+	// also passes no archive path to tar, sidestepping the Windows colon-in-path issue a
+	// `-f <path>` argument otherwise hits (tar reads the `C:` in `C:\out\x` as a host).


Either dumb down this part or rm imo

createPlainTarball shelled out to `tar -czf`, whose gzip is single-threaded and dominates packaging time for a large code_source tree. Pipe `tar -cf -` through klauspost/pgzip instead, spreading compression across cores. Measured ~4x on universe/research (2456 ms -> 609 ms) and ~18x on the 470 MiB gzip step alone; the output is an ordinary gzip stream. Level is BestSpeed since the uploaded size does not matter for this workflow, only latency. Compressing outside tar also passes no archive path to tar, which removes the Windows colon-in-path workaround (bare `-f` basename + -C) the -czf form needed. Co-authored-by: Isaac <no-reply@databricks.com>

Now that gzip is parallel the compression level is nearly free, so trade a little CPU for a smaller upload -- the plain_tar archive is re-uploaded on every run. DefaultCompression matches the old `tar -czf` size at ~18x the speed (research: 716 ms / 24 MB, vs BestSpeed 609 ms / 27 MB). Level 9 buys ~1% fewer bytes for ~2x the time, so 6 is the knee. Co-authored-by: Isaac <no-reply@databricks.com>

Vincent found the pack-step comment too dense. Cut it from the compression-level essay (that rationale lives in the PR description and commit message) down to the two non-obvious whys: parallel gzip vs tar's single-threaded -z, and compressing outside tar to avoid the Windows colon-in-path issue. Co-authored-by: Isaac <no-reply@databricks.com>

eng-dev-ecosystem-bot · 2026-09-08T22:23:02Z

Integration test report

Commit: 660b22e

Run: 34524646247

	Env	💚RECOVERED	✅pass	🙈skip	Time
💚	aws linux	1	275	15	6:44
💚	aws windows	1	277	13	7:45
💚	azure linux	1	274	15	6:47
💚	azure windows	1	276	13	7:27
💚	gcp linux	1	275	15	7:13
💚	gcp windows	1	277	13	8:45

	Test Name	aws linux	aws windows	azure linux	azure windows	gcp linux	gcp windows
💚	TestAccept	💚R	💚R	💚R	💚R	💚R	💚R

Top 6 slowest tests (at least 2 minutes):

duration	env	testname
8:41	gcp windows	TestAccept
7:43	aws windows	TestAccept
7:25	azure windows	TestAccept
4:08	azure linux	TestAccept
4:02	gcp linux	TestAccept
3:58	aws linux	TestAccept

… out to tar Per review, replace the `tar -cf -` subprocess in createPlainTarball with a Go-native tar writer. Add libs/tarpack, a small helper that writes a tar stream to a caller-supplied io.Writer, so compression stays the caller's choice: air run wraps it in klauspost/pgzip, keeping the parallel-gzip win while dropping the external tar dependency and its Windows colon-in-path workaround. Symlinks, file modes, and mtimes are preserved, matching what tar stored. Enumeration (git ls-files) and the git-ref path (git archive) are unchanged. The helper lives under libs/ so other packers can adopt it, but only air run is wired to it here. Co-authored-by: Isaac <no-reply@databricks.com>

eng-dev-ecosystem-bot · 2026-09-11T00:46:57Z

Integration test report

Commit: 6ec1ee1

Run: 34530515033

	Env	❌FAIL	🟨KNOWN	🙈SKIP	✅pass	🙈skip	Time
❌	aws linux	481	5	1	1109	1031	194:38
❌	aws windows	436	5	1	1060	1054	203:55
❌	azure linux	391	1	3	1016	1078	161:08
❌	azure windows	347	1	3	966	1101	169:24
❌	gcp linux	383	1	3	1010	1082	170:22
❌	gcp windows	337	1	3	962	1105	184:20

492 interesting tests: 486 FAIL, 5 KNOWN, 1 SKIP

	Test Name	aws linux	aws windows	azure linux	azure windows	gcp linux	gcp windows
🟨	TestAccept	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K
❌	TestAccept/bundle/apps/compute_size	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/apps/compute_size/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/apps/job_permissions	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/apps/job_permissions/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/config-remote-sync/cli_defaults	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/cli_defaults/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/config_edits	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/config_edits/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/dashboard_etag	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/dashboard_etag/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/flushed_cache	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/flushed_cache/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/formatting_preserved	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/formatting_preserved/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/job_fields	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/job_fields/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/job_multiple_tasks	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/job_multiple_tasks/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/job_params_variables	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/job_params_variables/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/job_pipeline_task	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/job_pipeline_task/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/json_status_selectors	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/json_status_selectors/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/multiple_files	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/multiple_files/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/multiple_resources	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/output_json	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/output_json/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/output_no_changes	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/output_no_changes/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/pipeline_fields	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/pipeline_fields/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/policy_injected_cluster_fields	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/policy_injected_cluster_fields/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/resolve_variables	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/resolve_variables/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/select_basic	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/select_basic/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/select_multiple	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/select_multiple/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/skip_deployment	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/skip_deployment/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/skip_permissions	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/skip_permissions/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/target_override	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/target_override/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/task_rename_revert	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/task_rename_revert/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/deploy/empty-bundle	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/deploy/empty-bundle/DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_YAML_SYNC=/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/deploy/empty-bundle/DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_YAML_SYNC=true/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/deploy/files/no-snapshot-sync	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/deploy/files/no-snapshot-sync/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/deploy/mlops-stacks	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/deploy/mlops-stacks/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/deploy/spark-jar-task	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/destroy/jobs-and-pipeline	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/destroy/jobs-and-pipeline/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/destroy/root-path-name-target	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/destroy/root-path-name-target/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/destroy/root-path-not-scoped	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/destroy/root-path-not-scoped/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/destroy/sibling-target	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/destroy/sibling-target/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/declined-deploy	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/declined-deploy/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/depends-on	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/depends-on/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/deployment-metadata-change	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/deployment-metadata-change/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/existing-state	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/existing-state/DATABRICKS_BUNDLE_ENGINE=direct/DMS=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/multiple-resources	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/multiple-resources/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/no-drift	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/no-drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/no-resources	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/no-resources/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/provenance	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/provenance/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/record-failure	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/record-failure/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/stale-plan	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/stale-plan/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/stale-plan/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/successful-recreate	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/successful-recreate/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/generate/pipeline_and_deploy	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/generate/pipeline_and_deploy/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/generate/python_job_and_deploy	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/generate/python_job_and_deploy/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/delete_idempotent	✅p	✅p	✅p	❌F	✅p	✅p
❌	TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/DMS=/INPUT_CONFIG=model.yml.tmpl/READPLAN=1	✅p	✅p	✅p	❌F	✅p	✅p
❌	TestAccept/bundle/invariant/no_drift	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=app.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=app.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=catalog.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=catalog.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=cluster_policy.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=cluster_policy.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=dashboard.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=dashboard.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=experiment.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=experiment.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=genie_space.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=genie_space.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=instance_pool.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=instance_pool.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_apply_policy_default_values_for_each_task.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_apply_policy_default_values_for_each_task.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_apply_policy_default_values_job_cluster.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_apply_policy_default_values_job_cluster.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_apply_policy_default_values_task_cluster.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_apply_policy_default_values_task_cluster.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_cross_resource_ref.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_cross_resource_ref.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_escaped_refs.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_escaped_refs.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_permission_ref.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_permission_ref.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_pydabs_10_tasks.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_pydabs_10_tasks.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_run.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_run.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_table_update_trigger.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_table_update_trigger.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_with_depends_on.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_with_depends_on.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_with_task.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_with_task.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=model.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=model.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=model_serving_endpoint.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=model_serving_endpoint.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=model_with_permissions.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=model_with_permissions.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=pipeline.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=pipeline.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=pipeline_allow_duplicate_names.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=pipeline_allow_duplicate_names.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=pipeline_apply_policy_default_values.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=pipeline_apply_policy_default_values.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=pipeline_config_dots.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=pipeline_config_dots.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=registered_model.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=registered_model.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=schema.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=schema.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=schema_grant_ref.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=schema_grant_ref.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=schema_uppercase_name.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=schema_uppercase_name.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=secret.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=secret_scope.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=secret_scope.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=secret_scope_default_backend_type.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=secret_scope_default_backend_type.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=sql_warehouse.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=sql_warehouse.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=vector_search_endpoint.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=vector_search_endpoint.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=vector_search_index.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=vector_search_index.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=volume.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=volume.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=volume_path_job_ref.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=volume_path_job_ref.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=volume_uppercase_name.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F

Top 50 slowest tests (at least 2 minutes):

duration	env	testname
13:39	gcp linux	TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
12:34	gcp windows	TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
11:30	aws linux	TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
8:15	gcp windows	TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
8:13	gcp windows	TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/DMS=/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=
7:59	aws windows	TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
7:57	aws linux	TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
7:52	gcp windows	TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
7:51	gcp linux	TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
7:43	azure windows	TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
7:22	azure linux	TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
7:14	gcp linux	TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
7:04	aws linux	TestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
7:00	aws linux	TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
6:18	gcp linux	TestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
5:56	azure windows	TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
5:50	azure linux	TestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
5:49	azure linux	TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
5:46	gcp windows	TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
5:43	azure linux	TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
5:24	gcp linux	TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
5:19	aws windows	TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
5:18	gcp linux	TestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
5:17	gcp windows	TestAccept/bundle/resources/apps/lifecycle-started-omitted/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
5:16	gcp linux	TestAccept/bundle/resources/apps/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
5:12	aws linux	TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
4:57	gcp linux	TestAccept/bundle/resources/apps/lifecycle-started-omitted/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
4:52	gcp windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
4:52	gcp windows	TestAccept/bundle/resources/apps/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
4:47	gcp linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
4:44	gcp linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
4:36	aws windows	TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
4:26	azure linux	TestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
4:21	gcp windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
4:20	azure windows	TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
4:13	gcp windows	TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/DMS=/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
4:12	azure linux	TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
4:11	aws linux	TestAccept/bundle/resources/apps/lifecycle-started-omitted/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
4:07	azure linux	TestAccept/bundle/resources/job_runs/failed_run/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
4:05	azure linux	TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
4:05	gcp windows	TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
4:04	gcp windows	TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
3:51	gcp linux	TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
3:49	aws linux	TestAccept/bundle/resources/clusters/deploy/data_security_mode/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
3:45	azure linux	TestAccept/bundle/resources/apps/lifecycle-started-omitted/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
3:45	aws linux	TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
3:43	aws windows	TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/DMS=/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
3:42	aws linux	TestAccept/bundle/resources/clusters/resize-terminated-fallback/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
3:34	gcp linux	TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
3:34	aws windows	TestAccept/bundle/resources/job_runs/failed_run/DATABRICKS_BUNDLE_ENGINE=direct/DMS=

(314 table rows omitted to keep the report under 60000 bytes)

## Summary Makes the `plain_tar` upload content-addressed so an unchanged working tree skips packaging **and** upload. Today `git_archive` names its tarball by `(commit, include_paths)` and, if that object is already uploaded, reuses it and moves no bytes. `plain_tar` (dirty working tree / no ref — the normal iterate-and-resubmit loop) used a **timestamped** name, so it re-packaged and re-uploaded the full tarball on *every* submission, even when nothing changed. This change: - **`snapshotFiles`** records each path's size, mtime, mode/type, and symlink target. Size, mtime, and mode/type come from the existing `Lstat`; only symlinks require an additional `Readlink`. - **`computePlainTarKey`** hashes the sorted `path+size+mtime+mode+symlink_target` set into a versioned working-tree fingerprint. - **`snapshotTarName`** names `plain_tar` `<dir>_<fingerprint>.tar.gz` (was `<dir>_<timestamp>.tar.gz`) and returns the file listing so packaging reuses it — the tree is walked once, not twice. - **`uploadSnapshotViaDABs`** runs the existing `snapshotExists` skip for **both** modes: if the object already exists, it reuses the remote path and uploads nothing. ## Validation - `go test ./experimental/air/cmd` passes. Tests cover remote deduplication and verify that path, size, mtime, mode/type, and symlink-target changes independently alter the cache key. - `./task lint-q` passes with zero issues. - **End-to-end on df1.** Submitting an unchanged universe tree twice: the second run logged `snapshot upload skipped; reusing …<fingerprint>.tar.gz`, returned the identical remote `code_source_path`, and skipped packaging + upload. Both runs launched (1×A10): | code source | first submit (package + upload) | second submit (unchanged → skipped) | |---|---|---| | `research` (~27 MB) | 5.7 s | 4.9 s | | `research` + `js` + `spark` (~97 MB) | 37.6 s | 4.8 s | These measurements preceded the parallel gzip improvement in databricks#6571, so the first-submit numbers are historical; the unchanged-tree skip remains the behavior demonstrated here. ## Notes / trade-offs - Regular-file content detection is size+mtime, not content — the same trade-off DABs file sync makes — so an edit preserving both is not detected. Mode/type and symlink-target changes are detected. - Remote objects accumulate (one per distinct tree state), same as `git_archive`; no eviction is added here. --- This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>

) ## Summary The `air run` plain_tar snapshot path (dirty working tree / no git ref) packages the code_source by shelling out to `tar -czf`. tar's built-in gzip is single-threaded, so for a large tree it dominates packaging latency. This pipes `tar -cf -` (uncompressed) through **klauspost/pgzip** (MIT), a parallel drop-in for gzip that spreads compression across cores and still emits an ordinary gzip stream. - Level is **DefaultCompression** — the *same level as the old `tar -czf`*, not BestSpeed. The archive is re-uploaded on every run, so its size matters. So this is a pure latency win with no upload-size regression. - Compressing outside tar means **no archive path is passed to tar**, which lets us drop the Windows colon-in-path workaround the `-f <path>` form required. ## Results (local packaging: git walk + tar + gzip; warm page cache) | Target | before (`tar -czf`, serial gz6) | after (`tar -cf -` \| pgzip, gz6) | |---|---|---| | large folder (7.4k files) | 2,456 ms | **716 ms** (~3.4×) | | gzip step alone, 470 MiB tar | 7,940 ms | **436 ms** (~18×) | Same compression level, so the tarball is the same size as before (~24 MB for large folder). Level chosen from the curve on a 476 MiB tar: L1→L6 costs +150 ms for ~17% fewer bytes (97→80 MB); L6→L9 doubles time for ~1%, so 6 is the knee. ## Validation - Output verified: `gzip -t` clean, entry count matches the file list, extracts correctly. - Existing `snapshot_package_test.go` unit tests pass; the `internal/build` license test passes for the new `pgzip` dep (`// MIT` in go.mod + NOTICE entry). --------- Co-authored-by: Isaac <no-reply@databricks.com>

## Summary Makes the `plain_tar` upload content-addressed so an unchanged working tree skips packaging **and** upload. Today `git_archive` names its tarball by `(commit, include_paths)` and, if that object is already uploaded, reuses it and moves no bytes. `plain_tar` (dirty working tree / no ref — the normal iterate-and-resubmit loop) used a **timestamped** name, so it re-packaged and re-uploaded the full tarball on *every* submission, even when nothing changed. This change: - **`snapshotFiles`** records each path's size, mtime, mode/type, and symlink target. Size, mtime, and mode/type come from the existing `Lstat`; only symlinks require an additional `Readlink`. - **`computePlainTarKey`** hashes the sorted `path+size+mtime+mode+symlink_target` set into a versioned working-tree fingerprint. - **`snapshotTarName`** names `plain_tar` `<dir>_<fingerprint>.tar.gz` (was `<dir>_<timestamp>.tar.gz`) and returns the file listing so packaging reuses it — the tree is walked once, not twice. - **`uploadSnapshotViaDABs`** runs the existing `snapshotExists` skip for **both** modes: if the object already exists, it reuses the remote path and uploads nothing. ## Validation - `go test ./experimental/air/cmd` passes. Tests cover remote deduplication and verify that path, size, mtime, mode/type, and symlink-target changes independently alter the cache key. - `./task lint-q` passes with zero issues. - **End-to-end on df1.** Submitting an unchanged universe tree twice: the second run logged `snapshot upload skipped; reusing …<fingerprint>.tar.gz`, returned the identical remote `code_source_path`, and skipped packaging + upload. Both runs launched (1×A10): | code source | first submit (package + upload) | second submit (unchanged → skipped) | |---|---|---| | `research` (~27 MB) | 5.7 s | 4.9 s | | `research` + `js` + `spark` (~97 MB) | 37.6 s | 4.8 s | These measurements preceded the parallel gzip improvement in #6571, so the first-submit numbers are historical; the unchanged-tree skip remains the behavior demonstrated here. ## Notes / trade-offs - Regular-file content detection is size+mtime, not content — the same trade-off DABs file sync makes — so an edit preserving both is not detected. Mode/type and symlink-target changes are detected. - Remote objects accumulate (one per distinct tree state), same as `git_archive`; no eviction is added here. --- This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>

…6577) Both DABs-built tgz paths gzip single-threaded today: tarballFromInclude via compress/gzip and tarballFromGit via git's --format=tar.gz. gzip dominates packaging time on a large tree. Pipe both through klauspost/pgzip (MIT), which spreads compression across cores and still emits an ordinary gzip stream, at the same default level so tarball sizes are unchanged. The tgz artifact is content-addressed on upload, so the compressed bytes must be reproducible. pgzip only parallelizes across fixed-size blocks, so pinning the block size keeps output identical regardless of the build host's core count; TestTarballFromGitIsReproducible guards this. Mirrors the pgzip approach in the air plain_tar packer (#6571), requested by pietern. Co-authored-by: Isaac <no-reply@databricks.com> Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>

This was referenced Sep 8, 2026

experimental/air: warm snapshot cache for the plain_tar path #6572

Draft

experimental/air: warm snapshot cache + parallel gzip for plain_tar #6560

Closed

ben-hansen-db requested a review from vinchenzo-db September 8, 2026 17:41

ben-hansen-db marked this pull request as ready for review September 8, 2026 17:42

ben-hansen-db requested a review from pietern September 8, 2026 17:42

vinchenzo-db approved these changes Sep 8, 2026

View reviewed changes

ben-hansen-db and others added 3 commits September 8, 2026 21:48

ben-hansen-db force-pushed the air-plain-tar-pgzip branch from b732496 to 8155ccf Compare September 8, 2026 21:52

vinchenzo-db mentioned this pull request Sep 8, 2026

bundle/artifacts: parallel gzip (pgzip) for the tgz artifact builder #6577

Merged

ben-hansen-db mentioned this pull request Sep 8, 2026

experimental/air: content-address the plain_tar upload #6578

Merged

pietern approved these changes Sep 10, 2026

View reviewed changes

ben-hansen-db enabled auto-merge September 10, 2026 20:37

ben-hansen-db added this pull request to the merge queue Sep 10, 2026

Merged via the queue into main with commit 6ec1ee1 Sep 10, 2026
39 checks passed

ben-hansen-db deleted the air-plain-tar-pgzip branch September 10, 2026 21:07

pietern added the AIR Databricks AI Runtime CLI label Sep 18, 2026

Navigation Menu

experimental/air: parallel gzip for the plain_tar snapshot packer - #6571

experimental/air: parallel gzip for the plain_tar snapshot packer#6571
ben-hansen-db merged 4 commits into
mainfrom
air-plain-tar-pgzip

ben-hansen-db commented Sep 8, 2026 •

edited

Loading

Uh oh!

vinchenzo-db left a comment

Uh oh!

vinchenzo-db Sep 8, 2026

Uh oh!

ben-hansen-db Sep 8, 2026 •

edited

Loading

Uh oh!

vinchenzo-db Sep 8, 2026

Uh oh!

ben-hansen-db Sep 8, 2026 •

edited

Loading

Uh oh!

eng-dev-ecosystem-bot commented Sep 8, 2026 •

edited

Loading

Uh oh!

Uh oh!

eng-dev-ecosystem-bot commented Sep 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

ben-hansen-db commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Results (local packaging: git walk + tar + gzip; warm page cache)

Validation

Uh oh!

vinchenzo-db left a comment

Choose a reason for hiding this comment

Uh oh!

vinchenzo-db Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

ben-hansen-db Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vinchenzo-db Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

ben-hansen-db Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eng-dev-ecosystem-bot commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integration test report

Uh oh!

Uh oh!

eng-dev-ecosystem-bot commented Sep 11, 2026

Integration test report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ben-hansen-db commented Sep 8, 2026 •

edited

Loading

ben-hansen-db Sep 8, 2026 •

edited

Loading

ben-hansen-db Sep 8, 2026 •

edited

Loading

eng-dev-ecosystem-bot commented Sep 8, 2026 •

edited

Loading