chore: drop Spark 3.4 support by manuzhang · Pull Request #5358 · apache/datafusion-comet

manuzhang · 2026-08-14T06:59:56Z

Which issue does this PR close?

Closes #4329.

Rationale for this change

Spark 3.4 was deprecated in Comet 1.0.0 and is scheduled for removal in 1.1.0. Dropping it reduces the supported build/test matrix and removes compatibility paths that are no longer needed now that Spark 3.5 is the oldest supported release.

What changes are included in this PR?

Remove the Spark 3.4 Maven profile, source/test shims, Spark SQL patch, CI jobs, release artifacts, generated compatibility page, and documentation references.
Simplify shared code and tests to assume Spark 3.5 or newer, including removing the Spark 3.4 AQE dynamic partition-pruning fallback.
Promote the Spark 3.5 plan-stability overrides into the base golden directories and keep Spark 4.x fallback chains intact.
Retarget the Iceberg 1.8 compatibility job to Spark 3.5.9 and JDK 17.

How are these changes tested?

make format PROFILES=-Pspark-3.5
./mvnw clean test-compile -DskipTests -Pspark-4.0
./mvnw clean test-compile -DskipTests -Pspark-4.1
./mvnw clean test-compile -DskipTests -Pspark-4.2
Focused Spark 3.5 runtime coverage for AQE DPP and hash joins
CometTPCDSV1_4_PlanStabilitySuite on Spark 3.5 (97 tests)
CometTPCDSV2_7_PlanStabilitySuite on Spark 3.5 (32 tests)
YAML, shell syntax, Python bytecode, active-reference, and git diff --check validation

manuzhang · 2026-08-18T04:51:44Z

@andygrove @coderfender @comphead I'd appreciate your review here.

coderfender · 2026-08-18T04:52:41Z

Sure! Thank you for the pr @manuzhang

comphead

Thanks @manuzhang

this is def on the table, however lets keep this PR opened for now.

There are some pending use cases still using 3.4.3, I would propose doing this in parts.
Currently it would make sense to:

remove 3.4 from CI
Add a warn message saying 3.4 support would be dropped in 1.1.0
update roadmap

manuzhang · 2026-08-19T08:57:11Z

@comphead It's already announced in 1.0.0 release blog.

Apache Spark 3.4 is deprecated and scheduled for removal in Comet 1.1.0.

Where else do we need to add a warn message and update?

comphead · 2026-08-19T17:32:22Z

@comphead It's already announced in 1.0.0 release blog.

Apache Spark 3.4 is deprecated and scheduled for removal in Comet 1.1.0.

Where else do we need to add a warn message and update?

I was thinking to add a warn runtime message that Spark 3.4 support is sunsetting. Not sure if it makes sense though.

But still I would propose doing a soft drop:

disable CI first to save resources

Keep the code for couple of more weeks and drop it closer to the dedicated release date

parthchandra · 2026-08-19T18:32:54Z

What does the community think about retaining the code for 3.4 (but maybe relaxed ci and backporting of fixes)? We have a ton of users (and even more production jobs) still on 3.4.

kazuyukitanimura · 2026-08-19T20:56:12Z


 /**
- * Shim for the `ignoreNulls` flag on `CollectList` / `CollectSet`. Spark 3.4 through 4.1 have no
+ * Shim for the `ignoreNulls` flag on `CollectList` / `CollectSet`. Spark 3.5 through 4.1 have no


Does it make sense to move spark-3.x to spark-3.5?

I don't think it's necessary. Just moving files around doesn't add much value.

I would say this is a great opportunity to clean up majorVerSrc in pom, not only moving files

hsiang-c · 2026-08-20T00:01:37Z

I'd like to have extended Spark 3.4 support for 2 quarters b/c lots of users are still with Spark 3.4.

manuzhang · 2026-08-20T01:56:02Z

I'd like to have extended Spark 3.4 support for 2 quarters b/c lots of users are still with Spark 3.4.

3.4 is still on branch-1.0 and bug fixes can be released on 1.0.x.

Keep the code for couple of more weeks and drop it closer to the dedicated release date

This doesn't make sense on the main branch since codes committed will never be shipped.

comphead · 2026-08-20T02:23:43Z

This doesn't make sense on the main branch since codes committed will never be shipped.

Thanks @manuzhang It's more than that, we focusing on stopping Spark 3.4 support soon, however grace period may take some time. Comet evolving fast and there are still bunch of scenarios still using 3.4. I'm afraid back porting major features to 3.4 would take much more efforts than just supporting it for now.

Do you see any risks in having 3.4 during following weeks?

We can keep this PR opened and get back to it as soon as we can, WDYT?

manuzhang · 2026-08-20T02:29:07Z

@comphead

I'm afraid back porting major features to 3.4 would take much more efforts than just supporting it for now.

Which major features do we need to land on 3.4? Any open issues and pull requests?

We can keep this PR opened and get back to it as soon as we can, WDYT?

No problem, but if we decide to add more features to 3.4, then we shouldn't drop the support now (for 1.1.0). We can't have both.

comphead · 2026-08-20T02:33:13Z

I would expect to see #4837 #4547 #5314 and some other things currently in progress like shuffle writes are slow for nested schemas, etc

andygrove

I went through the mechanical side of this closely: the spark-3.4 Maven profile and shim directories, dev/diffs/3.4.3.diff, the CI matrices in ci.yml, pr_build_linux.yml, and pr_build_macos.yml, dev/ci/compute-changes.py, and the user-facing docs including installation.md, spark-versions.md, and versioning_policy.md. It all holds together. isSpark35Plus is deleted rather than left behind as an always-true guard, CometSpark34AqeDppFallbackRule is gone along with every call site and tag field that referenced it, and the now-unreachable classOf[Base64] -> CometBase64 mapping in QueryPlanSerde is cleaned up since no supported Spark version constructs a plain Base64 node anymore. Only 3.4.3.diff was touched under dev/diffs/, none of the surviving diffs were hand-edited. I didn't find anything removed here that Spark 3.5, 4.0, 4.1, or 4.2 still depends on.

On the thread with @comphead: the three items cited as reasons to hold off, #4837, #4547, and #5314, are all closed or merged now. Does that change things, or is the concern more about the general rate of feature work still landing on main before 3.4 support closes out? That review is still sitting as changes requested, and I'd want to see it explicitly cleared before this merges rather than left open.

The branch is also showing as conflicting with main right now, so it will need a rebase before it can land regardless of how the above gets resolved.

manuzhang · 2026-09-09T14:21:01Z

@andygrove I don't think we can drop Spark 3.4 support now, since those efforts will be otherwise wasted.

Co-authored-by: Codex <codex@openai.com>

andygrove · 2026-09-09T15:45:22Z

@andygrove I don't think we can drop Spark 3.4 support now, since those efforts will be otherwise wasted.

Sorry, that was an autoamted review. I agree that we can't drop the support yet. We should update the docs to remove the claim that support will be dropped in 1.1.0 and instead say that it will be dropped in a future release

andygrove · 2026-09-09T20:45:36Z

Thanks for the rebase; the branch is on current main now and merges cleanly. But your own reply is the thing to settle before any of the mechanical review matters.

I don't think we can drop Spark 3.4 support now, since those efforts will be otherwise wasted.

If that is where you have landed, I would rather this sat as a draft, or closed with an issue tracking the removal, than stayed open as a rebase-and-conflict treadmill. It touches the Maven profiles, the shim directories, dev/diffs/3.4.3.diff, three CI matrices and compute-changes.py, so it will conflict with almost anything that lands, and someone will keep having to rebase it for a decision that has not been made. Which would you prefer? If the answer is "hold", I am happy to convert it to a draft myself.

On the substance, since it is worth recording either way: the mechanical work is sound. I went through it in detail and found nothing removed that 3.5, 4.0, 4.1 or 4.2 still needs. isSpark35Plus is deleted rather than left as an always-true guard, CometSpark34AqeDppFallbackRule is gone with all its call sites and tag fields, the unreachable classOf[Base64] -> CometBase64 mapping is cleaned up, and only 3.4.3.diff was touched under dev/diffs/ with no hand-editing of the surviving diffs. If the project decides to drop 3.4 later, this is the change to pick up rather than redo.

The three items @comphead cited as reasons to hold are all resolved now (#4837 closed, #4547 closed, #5314 merged), and their CHANGES_REQUESTED from 19 August is still the blocking review. @comphead, whichever way the timing question goes, could you clear or restate that? At the moment the PR is blocked on a review whose stated reasons no longer apply, which makes it hard to tell whether the remaining objection is those specific items or the broader "not yet".

My own view on the timing, for the record: I do not think the argument that in-flight 3.4 work would be wasted is a reason to keep 3.4 indefinitely, but it is a reason not to drop it in the middle of that work. So a tracked deferral with a target version seems right, rather than either merging now or closing without a plan.

manuzhang marked this pull request as ready for review August 14, 2026 08:14

kazuyukitanimura reviewed Aug 14, 2026

View reviewed changes

Comment thread .github/workflows/pr_build_linux.yml

manuzhang requested a review from kazuyukitanimura August 18, 2026 04:50

comphead mentioned this pull request Aug 18, 2026

refactor: hook native Parquet writes into Spark's WriteFilesExec seam #5293

Closed

comphead reviewed Aug 18, 2026

View reviewed changes

comphead requested changes Aug 19, 2026

View reviewed changes

kazuyukitanimura reviewed Aug 19, 2026

View reviewed changes

andygrove added enhancement New feature or request build Build environment labels Sep 6, 2026

andygrove reviewed Sep 9, 2026

View reviewed changes

chore: drop Spark 3.4 support

da49929

Co-authored-by: Codex <codex@openai.com>

manuzhang force-pushed the agent/remove-spark-3-4 branch from 5b0098a to da49929 Compare September 9, 2026 15:40

github-actions Bot added area:writer Native Parquet writer area:shuffle Shuffle (JVM and native) area:aggregation Hash aggregates, aggregate expressions area:scan Parquet scan / data reading area:expressions Expression evaluation area:Iceberg labels Sep 9, 2026

github-actions Bot added the area:udf label Sep 9, 2026

Conversation

manuzhang commented Aug 14, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

How are these changes tested?

Uh oh!

Uh oh!

manuzhang commented Aug 18, 2026

Uh oh!

coderfender commented Aug 18, 2026

Uh oh!

comphead left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

manuzhang commented Aug 19, 2026

Uh oh!

comphead commented Aug 19, 2026

Uh oh!

parthchandra commented Aug 19, 2026

Uh oh!

kazuyukitanimura Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

manuzhang Aug 22, 2026

Choose a reason for hiding this comment

Uh oh!

kazuyukitanimura Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

hsiang-c commented Aug 20, 2026

Uh oh!

manuzhang commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

comphead commented Aug 20, 2026

Uh oh!

manuzhang commented Aug 20, 2026

Uh oh!

comphead commented Aug 20, 2026

Uh oh!

andygrove left a comment

Choose a reason for hiding this comment

Uh oh!

manuzhang commented Sep 9, 2026

Uh oh!

andygrove commented Sep 9, 2026

Uh oh!

andygrove commented Sep 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

comphead left a comment •

edited

Loading

manuzhang commented Aug 20, 2026 •

edited

Loading