[SQL] Optimization to share expensive expressions across sibling map operators - #6831
Conversation
mythical-fred
left a comment
There was a problem hiding this comment.
APPROVE.
New FuseExpensiveMaps pass hoists shared expensive subexpressions across sibling DBSPMapOperators that share an input port. Groups by input, collects minimal expensive subexpressions per closure body (Expensive predicate + closed-over-parameter check), fingerprints via CanonicalForm for fast fingerprint-equality union-find, and rewrites each cluster into one fused map (union of distinct columns) followed by one projection per original member. Single-map "clusters" are still fused when the map itself duplicates an expensive expression (hasDuplicatedExpensiveField) — that path is exactly what testWithinSingleMap exercises.
Correctness sanity that I walked through:
equivalent(left, newLeft, right, newRight)comparesnewLeft.closure(left.param())vsnewRight.closure(right.param()), which normalises viaEquivalenceContextalpha-equivalence. Since maps in a group share their input port and therefore their row type, usingfirst.param().getType().var()as the fresh variable inFuse.fuseand callingmember.function().call(var).reduce()is type-safe.ContainsNow(compiler, /*perExpression*/ true)resetsfoundon eachstartVisit, so reusing one instance across all map operators inFindFusableMaps.postorderis fine — good detail, easy to get wrong.CollectExpensiveExpressions.closed()correctly refuses expressions that reference variables declared outside the expression and not equal to the closure parameter — necessary because expensive expressions get hoisted to the fused map's body. TheResolveReferencesfix topreorder(DBSPLetExpression)(declaring the let variable) is a prerequisite and is included in this PR.- The postorder handshake in
Fuse.postorder(skip if already inremap, else fuse the whole cluster on first sight and register projections for every member) means members visited later in the traversal find themselves already mapped — clean. shouldInlineCompositionguard onDBSPClosureExpressionprevents a later projection-fusion pass from re-inlining a duplicated-expensive-field projection and undoing this work. Nice.
ConstantViews warning pass and the frontend.calciteCompiler.optimizer logger scope are unrelated janitorial additions and look harmless.
Test coverage is thorough: cross-view sharing, UNION ALL with expensive-inside-larger-expensive, within-single-map dedup (with CountLazy == 0 assertion catching that CSE didn't have to step in), and VARIANT-column sharing. CountCalls asserting exactly 1 call is a strong, easy-to-read invariant.
Non-blocking nit: FuseExpensiveMapsTests.testVariantSharing opens with showFinalVerbose(4);. That's a global Logger.INSTANCE.setLoggingLevel(DBSPCompiler.class, 4) side-effect that persists for the rest of the JVM/test-class run — same debug-leftover pattern I flagged on #6824 (showFinal() in AntiJoinDistinctTests.testSegment). Drop the line before merging.
Style micro-nit (not blocking): FindFusableMaps.find is a plain iterative walk with no path compression / union-by-rank. Groups per input port are typically tiny so it doesn't matter — just noting it in case a future pathological case shows up.
Commits are signed-off, no AI-attribution trailers.
|
@mihaibudiu this could be done for filters too right? |
that was done in #6815 |
mythical-fred
left a comment
There was a problem hiding this comment.
Rebased tip ab855bbc95 over 42639f04. Delta vs prior approved tip is a straight rebase — same 8 files, same semantics (FuseExpensiveMaps + ConstantViews passes, ResolveReferences let-var declaration, shouldInlineComposition guard, ContainsNow(perExpression=true)). Prior nits still stand (showFinalVerbose(4) debug leftover in testVariantSharing; no path-compression in the tiny union-find — both non-blocking). Re-approving on the current base.
Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
…operators Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
Fixes #6817
Checklist
There are 3 commits, two of which do some janitorial stuff.
The third one implements an optimization that works when you have multiple views that may invoke the same expensive function in a map operator. Something like:
This will be compiled into something equivalent to:
For some customer programs this can produce significant savings in computation cost, even thought the optimized operators are all linear.