Skip to content

[SQL] Optimization to pull expensive operations out of filters - #6818

Merged
mihaibudiu merged 1 commit into
feldera:mainfrom
mihaibudiu:issue6815
Aug 7, 2026
Merged

[SQL] Optimization to pull expensive operations out of filters#6818
mihaibudiu merged 1 commit into
feldera:mainfrom
mihaibudiu:issue6815

Conversation

@mihaibudiu

Copy link
Copy Markdown
Contributor

Fixes #6815

This is most useful for temporal filters. When compiling something like WHERE expensive(x) <= NOW() ... this makes sure that expensive(x) is evaluated only once. As a bonus, if the x field is not needed after the where, the optimizer can actually drop it.

Checklist

  • Unit tests added/updated

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice optimization. Hoisting expensive predicate operands into a preceding map lets ImplementNow see a simple field-vs-window comparison, and as a bonus lets projection pushdown drop the source columns whose only role was feeding the expensive call. The VARIANT test asserting no VARIANT-typed operator output past the hoisting map is a strong invariant to lock in.

A few things I liked:

  • Dedup via EquivalenceContext.equiv on the hoisted list — BETWEEN and other patterns that duplicate operands (like the two TRANSFORM(...) copies from a range check) collapse to a single hoisted column. The nested-closure test pins that.
  • Correctly skipping any subexpression that contains now() — otherwise ImplementNow would lose the temporal shape.
  • Marking VARIANT casts and VARIANT_INDEX as expensive in Expensive.java — these really are runtime function calls under the hood, so the existing containsExternalFunction heuristic was missing them.
  • Simplify pass before analysis normalizes the predicate shape so collect/rewrite see a predictable tree.
  • Result-restoring projection is annotated IsProjection, so downstream projection pushdown can peel it off when the extra columns are unused.

Approving.

@ryzhyk ryzhyk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works well on a real world program with hundreds of filters.

@ryzhyk
ryzhyk added this pull request to the merge queue Aug 6, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Aug 6, 2026
@mihaibudiu
mihaibudiu added this pull request to the merge queue Aug 6, 2026
@gz
gz removed this pull request from the merge queue due to a manual request Aug 6, 2026
@mihaibudiu
mihaibudiu added this pull request to the merge queue Aug 6, 2026
@gz
gz removed this pull request from the merge queue due to a manual request Aug 6, 2026
@mihaibudiu
mihaibudiu added this pull request to the merge queue Aug 7, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Aug 7, 2026
@gz
gz added this pull request to the merge queue Aug 7, 2026
@gz
gz removed this pull request from the merge queue due to a manual request Aug 7, 2026
@gz
gz added this pull request to the merge queue Aug 7, 2026
@gz
gz removed this pull request from the merge queue due to a manual request Aug 7, 2026
@gz
gz added this pull request to the merge queue Aug 7, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Aug 7, 2026
@gz
gz added this pull request to the merge queue Aug 7, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Aug 7, 2026
@mihaibudiu
mihaibudiu enabled auto-merge August 7, 2026 05:58
@mihaibudiu
mihaibudiu added this pull request to the merge queue Aug 7, 2026
@mihaibudiu
mihaibudiu removed this pull request from the merge queue due to a manual request Aug 7, 2026
Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
@mihaibudiu
mihaibudiu added this pull request to the merge queue Aug 7, 2026
Merged via the queue into feldera:main with commit d6b5b0c Aug 7, 2026
1 check passed
@mihaibudiu
mihaibudiu deleted the issue6815 branch August 7, 2026 09:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[SQL] Optimize expensive filters by pulling expensive operations into a prior map

4 participants