Skip to content

[SQL] JSON_* and VARIANT_* functions for filtering and transformations - #6686

Merged
mihaibudiu merged 1 commit into
feldera:mainfrom
mihaibudiu:issue6685
Jul 30, 2026
Merged

[SQL] JSON_* and VARIANT_* functions for filtering and transformations#6686
mihaibudiu merged 1 commit into
feldera:mainfrom
mihaibudiu:issue6685

Conversation

@mihaibudiu

Copy link
Copy Markdown
Contributor

Fixes #6685
Fixes #6424

Checklist

  • Unit tests added/updated
  • Documentation updated

The documentation describes in detail all the functions added. The JSON_* functions are targeted for VARIANT values produced by JSON_PARSE (but work correctly for arbitrary variants). The VARIANT_* functions are designed for higher-order transformations on tree-shaped VARIANT values.

@mihaibudiu
mihaibudiu requested review from anandbraman and gz July 21, 2026 06:45

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — clean design, thorough docs, solid test coverage. One question inline about the scope of the standard-function filter.

Comment thread crates/sqllib/src/variant.rs
Comment thread crates/sqllib/src/variant.rs
Comment thread crates/sqllib/src/variant.rs
Comment thread crates/sqllib/src/variant.rs

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE. This is a substantial and carefully designed addition — the JSON_* extraction family, the two filter/map higher-order pairs (shallow + deep, with the intentional asymmetry that shallow VARIANT_MAP passes the raw Variant key while VARIANT_DEEP_MAP labels leaves by their dot-joined path), and a JSON Merge Patch style VARIANT_MERGE with the well-documented deviation that JSON null does not delete. Path quoting matches BigQuery's JSON_KEYS strict mode, so {"a.b": 1} and {"a": {"b": 1}} never collide.

The docs are unusually good — every rule is spelled out (non-string keys skipped, SQL NULL predicate drops like WHERE, arrays not concatenated on merge, date/time grammars accepted from strings, JSON_EACH_BIGINT never truncates), and the accompanying tests cover the corners I would have asked about (non-map inputs → empty result, top-level array element-wise filter, quoted path with embedded escapes, variant_merge shortcut on empty right).

CI is red on a self-hosted-runner infra failure ("Executing the custom container implementation failed. Please contact your self hosted runner administrator") — unrelated to the diff. A rerun should clear it.

A few small non-blocking notes:

  • is_integral_numeric_variant for SqlDecimal((sig, exp)) uses 10i128.checked_pow(exp as u32); for exp > 38 the pow overflows and the fallback *sig == 0 only accepts an exact zero. Fine in practice (a decimal with scale ≥ 39 is almost certainly fractional), just worth an inline comment.
  • The shallow VARIANT_MAP includes non-string keys (mapper receives them as raw Variant), whereas VARIANT_DEEP_MAP and VARIANT_DEEP_FILTER skip non-string keys and keep those fields untouched. This asymmetry is deliberate (paths must be strings) but the shallow docs don't explicitly call it out — one sentence in the VARIANT_MAP bullet list would preempt user surprise.
  • VARIANT_MAP docs say "does not recurse into nested objects"; strictly it also doesn't recurse into arrays. Minor wording tweak.
  • is_identifier_key is ASCII-only (BigQuery-compatible). Worth mentioning in the docs alongside the quoting rule so users with Unicode keys know they'll be quoted.

None of these should hold the PR.

@mihaibudiu
mihaibudiu force-pushed the issue6685 branch 2 times, most recently from 3571dfd to 01ef75e Compare July 30, 2026 01:18
@mihaibudiu

Copy link
Copy Markdown
Contributor Author

@gz I have added implementations for the new flat_variant. I plan to merge this.

@mihaibudiu
mihaibudiu enabled auto-merge July 30, 2026 01:21
@mihaibudiu
mihaibudiu added this pull request to the merge queue Jul 30, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 30, 2026

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewing at eae2406cf413. The prior APPROVE was at 429ba845; the new tip lands the FlatVariant parallel implementation of the whole JSON_/VARIANT_ family plus a differential test suite.

New content is crates/sqllib/src/flat_variant/functions.rs (+461/-6) and additions to crates/sqllib/src/flat_variant.rs (+185/-2):

  • json_each_typed<T: DecodeFV> walks a TAG_MAP container, applies a keep predicate on the raw encoded value bytes, and decodes matching entries via T::decode. Non-map roots return an empty map, matching the enum path.
  • The json_each! macro instantiates the six typed variants (bigint, string, boolean, date, time, timestamp) with the same keep predicates that the enum side used, and bigint reuses the same is_integral_numeric shape — including the 10i128.checked_pow(scale as u32) fallback-to-sig == 0 arm for scale ≥ 39. That matches the enum-side arm I flagged earlier, so the two paths stay in lockstep.
  • variant_filter_fv_ / variant_map_fv_ / variant_deep_filter_fv_ / variant_deep_map_fv_ are structural parallels of the enum ops, sharing append_path_component and predicate_keeps with crate::variant (with the noted future move once Variant is deprecated).
  • variant_merge_FV_FV implements RFC 7386 merge patch on the flat encoding.

Differential test harness is the interesting part: #![proptest_config(ProptestConfig::with_cases(500))] runs 500 random variant() instances through both the enum and FlatVariant paths for every op — filter, map, deep_filter, deep_map, all six json_each, json_object_keys, json_keys, and variant_merge — asserting the two implementations agree. That is exactly the right validation for a parallel encoding: if the FV path drifts from the enum path on any input the proptest catches it. Explicit edge-case tests cover the fractional/decimal filter arms that random JSON alone would not reach (float/decimal values never come from JSON text, so they are constructed directly).

Doc string on flat_variant.rs correctly extends the mode explainer to mention the new function family and points at flat_variant::functions.

CI is green. Single commit, no AI trailers.

Approving.

Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
@mihaibudiu
mihaibudiu enabled auto-merge July 30, 2026 05:42
@mihaibudiu
mihaibudiu added this pull request to the merge queue Jul 30, 2026
Merged via the queue into feldera:main with commit b384c85 Jul 30, 2026
1 check passed
@mihaibudiu
mihaibudiu deleted the issue6685 branch July 30, 2026 06:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants