[SQL] JSON_* and VARIANT_* functions for filtering and transformations - #6686
Conversation
mythical-fred
left a comment
There was a problem hiding this comment.
LGTM — clean design, thorough docs, solid test coverage. One question inline about the scope of the standard-function filter.
mythical-fred
left a comment
There was a problem hiding this comment.
APPROVE. This is a substantial and carefully designed addition — the JSON_* extraction family, the two filter/map higher-order pairs (shallow + deep, with the intentional asymmetry that shallow VARIANT_MAP passes the raw Variant key while VARIANT_DEEP_MAP labels leaves by their dot-joined path), and a JSON Merge Patch style VARIANT_MERGE with the well-documented deviation that JSON null does not delete. Path quoting matches BigQuery's JSON_KEYS strict mode, so {"a.b": 1} and {"a": {"b": 1}} never collide.
The docs are unusually good — every rule is spelled out (non-string keys skipped, SQL NULL predicate drops like WHERE, arrays not concatenated on merge, date/time grammars accepted from strings, JSON_EACH_BIGINT never truncates), and the accompanying tests cover the corners I would have asked about (non-map inputs → empty result, top-level array element-wise filter, quoted path with embedded escapes, variant_merge shortcut on empty right).
CI is red on a self-hosted-runner infra failure ("Executing the custom container implementation failed. Please contact your self hosted runner administrator") — unrelated to the diff. A rerun should clear it.
A few small non-blocking notes:
is_integral_numeric_variantforSqlDecimal((sig, exp))uses10i128.checked_pow(exp as u32); forexp > 38the pow overflows and the fallback*sig == 0only accepts an exact zero. Fine in practice (a decimal with scale ≥ 39 is almost certainly fractional), just worth an inline comment.- The shallow
VARIANT_MAPincludes non-string keys (mapper receives them as raw Variant), whereasVARIANT_DEEP_MAPandVARIANT_DEEP_FILTERskip non-string keys and keep those fields untouched. This asymmetry is deliberate (paths must be strings) but the shallow docs don't explicitly call it out — one sentence in theVARIANT_MAPbullet list would preempt user surprise. VARIANT_MAPdocs say "does not recurse into nested objects"; strictly it also doesn't recurse into arrays. Minor wording tweak.is_identifier_keyis ASCII-only (BigQuery-compatible). Worth mentioning in the docs alongside the quoting rule so users with Unicode keys know they'll be quoted.
None of these should hold the PR.
3571dfd to
01ef75e
Compare
|
@gz I have added implementations for the new flat_variant. I plan to merge this. |
mythical-fred
left a comment
There was a problem hiding this comment.
Re-reviewing at eae2406cf413. The prior APPROVE was at 429ba845; the new tip lands the FlatVariant parallel implementation of the whole JSON_/VARIANT_ family plus a differential test suite.
New content is crates/sqllib/src/flat_variant/functions.rs (+461/-6) and additions to crates/sqllib/src/flat_variant.rs (+185/-2):
json_each_typed<T: DecodeFV>walks aTAG_MAPcontainer, applies akeeppredicate on the raw encoded value bytes, and decodes matching entries viaT::decode. Non-map roots return an empty map, matching the enum path.- The
json_each!macro instantiates the six typed variants (bigint,string,boolean,date,time,timestamp) with the samekeeppredicates that the enum side used, andbigintreuses the sameis_integral_numericshape — including the10i128.checked_pow(scale as u32)fallback-to-sig == 0arm for scale ≥ 39. That matches the enum-side arm I flagged earlier, so the two paths stay in lockstep. variant_filter_fv_/variant_map_fv_/variant_deep_filter_fv_/variant_deep_map_fv_are structural parallels of the enum ops, sharingappend_path_componentandpredicate_keepswithcrate::variant(with the noted future move onceVariantis deprecated).variant_merge_FV_FVimplements RFC 7386 merge patch on the flat encoding.
Differential test harness is the interesting part: #![proptest_config(ProptestConfig::with_cases(500))] runs 500 random variant() instances through both the enum and FlatVariant paths for every op — filter, map, deep_filter, deep_map, all six json_each, json_object_keys, json_keys, and variant_merge — asserting the two implementations agree. That is exactly the right validation for a parallel encoding: if the FV path drifts from the enum path on any input the proptest catches it. Explicit edge-case tests cover the fractional/decimal filter arms that random JSON alone would not reach (float/decimal values never come from JSON text, so they are constructed directly).
Doc string on flat_variant.rs correctly extends the mode explainer to mention the new function family and points at flat_variant::functions.
CI is green. Single commit, no AI trailers.
Approving.
Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
Fixes #6685
Fixes #6424
Checklist
The documentation describes in detail all the functions added. The JSON_* functions are targeted for VARIANT values produced by JSON_PARSE (but work correctly for arbitrary variants). The VARIANT_* functions are designed for higher-order transformations on tree-shaped VARIANT values.