Conversation
optimizations will follow up
FunctionsRegistry, parsing, tests.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Adds tests for the parts of the GQL MATCH pipeline that had no direct
coverage, and pins five defects the new coverage uncovered. No engine
behaviour changes.
New suites:
- FunctionRegistrySuite: the whitelist dispatch table itself (every
accepted name at its documented arity, aliases, case-insensitive
dispatch, all three arity guards, the litStr/litInt slots, the
unknown-name error). Only 4 of ~90 names were exercised before.
- ExpressionLoweringSuite: PrefixEnv and every ExpressionLowering rule,
asserted against the hand-written Spark Column each is documented to
produce, plus literal narrowing and unbound-variable errors.
- PropertyGroupGetDataSuite: the getData(filter, requestedProperties)
contract -- column order, silently dropped unknown names, unmasked
property pass-through, id masking, filter-before-masking, and the
undirected-edge orientation union.
Extended: AstBuilderSuite (literals, malformed patterns, quantifier
parsing, the reserved-word boundary), ResolverSuite (the WHERE
classification buckets with no coverage: edge-variable scan filters,
edge+node and edge+edge conjuncts, non-adjacent and literal-only
conjuncts, multi-position variables, label-validation messages),
QueryExecutorSuite (ScanKey equality, the output schema as a contract,
RETURN *, fan-out RETURN over a missing property),
PropertyGraphFrameQuerySuite (QueryOptions defaults and factories, the
previously untested maxEnumeratedPaths guard and its boundary, explain's
bypass of it), SchemaGraphSnapshotSuite (edge indexing, parallel groups).
Defects pinned by KNOWN LIMITATION tests, each commented with the cause
and a suggested fix:
1. nullif is whitelisted but unusable anywhere -- functions.nullif reads
left.dataType in its constructor and every lowered argument is an
UnresolvedAttribute. nvl/ifnull/coalesce are unaffected.
2. A pattern binding one variable twice fails with AMBIGUOUS_REFERENCE:
PrefixEnv.nodePrefix uses the variable as the column prefix, so both
positions rename to the same columns. Resolution already supports
repeated bindings, so this is an executor-side gap.
3. A scan-local predicate on a bare variable (or on .id) requires the
primary key column to be named literally "id", because scan-local
filters are lowered against raw group columns.
4. An empty result and a non-empty result do not share a schema: the
projection builds lit(...) columns Spark types as non-nullable, while
outputSchema declares every field nullable.
5. RETURN * shares a branch with Projection.Default, so it yields the
fixed output schema rather than the bound variables that explain
advertises.
Also applies the one-line scalafmt rewrap in GraphStatistics.scala that
scalafmtCheckAll (a mandatory pre-PR check) requires.
Verified on Spark 3.5.8 / Scala 2.12.21: core/test passes with no
regressions, scalafmtCheckAll and "scalafixAll --check" pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BxbDi8maZPEWKVYx81Rn1h
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Test coverage for the parts of the GQL
MATCHpipeline added in #849 that currently have no directcoverage, plus characterisation tests for five defects the new coverage uncovered.
No engine behaviour changes — the only non-test change is a one-line scalafmt rewrap (see below).
New suites
FunctionRegistrySuitearity/arityBetween/arityAtLeast), thelitStr/litIntliteral slots, and the unknown-name error. Previously only 4 of ~90 names were exercised.ExpressionLoweringSuitePrefixEnv(named vs positional prefixes,prefixFor, the raw environment) and everyExpressionLoweringrule, asserted against the hand-written SparkColumnit is documented to produce, including literal narrowing and the unbound-variable error.PropertyGroupGetDataSuitegetData(filter, requestedProperties)contract added for the engine: column order, silent dropping of unknown names, unmasked property pass-through, id masking on/off, filter-before-masking, and the undirected-edge orientation union.Extended suites
AstBuilderSuite—NULL/ boolean / empty-string / escaped-quote literals, the absence of unary minus, malformed pattern andRETURNshapes, variable-length quantifier parsing, and the reserved-word boundary.ResolverSuite— the WHERE-classification buckets that had no coverage: edge-variable scan filters, edge+node and edge+edge conjuncts, non-adjacent two-variable conjuncts, literal-only conjuncts, a single WHERE split across all four destinations at once, multi-position variables, and the label-validation error messages.QueryExecutorSuite—ScanKeyequality semantics (the contract the scan memo depends on), the fixed output schema as a contract,RETURN *, and fan-outRETURNwhen a property is missing from one candidate group.PropertyGraphFrameQuerySuite—QueryOptionsdefaults and factories, themaxEnumeratedPathsguard and its boundary (previously untested),explain's bypass of it, and the convenience overloads.SchemaGraphSnapshotSuite—outgoing/incomingindexing, parallel edge groups, and the fact that DOT /toStringrender directed and undirected groups identically.Defects found
Each is pinned by a test named
KNOWN LIMITATION: …that asserts today's behaviour and carries acomment naming the cause and a suggested fix, so the suite stays green and the issue is visible
rather than latent. None are fixed here — those calls belong to the author of #849.
nullifis unusable anywhere in a query.functions.nullifreadsleft.dataTypein itsconstructor, and every argument reaching
FunctionRegistryis anUnresolvedAttribute, so itthrows
UnresolvedExceptioninWHERE, inRETURNand in a join predicate alike. Either dropit from the whitelist or lower it to
when(a === b, lit(null)).otherwise(a).nvl/ifnull/coalesceare unaffected.AMBIGUOUS_REFERENCE.PrefixEnv.nodePrefixuses the bound variable as the column prefix, so
(a)-[]->(b)-[]->(c)-[]->(a)renames nodes 0and 3 both to
a_id. Resolution already supports repeated bindings (classifyWhere'smulti-position handling,
attachNodeScanFilters), so this is an executor-side gap. Cyclicpatterns are a core reason to have a pattern language at all, so it seems worth fixing before
the API is public.
WHERE a = …andWHERE a.id = …require the primary key column to be namedliterally
id. Scan-local filters are lowered withPrefixEnv.rawand applied bygetDataagainst the raw group data, where the masked
iddoes not yet exist. The same predicate in ajoin position or in
RETURNworks, which isolates it to the scan-local path.outputSchemadeclares everyfield nullable, but the projection builds the property-group and
pathcolumns fromlit(...),which Spark types non-nullable. Benign for the per-plan union, but it breaks
df.schema == other.schemafor callers of a documented "fixed output schema".RETURN *shares a branch withProjection.Default, so it yields the fixed six-columnschema, while
explainrenders it as* (all matched variables). Pinned so that implementing areal
*projection is a deliberate change.Two smaller notes, not pinned by tests: keywords (
as,is,in,not, …) cannot be used asvariable, property or label names and the grammar offers no quoted-identifier escape; and
QueryOptions.withDefualtsis misspelled (the tests call it as-is, so a rename shows up here).The one non-test change
core/src/main/scala/org/graphframes/propertygraph/internal/GraphStatistics.scala— a one-linescaladoc rewrap that
scalafmtAllapplies.scalafmtCheckAllis a mandatory pre-PR check perAGENTS.mdand currently fails on #849's head without it.Why are the changes needed?
#849 adds roughly 3,400 lines of engine across a six-stage compiler pipeline. Its own test suite is
substantial, but the coverage is uneven:
FunctionRegistry,ExpressionLowering,PrefixEnvandthe new
getDataoverload had no suite of their own. Those untested regions turned out to be wherethe defects are — four of the five findings above sit in code with zero or near-zero direct
coverage.
The
KNOWN LIMITATIONtests matter independently of the coverage numbers.nullifand therepeated-variable failure are the kind of thing a user discovers after release; pinning them now
means each gets a deliberate decision before the API is public, and every one of those tests
converts into a regression guard the moment a fix lands.
How was this patch tested?
Spark 3.5.8 / Scala 2.12.21, Linux, Java 21:
build/sbt core/test— 715 succeeded, 0 failed, 14 canceled. All tests passed. The 14cancellations are the suite's own
requireSparkVersionGE(4, 1, …)gates inTriangleCountSuiteand the label-propagation suites, skipped by design when running against Spark 3.5.
build/sbt "core/testOnly org.graphframes.propertygraph.*"— 395 tests across 13 suites, all pass.build/sbt scalafmtCheckAllandbuild/sbt "scalafixAll --check"— pass.One note for anyone reproducing locally:
org.graphframes.ldbc.TestLDBCCasesneeds thezstdbinary on
PATHto unpack the benchmark data it downloads. Without it, all 8 of its tests fail withRuntimeException: zstd is not available or not foundfromLDBCUtils.checkZSTD— unrelated tothis change, but easy to mistake for a regression.
Not run here: the Spark 4.0.1 / 4.1.0 matrix from the
AGENTS.mdchecklist, which CI covers.🤖 Generated with Claude Code
https://claude.ai/code/session_01BxbDi8maZPEWKVYx81Rn1h