Skip to content

fix: use qualified function identifiers for temp functions - #900

Merged
SemyonSinchenko merged 1 commit into
graphframes:mainfrom
ledanieln:fix-qualified-function-identifier
Sep 8, 2026
Merged

SemyonSinchenko merged 1 commit into
graphframes:mainfrom
ledanieln:fix-qualified-function-identifier

Conversation

@ledanieln

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

RandomizedContraction and KCore algorithms register temporary functions _axpb and _kcoreMerge by calling functionRegistry.registerFunction with FunctionIdentifier objects with a function name string. This PR switches registration to functionRegistry.createOrReplaceTempFunction which takes a plain name and applies the correct temporary qualifications (database and catalog) and de-registration to SessionCatalog.unregisterFunction which resolves the unqualified identifier the same way.

Why are the changes needed?

We hit this running connectedComponents using the RandomizedContraction algorithm on Spark 4.2:

[info]   java.lang.AssertionError: assertion failed: Function identifier must be fully qualified (3-part): _axpb
[info]   at scala.Predef$.assert(Predef.scala:279)
[info]   at org.apache.spark.sql.catalyst.analysis.SimpleFunctionRegistryBase.normalizeFuncName(FunctionRegistry.scala:225)
[info]   at org.apache.spark.sql.catalyst.analysis.SimpleFunctionRegistryBase.normalizeFuncName$(FunctionRegistry.scala:223)
[info]   at org.apache.spark.sql.catalyst.analysis.SimpleFunctionRegistry.normalizeFuncName(FunctionRegistry.scala:331)
[info]   at org.apache.spark.sql.catalyst.analysis.SimpleFunctionRegistryBase.registerFunction(FunctionRegistry.scala:236)
[info]   at org.apache.spark.sql.catalyst.analysis.SimpleFunctionRegistryBase.registerFunction$(FunctionRegistry.scala:232)
[info]   at org.apache.spark.sql.catalyst.analysis.SimpleFunctionRegistry.registerFunction(FunctionRegistry.scala:331)
[info]   at org.apache.spark.sql.catalyst.analysis.FunctionRegistryBase.registerFunction(FunctionRegistry.scala:71)
[info]   at org.apache.spark.sql.catalyst.analysis.FunctionRegistryBase.registerFunction$(FunctionRegistry.scala:56)
[info]   at org.apache.spark.sql.catalyst.analysis.SimpleFunctionRegistry.registerFunction(FunctionRegistry.scala:331)
[info]   ...

This can be reproduced by running the test suites using 4.2.0: ./build/sbt -Dspark.version=4.2.0 "testOnly *RandomizedContractionSuite *KCoreSuite". NOTE: I had to remove the legacy log4j.logger.* entries from core/src/test/resources/log4j2.properties in order to test on 4.2.0.

This assert statement was introduced in apache/spark#54765 to prevent unqualified functions shadowing and colliding with builtin or session functions. The change requires all function identifiers to be fully qualified with the three keys: name, database, and catalog: https://github.com/apache/spark/blob/2a7cfea06ba135cf0ddc62902eb0daf5a835c672/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala#L225

CI doesn't catch this yet because the matrix covers 3.5, 4.0, and 4.1. The fix works on all currently supported versions. createOrReplaceTempFunction registers under a plain string on 3.5/4.0/4.1 and under the qualified one on 4.2. unregisterFunction matches directly on the older versions, and on 4.2 re-qualifies via tempFunctionIdentifier when the database is empty.

I verified with testOnly *RandomizedContractionSuite *KCoreSuite against the Spark versions 3.5.8, 4.0.3, 4.1.2 and 4.2.0 and test against the versions only in the CI test matrix.

@ledanieln

Copy link
Copy Markdown
Contributor Author

First time contributing.. Used GraphFrames for a couple years now and it's been great!! Thank you for keeping up with it and happy to make any changes.

@SemyonSinchenko

Copy link
Copy Markdown
Collaborator

Thanks for this. We used registry incorrectly, but only 4.2 explicitly forbid this, so I realized it only after the 4.2 release 😅

@ledanieln
ledanieln force-pushed the fix-qualified-function-identifier branch from 8f0f5b1 to 9676a30 Compare September 5, 2026 13:01
@ledanieln
ledanieln changed the base branch from master to main September 5, 2026 13:02
@ledanieln

ledanieln commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

Realized I targeted master instead of main. I just rebased my changes onto main and re-ran the tests. Everything still passes. There were some differences (#898) between main and master, not sure if you guys are aware of that already.

@SemyonSinchenko SemyonSinchenko left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution!

@SemyonSinchenko
SemyonSinchenko merged commit e8f2642 into graphframes:main Sep 8, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants