fix: connectedComponents max_iter default evaluated to 31 due to ^ precedence - #902
Merged
SemyonSinchenko merged 1 commit intoSep 13, 2026
Merged
Conversation
…ecedence In Python `^` is bitwise XOR and `-` binds tighter than `^`, so the declared default `2 ^ 31 - 2` evaluated to `2 ^ 29` = 31 instead of the intended 2147483646. The Python default is always forwarded to the JVM, so the Scala default of Int.MaxValue never applied when calling from Python, and algorithm="graphx" silently returned split components for any graph needing more than 31 Pregel supersteps. Scope is limited to algorithm="graphx": TwoPhase and RandomizedContraction contain no references to maxIter and run to convergence regardless. The documented behaviour already promised the fix -- 05-traversals.md states the maxIter default is Integer.MAX_VALUE (unlimited). Also adds the missing :param max_iter: docstring entry, and two regression tests: a behavioural one on a 50-vertex path graph (diameter 49), which returns 19 components before the fix and 1 after, and a signature guard that catches a reintroduction without starting Spark. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SemyonSinchenko
approved these changes
Sep 13, 2026
Collaborator
|
Thanks for the contribution ! |
Contributor
Author
|
@SemyonSinchenko , Thanks for the review. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
GraphFrame.connectedComponentsdeclared itsmax_iterdefault as2 ^ 31 - 2. In Python^isbitwise XOR and
-binds tighter than^, so the expression is2 ^ 29, which evaluates to 31rather than the intended
2 ** 31 - 2(2147483646):Three changes, all under
python/:The fix —
python/graphframes/graphframe.py:max_iter: int = 2 ^ 31 - 2becomesmax_iter: int = 2**31 - 2.A docstring entry —
connectedComponentshad no:param max_iter:line at all. Added threelines in the style of its neighbours, noting that it applies only to
algorithm="graphx"andthat the other algorithms ignore it.
Two regression tests in
python/tests/test_graphframes.py:test_connected_components_graphx_default_max_iter_is_unlimited— behavioural. A 50-vertexpath graph (diameter 49) run with
algorithm="graphx"and nomax_iterargument, so thedefault itself is what is under test.
test_connected_components_max_iter_default_value— a signature guard that catches a**→^regression without starting Spark.Both are plain functions on the existing module-scoped
sparkfixture, with no mode-specificbranches or skips, so they run unchanged in classic, Connect and thin-client modes.
Red/green. On the unfixed tree the behavioural test fails with exactly the count a 31-superstep
cap predicts:
Vertices 0–31 collapse onto component
0; vertices 32–49 retain 18 stale labels, giving 19.result.count() == 50passes, so all rows are present and only the component assignment is wrong —the failure mode is silent. With the fix, both tests pass in classic and in Connect mode.
No Scala, build, protobuf or CI files are touched:
Why are the changes needed?
The Python default is always forwarded to the JVM (
graphframe.py→classic/graphframe.pyjava_cc.maxIter(max_iter), and over Spark Connect viaConnectedComponents.max_iter), so theScala-side default of
Int.MaxValuenever applies when calling from Python.ConnectedComponents.scala:143passesmaxIteronly torunGraphX, which hands it straight toPregel(...); that loop iswhile (isActiveMessagesNonEmpty && i < maxIterations)with noconvergence check. The result is that
algorithm="graphx"returns silently wrong, splitcomponents for any graph needing more than 31 supersteps — no exception, no warning. For a
long-diameter graph (identity resolution, session/device linkage) that is a correctness incident,
not a performance nuisance.
This is a defect against stated intent rather than a behaviour change:
docs/src/04-user-guide/05-traversals.md:318already documents themaxIterdefault asInteger.MAX_VALUE(unlimited).Scope is limited to
algorithm="graphx"—TwoPhase.scalaandRandomizedContraction.scalacontain no references to
maxIterand run to convergence regardless.The existing suite does exercise
algorithm="graphx"(PREGEL_ARGUMENTSattest_graphframes.py:40), buttest_connected_components/test_connected_components2use 1- and2-vertex graphs, whose diameter is far below 31 — which is why this was never caught.
One question for maintainers: the minimal fix is the corrected literal. An alternative would be
max_iter: int | None = None, forwarding only when set and lettingmaxIter.getOrElse(Int.MaxValue)govern. That is arguably cleaner but widens the public signatureand touches both backends, so I have not assumed it here — happy to follow up separately if you
prefer it.
Local verification
Run on Windows 11 / JDK 17.0.20.1 against the workflow definitions.
lint— blacklint— isortlint— flake8codespell .(2.4.2)