Skip to content

fix: connectedComponents max_iter default evaluated to 31 due to ^ precedence - #902

Merged
SemyonSinchenko merged 1 commit into
graphframes:mainfrom
simplegaurav:fix/cc-max-iter-default
Sep 13, 2026
Merged

SemyonSinchenko merged 1 commit into
graphframes:mainfrom
simplegaurav:fix/cc-max-iter-default

Conversation

@simplegaurav

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

GraphFrame.connectedComponents declared its max_iter default as 2 ^ 31 - 2. In Python ^ is
bitwise XOR and - binds tighter than ^, so the expression is 2 ^ 29, which evaluates to 31
rather than the intended 2 ** 31 - 2 (2147483646):

$ python -c "print(2 ^ 31 - 2)"
31

Three changes, all under python/:

  1. The fixpython/graphframes/graphframe.py: max_iter: int = 2 ^ 31 - 2 becomes
    max_iter: int = 2**31 - 2.

  2. A docstring entryconnectedComponents had no :param max_iter: line at all. Added three
    lines in the style of its neighbours, noting that it applies only to algorithm="graphx" and
    that the other algorithms ignore it.

  3. Two regression tests in python/tests/test_graphframes.py:

    • test_connected_components_graphx_default_max_iter_is_unlimited — behavioural. A 50-vertex
      path graph (diameter 49) run with algorithm="graphx" and no max_iter argument, so the
      default itself is what is under test.
    • test_connected_components_max_iter_default_value — a signature guard that catches a
      **^ regression without starting Spark.

    Both are plain functions on the existing module-scoped spark fixture, with no mode-specific
    branches or skips, so they run unchanged in classic, Connect and thin-client modes.

Red/green. On the unfixed tree the behavioural test fails with exactly the count a 31-superstep
cap predicts:

>       assert result.select("component").distinct().count() == 1
E       AssertionError: assert 19 == 1

Vertices 0–31 collapse onto component 0; vertices 32–49 retain 18 stale labels, giving 19.
result.count() == 50 passes, so all rows are present and only the component assignment is wrong —
the failure mode is silent. With the fix, both tests pass in classic and in Connect mode.

No Scala, build, protobuf or CI files are touched:

$ git diff --stat
 python/graphframes/graphframe.py |  5 ++++-
 python/tests/test_graphframes.py | 23 +++++++++++++++++++++++
 2 files changed, 27 insertions(+), 1 deletion(-)

Why are the changes needed?

The Python default is always forwarded to the JVM (graphframe.pyclassic/graphframe.py
java_cc.maxIter(max_iter), and over Spark Connect via ConnectedComponents.max_iter), so the
Scala-side default of Int.MaxValue never applies when calling from Python.

ConnectedComponents.scala:143 passes maxIter only to runGraphX, which hands it straight to
Pregel(...); that loop is while (isActiveMessagesNonEmpty && i < maxIterations) with no
convergence check. The result is that algorithm="graphx" returns silently wrong, split
components
for any graph needing more than 31 supersteps — no exception, no warning. For a
long-diameter graph (identity resolution, session/device linkage) that is a correctness incident,
not a performance nuisance.

This is a defect against stated intent rather than a behaviour change:
docs/src/04-user-guide/05-traversals.md:318 already documents the maxIter default as
Integer.MAX_VALUE (unlimited).

Scope is limited to algorithm="graphx"TwoPhase.scala and RandomizedContraction.scala
contain no references to maxIter and run to convergence regardless.

The existing suite does exercise algorithm="graphx" (PREGEL_ARGUMENTS at
test_graphframes.py:40), but test_connected_components / test_connected_components2 use 1- and
2-vertex graphs, whose diameter is far below 31 — which is why this was never caught.

One question for maintainers: the minimal fix is the corrected literal. An alternative would be
max_iter: int | None = None, forwarding only when set and letting
maxIter.getOrElse(Int.MaxValue) govern. That is arguably cleaner but widens the public signature
and touches both backends, so I have not assumed it here — happy to follow up separately if you
prefer it.

Local verification

Run on Windows 11 / JDK 17.0.20.1 against the workflow definitions.

CI job / cell Result
python-ci lint — black PASS (exit 0, 27s, 40 files)
python-ci lint — isort PASS (exit 0, 17s)
python-ci lint — flake8 PASS (exit 0, 19s)
pre-commit black / isort / flake8 PASS (exit 0 each)
spellcheck — codespell . (2.4.2) PASS (exit 0)
python-ci cell 2 (Spark 4.0.3 / Py 3.12 / JDK 17) — new tests, classic PASS (2 passed, 77s)
python-ci cell 2 — new tests, Connect PASS (2 passed, 92s)
python-ci cell 2 — full suite, classic 107 passed, 2 failed, 2 skipped (1:26:10) — see note
python-ci cell 2 — full suite, Connect PASS (104 passed, 7 skipped, 0 failed, 1:14:00)
python-ci cell 3 (Spark 4.1.2 / JDK 17) — new tests, classic PASS (2 passed, 88s) — Python 3.12, CI uses 3.13
python-ci cell 3 — new tests, Connect PASS (2 passed, 91s) — same deviation

…ecedence

In Python `^` is bitwise XOR and `-` binds tighter than `^`, so the declared
default `2 ^ 31 - 2` evaluated to `2 ^ 29` = 31 instead of the intended
2147483646. The Python default is always forwarded to the JVM, so the Scala
default of Int.MaxValue never applied when calling from Python, and
algorithm="graphx" silently returned split components for any graph needing
more than 31 Pregel supersteps.

Scope is limited to algorithm="graphx": TwoPhase and RandomizedContraction
contain no references to maxIter and run to convergence regardless. The
documented behaviour already promised the fix -- 05-traversals.md states the
maxIter default is Integer.MAX_VALUE (unlimited).

Also adds the missing :param max_iter: docstring entry, and two regression
tests: a behavioural one on a 50-vertex path graph (diameter 49), which
returns 19 components before the fix and 1 after, and a signature guard that
catches a reintroduction without starting Spark.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@SemyonSinchenko

Copy link
Copy Markdown
Collaborator

Thanks for the contribution !

@simplegaurav

Copy link
Copy Markdown
Contributor Author

@SemyonSinchenko , Thanks for the review.

@SemyonSinchenko
SemyonSinchenko merged commit 2caf5d8 into graphframes:main Sep 13, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants