Skip to content

docs: sync connected components docs with the code - #903

Merged
SemyonSinchenko merged 1 commit into
graphframes:mainfrom
simplegaurav:docs/sync-cc-docs
Sep 15, 2026
Merged

SemyonSinchenko merged 1 commit into
graphframes:mainfrom
simplegaurav:docs/sync-cc-docs

Conversation

@simplegaurav

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

The configuration page and the connected components argument list had drifted from the
implementation. Every correction below is against code on main.

docs/src/04-user-guide/13-configurations.md

Documented Actual
supported algorithms: graphframes, graphx; default graphframes four are supported; the default is two_phase (ConnectedComponents.scala:56), and graphframes is a deprecated alias that logs a warning (:86-88)
useLabelsAsComponents defaults to true it defaults to false (mixins.scala:154)
broadcastthreshold "only used when the algorithm is graphframes" used by two_phase (ConnectedComponents.scala:146-163)
checkpointinterval "only used when the algorithm is graphframes" used by two_phase and randomized_contraction (:149, :158, :170)
broadcastthreshold = -1 (AQE mode) not documented at all
local checkpoints are "less error prone" they are less reliable — they do not survive the loss of an executor (mixins.scala:174-178)

Also: all six code examples set the deprecated "graphframes" value, so following the page
produced a deprecation warning. They now use "two_phase".

Two things the page never mentioned, both of which matter in practice:

  • checkpointing falls back to the spark.checkpoint.dir configuration when
    SparkContext.setCheckpointDir is not set (TwoPhase.scala:278,
    RandomizedContraction.scala:69). This is the only route available when sparkContext is
    not reachable, for example from a Spark Connect client.
  • two_phase in AQE mode (broadcastthreshold = -1) uses Spark's own DataFrame checkpointing
    rather than its own named sub-directory (TwoPhase.scala:534), unlike the other paths.

docs/src/04-user-guide/05-traversals.md

The connected components argument list documented checkpoint_interval,
broadcast_threshold and use_labels_as_components. Those names match neither API — the
Python parameters are checkpointInterval, broadcastThreshold and useLabelsAsComponents,
and the Scala setters are setCheckpointInterval, setBroadcastThreshold and
setUseLabelsAsComponents. Copying from the page raised TypeError. They now use the
(Python) / (Scala) convention already used further up the same page for all_paths.

maxIter is no longer described as defaulting to Integer.MAX_VALUE, since the Python and
Scala defaults differ by one; it is described as effectively unlimited instead.

One typo outside the connected components section is also fixed: the shortest paths argument
list documented is_direted; the parameter is is_directed. Happy to split that out if you
would rather keep this PR strictly scoped.

Note that checkpoint_interval is deliberately left untouched in the shortest paths, cycles
detection and aggregate neighbors sections — it is the correct Python name for those methods.
Only connectedComponents uses camelCase.

Why are the changes needed?

This is the page a user lands on when tuning connected components for a large job. As written
it gave the wrong default component ID type, hid the algorithm the project itself recommends
as roughly 5x faster, documented three parameter names that cannot be typed, and stated the
reliability of local checkpoints backwards.

Verification

Documentation only; no code, build or CI files are touched.

Check Result
Laika parse + internal link resolution (sbt docs/laikaAST) no new errors — output identical to a clean main baseline
codespell . (2.4.2, as in the spellcheck workflow) exit 0
Markdown table structure all 8 rows, 4 cells each
Both citation URLs byte-match GraphFramesConf.scala:36,38
Python parameter names and Scala setter names each verified against the current signatures
Per-algorithm applicability of all six config keys traced case by case through ConnectedComponents.run()

sbt docs/laikaHTML could not be run to completion, for a reason unrelated to this PR:
docs/generateAtomFeed fails on a clean checkout with
DateTimeParseException: Text '1970-01-01:T00:00:00Z' could not be parsed at index 10
(LaikaCustoms.scala:67 and :87 hardcode that literal, which carries a stray colon).

The configuration page and the connected components argument list had drifted
from the implementation.

13-configurations.md:

- the algorithm table listed only "graphframes" and "graphx" and named
  "graphframes" as the default. The default is "two_phase"
  (ConnectedComponents.scala), and "randomized_contraction" is also supported;
  "graphframes" is a deprecated alias that logs a warning;
- useLabelsAsComponents was documented as defaulting to true; it defaults to
  false (mixins.scala);
- broadcastthreshold and checkpointinterval were documented as applying only
  when the algorithm is "graphframes". broadcastthreshold applies to
  "two_phase", checkpointinterval to "two_phase" and
  "randomized_contraction";
- broadcastthreshold = -1, which hands skew handling to AQE, was not
  documented at all;
- every example set the deprecated "graphframes" value;
- the local checkpoints note claimed they are less error prone. They are less
  reliable: they do not survive the loss of an executor;
- checkpointing did not mention the spark.checkpoint.dir fallback used when
  SparkContext.setCheckpointDir is not reachable, nor that "two_phase" in AQE
  mode uses Spark's own DataFrame checkpointing.

05-traversals.md:

- the connected components argument list documented checkpoint_interval,
  broadcast_threshold and use_labels_as_components, which match neither the
  Python nor the Scala API. Renamed to the real names, using the
  "(Python) / (Scala)" convention already used elsewhere on the page;
- the maxIter default is described without naming a constant, since the Python
  and Scala defaults differ by one;
- one typo in the shortest paths argument list: is_direted -> is_directed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@SemyonSinchenko SemyonSinchenko left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

~LGTM

@SemyonSinchenko

Copy link
Copy Markdown
Collaborator

Thank you for the contribution!

@SemyonSinchenko
SemyonSinchenko merged commit e4bf8ba into graphframes:main Sep 15, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants