Skip to content

feat: set Spark job descriptions for iterative algorithms - #898

Merged
james-willis merged 2 commits into
graphframes:masterfrom
james-willis:jw/797-job-descriptions
Sep 2, 2026
Merged

james-willis merged 2 commits into
graphframes:masterfrom
james-willis:jw/797-job-descriptions

Conversation

@james-willis

@james-willis james-willis commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

What changes were proposed in this pull request?

Iterative algorithms now set a Spark job description before triggering the actions of each iteration, so the Jobs and Stages pages of the Spark UI show entries like GraphFrames Pregel: iteration 7 / 20 instead of indistinguishable generic names:

  • Pregel.run() sets <prefix>: iteration <i> / <maxIter> per iteration (the / <maxIter> part is omitted when maxIter == Int.MaxValue, as used by algorithms that rely on early stopping) and <prefix>: materializing final result for the final materialization. The prefix defaults to GraphFrames Pregel and is configurable via a new setJobDescriptionPrefix builder method, so concurrent runs in one application can be told apart.
  • Pregel-based algorithms (ShortestPaths, LabelPropagation, StructureAwareLabelPropagation, KCore, DetectingCycles) pass their own algorithm name as the prefix.
  • All three DataFrame-based connected components paths (two-phase, two-phase AQE, randomized contraction) label their phases as GraphFrames ConnectedComponents [<runId>]: preparing graph / iteration <i> / materializing final result, reusing the run id that already appears in their driver logs so UI entries can be correlated with log lines.
  • RandomWalkBase sets <prefix> [<runID>]: batch <i> of <numBatches> per batch — the prefix comes from a base-class override point, and RandomWalkWithRestart reports its own name — and MaximalIndependentSet labels each of its iterations (added on review request).

The caller's job description (a thread-local SparkContext property) is saved before a run and restored afterwards — including when the run fails — via a small JobDescription.withRestoredJobDescription helper in mixins.scala. Only the job description is set, never the job group, so applications that rely on SparkContext.cancelJobGroup for cancellation are unaffected.

Since this lives in the Scala core, it covers every API surface (Scala, PySpark classic, both Spark Connect paths) — the descriptions are set on the JVM driver, where the algorithms execute.

Not covered here, possible follow-ups: the GraphX-backed algorithm paths, single-action algorithms (BFS, AggregateMessages, ...), and exposing setJobDescriptionPrefix through the PySpark/Connect Pregel builders.

Tested by the new JobDescriptionSuite, which uses a SparkListener to assert that per-iteration descriptions are attached to the jobs of Pregel runs (default and custom prefix), a Pregel-based algorithm (ShortestPaths), all three DataFrame connected components paths, MaximalIndependentSet, and RandomWalkWithRestart, and that the caller's description is restored after both successful and failing runs.

Why are the changes needed?

Closes #797. On a long iterative run today every Spark job looks identical in the UI, so there is no way to tell iteration 2 from iteration 40 — or a healthy run from a stuck one — without parsing driver logs.

Set per-iteration Spark job descriptions (shown in the Jobs/Stages pages
of the Spark UI) for Pregel and both DataFrame-based connected
components implementations, so the progress of long runs is visible
without reading driver logs. Pregel-based algorithms report their own
algorithm name, and Pregel exposes setJobDescriptionPrefix to tell
concurrent runs apart. The caller's job description is restored after
the run, including on failure.

Closes graphframes#797
@james-willis
james-willis marked this pull request as ready for review September 2, 2026 16:49
@SemyonSinchenko

Copy link
Copy Markdown
Collaborator

Could you add Random Walks (base class if possible) and MIS please?

RandomWalkBase sets per-batch job descriptions, with the prefix
overridable per subclass (RandomWalkWithRestart reports its own name),
and MaximalIndependentSet labels its iterations. Requested in review.
@james-willis
james-willis merged commit 41a804d into graphframes:master Sep 2, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants