feat: set Spark job descriptions for iterative algorithms - #898
Merged
james-willis merged 2 commits intoSep 2, 2026
Merged
Conversation
Set per-iteration Spark job descriptions (shown in the Jobs/Stages pages of the Spark UI) for Pregel and both DataFrame-based connected components implementations, so the progress of long runs is visible without reading driver logs. Pregel-based algorithms report their own algorithm name, and Pregel exposes setJobDescriptionPrefix to tell concurrent runs apart. The caller's job description is restored after the run, including on failure. Closes graphframes#797
james-willis
marked this pull request as ready for review
September 2, 2026 16:49
Collaborator
|
Could you add Random Walks (base class if possible) and MIS please? |
RandomWalkBase sets per-batch job descriptions, with the prefix overridable per subclass (RandomWalkWithRestart reports its own name), and MaximalIndependentSet labels its iterations. Requested in review.
SemyonSinchenko
approved these changes
Sep 2, 2026
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Iterative algorithms now set a Spark job description before triggering the actions of each iteration, so the Jobs and Stages pages of the Spark UI show entries like
GraphFrames Pregel: iteration 7 / 20instead of indistinguishable generic names:Pregel.run()sets<prefix>: iteration <i> / <maxIter>per iteration (the/ <maxIter>part is omitted whenmaxIter == Int.MaxValue, as used by algorithms that rely on early stopping) and<prefix>: materializing final resultfor the final materialization. The prefix defaults toGraphFrames Pregeland is configurable via a newsetJobDescriptionPrefixbuilder method, so concurrent runs in one application can be told apart.ShortestPaths,LabelPropagation,StructureAwareLabelPropagation,KCore,DetectingCycles) pass their own algorithm name as the prefix.GraphFrames ConnectedComponents [<runId>]: preparing graph / iteration <i> / materializing final result, reusing the run id that already appears in their driver logs so UI entries can be correlated with log lines.RandomWalkBasesets<prefix> [<runID>]: batch <i> of <numBatches>per batch — the prefix comes from a base-class override point, andRandomWalkWithRestartreports its own name — andMaximalIndependentSetlabels each of its iterations (added on review request).The caller's job description (a thread-local
SparkContextproperty) is saved before a run and restored afterwards — including when the run fails — via a smallJobDescription.withRestoredJobDescriptionhelper inmixins.scala. Only the job description is set, never the job group, so applications that rely onSparkContext.cancelJobGroupfor cancellation are unaffected.Since this lives in the Scala core, it covers every API surface (Scala, PySpark classic, both Spark Connect paths) — the descriptions are set on the JVM driver, where the algorithms execute.
Not covered here, possible follow-ups: the GraphX-backed algorithm paths, single-action algorithms (BFS, AggregateMessages, ...), and exposing
setJobDescriptionPrefixthrough the PySpark/Connect Pregel builders.Tested by the new
JobDescriptionSuite, which uses aSparkListenerto assert that per-iteration descriptions are attached to the jobs of Pregel runs (default and custom prefix), a Pregel-based algorithm (ShortestPaths), all three DataFrame connected components paths,MaximalIndependentSet, andRandomWalkWithRestart, and that the caller's description is restored after both successful and failing runs.Why are the changes needed?
Closes #797. On a long iterative run today every Spark job looks identical in the UI, so there is no way to tell iteration 2 from iteration 40 — or a healthy run from a stuck one — without parsing driver logs.