Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions core/src/main/scala/org/graphframes/GraphFrame.scala
Original file line number Diff line number Diff line change
Expand Up @@ -592,6 +592,23 @@ class GraphFrame private (
* This can return duplicate rows. E.g., a query `"(u)-[]->()"` will return a result for each
* matching edge, even if those edges share the same vertex `u`.
*
* ==Performance==
* Motif finding translates patterns into a series of joins. Enabling Spark's Cost-Based
* Optimizer (CBO) and join reordering can significantly improve performance by letting Spark
* choose more efficient join orderings based on table statistics:
* {{{
* spark.conf.set("spark.sql.cbo.enabled", "true")
* spark.conf.set("spark.sql.cbo.joinReorder.enabled", "true")
* }}}
* The join reorder algorithm is bounded by `spark.sql.cbo.joinReorder.dp.threshold` (default:
* `12`). If the estimated number of joins in your motif exceeds this threshold, increase it
* accordingly:
* {{{
* spark.conf.set("spark.sql.cbo.joinReorder.dp.threshold", "20")
* }}}
* CBO relies on table statistics, so run `ANALYZE TABLE <tableName> COMPUTE STATISTICS` on the
* vertices and edges tables to ensure accurate statistics are available.
*
* @param pattern
* Pattern specifying a motif to search for.
* @return
Expand Down
17 changes: 17 additions & 0 deletions docs/src/04-user-guide/04-motif-finding.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,23 @@ can be expressed by applying filters to the result `DataFrame`.
This can return duplicate rows. E.g., a query `"(u)-[]->()"` will return a result for each
matching edge, even if those edges share the same vertex `u`.

## Performance

Motif finding translates structural patterns into a series of joins. Enabling Spark's Cost-Based Optimizer (CBO) and join reordering allows Spark to pick more efficient join orderings based on table statistics, which can significantly boost motif finding performance:

```
spark.conf.set("spark.sql.cbo.enabled", "true")
spark.conf.set("spark.sql.cbo.joinReorder.enabled", "true")
```

The join reorder algorithm uses dynamic programming and is bounded by `spark.sql.cbo.joinReorder.dp.threshold` (default: `12`). If the estimated number of joins in your motif exceeds this threshold, increase it accordingly:

```
spark.conf.set("spark.sql.cbo.joinReorder.dp.threshold", "20")
```

Note that CBO relies on table statistics. Run `ANALYZE TABLE <tableName> COMPUTE STATISTICS` on the vertices and edges tables, or use `spark.sql("ANALYZE TABLE ...")`, to ensure accurate statistics are available.

## Python API

For API details, refer to the @:pydoc(graphframes.GraphFrame.find).
Expand Down
28 changes: 24 additions & 4 deletions python/graphframes/graphframe.py
Original file line number Diff line number Diff line change
Expand Up @@ -401,12 +401,32 @@ def pregel(self) -> Pregel:

def find(self, pattern: str) -> DataFrame:
"""
Motif finding.
Motif finding: searching the graph for structural patterns.

See Scala documentation for more details.
Motif finding uses a simple Domain-Specific Language (DSL) for expressing structural
queries. For example, ``graph.find("(a)-[e1]->(b); (b)-[e2]->(a)")`` will search for
pairs of vertices ``a``, ``b`` connected by edges in both directions. It returns a
:class:`DataFrame` of all such structures, with columns for each named element (vertex
or edge) in the motif.

**Performance tip:** Motif finding translates patterns into a series of joins. Enabling
Spark's Cost-Based Optimizer (CBO) and join reordering can significantly improve
performance::

spark.conf.set("spark.sql.cbo.enabled", "true")
spark.conf.set("spark.sql.cbo.joinReorder.enabled", "true")

The join reorder algorithm is bounded by ``spark.sql.cbo.joinReorder.dp.threshold``
(default: ``12``). If the estimated number of joins in your motif exceeds this threshold,
increase it accordingly::

spark.conf.set("spark.sql.cbo.joinReorder.dp.threshold", "20")

CBO relies on table statistics, so run ``ANALYZE TABLE <tableName> COMPUTE STATISTICS`` on
the vertices and edges tables (or temp views) to ensure accurate statistics are available.

:param pattern: String describing the motif to search for.
:return: DataFrame with one Row for each instance of the motif found
:param pattern: String describing the motif to search for.
:return: DataFrame with one Row for each instance of the motif found.
"""
return self._impl.find(pattern=pattern)

Expand Down
Loading