Hi,

(I'm working with Amy on this so I'll share what we've been doing)

Weirdly the example is working fine now after restarting pyspark but I have another weird example (which also works fine on a fresh pyspark).

So I load the graph like this:

CSV files:
https://github.com/neo4j-graph-analytics/book/blob/master/data/transport-nodes.csv
https://github.com/neo4j-graph-analytics/book/blob/master/data/transport-relationships.csv

v = spark.read.csv("data/transport-nodes.csv", header=True)
e = spark.read.csv("data/transport-relationships.csv", header=True)
g = GraphFrame(v, e)

>>> g.vertices.show()
+----------------+---------+---------+
|              id| latitude|longitude|
+----------------+---------+---------+
|       Amsterdam|52.379189| 4.899431|
|         Utrecht|52.092876| 5.104480|
|        Den Haag|52.078663| 4.288788|
|       Immingham| 53.61239| -0.22219|
|       Doncaster| 53.52285| -1.13116|
|Hoek van Holland|  51.9775|  4.13333|
|      Felixstowe| 51.96375|   1.3511|
|         Ipswich| 52.05917|  1.15545|
|      Colchester| 51.88921|  0.90421|
|          London|51.509865|-0.118092|
|       Rotterdam|  51.9225|  4.47917|
|           Gouda| 52.01667|  4.70833|
+----------------+---------+---------+

And then I ask to find the shortest path to Gouda:

>>> g.shortestPaths(["Gouda"])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/spark-c603124e-3913-43a2-a539-0981e2474be3/userFiles-2b39beea-849b-47af-9cd0-3c6f3992ee73/graphframes_graphframes-0.5.0-spark2.1-s_2.11.jar/graphframes/graphframe.py", line 332, in shortestPaths
  File "/home/markhneedham/projects/oreilly-graph-algorithms-book-examples/spark-2.3.1-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
  File "/home/markhneedham/projects/oreilly-graph-algorithms-book-examples/spark-2.3.1-bin-hadoop2.7/python/pyspark/sql/utils.py", line 63, in deco
    return f(*a, **kw)
  File "/home/markhneedham/projects/oreilly-graph-algorithms-book-examples/spark-2.3.1-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o1065.run.
: org.graphframes.NoSuchVertexException: GraphFrame algorithm given vertex ID which does not exist in Graph. Vertex ID Gouda not contained in GraphFrame(v:[id: string, latitude: string ... 1 more field], e:[src: string, dst: string ... 2 more fields])
	at org.graphframes.lib.GraphXConversions$.integralId(GraphXConversions.scala:184)
	at org.graphframes.lib.ShortestPaths$$anonfun$1.apply(ShortestPaths.scala:69)
	at org.graphframes.lib.ShortestPaths$$anonfun$1.apply(ShortestPaths.scala:69)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.Iterator$class.foreach(Iterator.scala:893)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
	at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
	at scala.collection.AbstractTraversable.map(Traversable.scala:104)
	at org.graphframes.lib.ShortestPaths$.org$graphframes$lib$ShortestPaths$$run(ShortestPaths.scala:69)
	at org.graphframes.lib.ShortestPaths.run(ShortestPaths.scala:61)
	at sun.reflect.GeneratedMethodAccessor427.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.lang.Thread.run(Thread.java:748)

Could there be some caching going on that I'm not spotting?

feat: add tests for PageRank with disconnected components #285

Description

Activity

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions