Connectes components produce non correct results

[i100van]

Hello good evening,

I am suffering from the same problem, after generating the connected components, a component is generated that attaches many nodes that are unconnected. I have tried the options mentioned in the thread (collapse the spark graph using the round trip, make a previous count, use a repartition) but nothing works, I keep receiving as output a component that agglutinates millions of nodes, and whose number is usually 0 but oscillates.

I have no more ideas to resort to, I don't know if anyone has any idea that could help, it would be of great help.

Thank you very much for your help.

[mattjamison]

Fwiw, I've noticed this behavior when AQE is enabled. If I disable it directly before calling connected components, it works as expected in my case ...

spark.conf.set('spark.sql.adaptive.enabled', 'false')

[pgrandjean]

Same issue. Writing/Reading vertices and edges to disk worked in my case.

Another workaround: replace call to monotonically_increasing_id() by zipWithUniqueId. This means the DataFrame has to be converted to an RDD and back to a DataFrame again. It seems to work as well.