ModuleNotFoundError: No module named 'pyspark.tests'

[MordicusEtCubitus]

Dears,

First thanks for this great graphframe module.

I have an issue using Graphframe: 0.8.2-spark3.2-s_2.12 with Spark 3.2.1 on Linux

I've tried to import graphframes examples like this but:

from graphframes.examples import Graphs

Which raise an exception:

~/anaconda3/envs/Book/lib/python3.8/site-packages/graphframes/examples/__init__.py in <module>
      1 
----> 2 from .belief_propagation import BeliefPropagation
      3 from .graphs import Graphs
      4 
      5 __all__ = ['BeliefPropagation', 'Graphs']

~/anaconda3/envs/Book/lib/python3.8/site-packages/graphframes/examples/belief_propagation.py in <module>
     20 from pyspark import SparkConf, SparkContext
     21 from pyspark.sql import SQLContext, functions as sqlfunctions, types
---> 22 from pyspark.tests import QuietTest as SuppressSparkLogs
     23 
     24 from graphframes import GraphFrame

ModuleNotFoundError: No module named 'pyspark.tests'

As explained here #344 it appears that pyspark.test is now named pyspark.testing

Could it be solved ?

Thanks in advance,

Gaël,

[snehilsinha23]

@MordicusEtCubitus I am facing the same issue. Were you able to resolve this?

[MordicusEtCubitus]

I do not remember, but yes, I must have because my notebook is working, here are my notes on installation first a try to translate in english then my original in french:

Graphframes installation

Installation can be done with the command described on the package page with one of the commands spark-shell, pyspark or spark -submit

bash$ $SPARK_HOME/bin/spark-shell --packages graphframes:graphframes:<version>

In our case:

bash$ $SPARK_HOME/bin/pyspark --packages graphframes:graphframes:0.8.2-spark3.2-s_2.12

OR

bash$ pyspark --packages graphframes:graphframes:0.8.2-spark3.2-s_2.12

With conda the command is simpler: conda install graphframes but the Anaconda package is at the time of writing in a somewhat old version: 0.6.

Finally, if you have any difficulty, you can try this tutorial which explains how to [manually install the graphframes package](https://jentekllc8888.medium.com/quick-and-easy-setup-graphframes-with-apache-spark- for-python-98b716c91310), this one is also interesting.

In order to make this lab work with jupyter, the JAR of graphframes has been copied into the JARS folder of spark:

cp graphframes-0.8.2-spark3.2-s_2.12.jar $SPARK_HOME/jars

Installation de graphframe

L'installation peut se faire avec la commande décrite sur la page du package avec une des commandes spark-shell, pyspark ou spark-submit

bash$ $SPARK_HOME/bin/spark-shell --packages graphframes:graphframes:<version>

Dans notre cas:

bash$ $SPARK_HOME/bin/pyspark --packages graphframes:graphframes:0.8.2-spark3.2-s_2.12

OU

bash$ pyspark --packages graphframes:graphframes:0.8.2-spark3.2-s_2.12

Avec conda la commande est plus simple: conda install graphframes mais le package d'Anaconda est à l'heure d'écriture de ces lignes dans une version un peu ancienne: 0.6.

Enfin, si vous rencontrez des difficultés, vous pouvez essayer ce tutoriel qui explique comment installer manuellement le package graphframes, celui-ci est aussi intéressant.

Afin de faire fonctionner ce TP avec jupyter, le JAR de graphframes a été copié dans le dossier JARS de spark:

cp graphframes-0.8.2-spark3.2-s_2.12.jar $SPARK_HOME/jars