1. Download spark-2.3.1-bin-hadoop2.7.tgz from http://spark.apache.org/downloads.html
2. unzip the file spark-2.3.1-bin-hadoop2.7.tgz
3. Set the SPARK_HOME and PATH
export SPARK_HOME=/home/myname/spark-2.3.1-bin-hadoop2.7
export PATH=$PATH:$SPARK_HOME/bin
4. Set python3 as the default python version
export PYSPARK_PYTHON=python3
5. Enable our sets
source /etc/profile
issues:
1) py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
Please try djk8