Mac安装Spark，并使用pyspark和Jupyter写代码

在个人Mac电脑上安装并使用Spark：

第一步，网站上下载最新Spark包。

官网地址：https://spark.apache.org/downloads.html

第二步，查看是否运行良好，是否需要安装其他工具，比如JDK。【SSH连接本地Local Shell】

#cd 到安装的Spark文件夹下
cd spark-3.0.0-preview2-bin-hadoop2.7
#执行Spark自带的一个例子，查看运行情况
./bin/run-example org.apache.spark.examples.SparkPi

如果运行后显示的是如下：那就需要安装JDK（Java Development Kit）。

No Java runtime present, requesting install.

JDK官网下载地址： https://www.oracle.com/java/technologies/javase-jdk14-downloads.html

安装好JDK后，重复上面的./bin/run-example命令，正常运行后，会得到一个结果行:

...
20/04/01 19:19:53 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 1.823470s 
Pi is roughly 3.1408157040785203
...

这样便是安装并运行成功了。

如果可以在集群上跑Spark,可以运行下面代码试试：

#Ip、port是集群IP、port
MASTER=spark://IP:PORT ./bin/run-example org.apache.spark.examples.SparkPi

第三步，使用Spark-shell写代码。

如果使用的是scala，运行 ./bin/spark-shell。如果使用的是Python语言，运行 ./bin/pyspark。

./bin/pyspark

进入如下环境：

(base) 192:spark-3.0.0-preview2-bin-hadoop2.7 gao$ ./bin/pyspark 
Python 3.7.3 (default, Mar 27 2019, 16:54:48) 
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/Users/gao/spark-3.0.0-preview2-bin-hadoop2.7/jars/spark-unsafe_2.12-3.0.0-preview2.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
20/04/01 19:55:55 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 3.0.0-preview2
      /_/

Using Python version 3.7.3 (default, Mar 27 2019 16:54:48)
SparkSession available as 'spark'.
>>> 

第四步，提交一个pythonapp.py脚本

#这里的$SPARK_HOME是Spark安装包所在的本地路径
$SPARK_HOME/bin/spark-submit pythonapp.py

第五步，使用Jupyter

#启用Jupyter
PYSPARK_DRIVER_PYTHON=jupyter PYSPARK_DRIVER_PYTHON_OPTS='notebook' ./spark-3.0.0-preview2-bin-hadoop2.7/bin/pyspark

打开Jupyter后，新建一个Python文件，使用pyspark探索数据，示例如下：

import matplotlib.pyplot as plt
#加载数据
user_data = sc.textFile("/Users/gao/data/ml-100k/u.user") 
user_data.first()
'''
用户id|年龄|性别|职业|邮编
1|24|M|technician|85711'''
#按|拆分数据，并统计数据
user_fields = user_data.map(lambda line: line.split("|"))
num_users = user_fields.map(lambda fields: fields[0]).count()
num_genders = user_fields.map(lambda fields:fields[2]).distinct().count()
num_occupations = user_fields.map(lambda fields:fields[3]).distinct().count()
num_zipcodes = user_fields.map(lambda fields:fields[4]).distinct().count()
print ("Users: %d, genders: %d, occupations: %d, ZIP codes: %d" % (num_users, num_genders, num_occupations, num_zipcodes))

#建立分布直方图做统计
ages = user_fields.map(lambda x: int(x[1])).collect()
fig = plt.gcf()
fig.set_size_inches(16, 10)
plt.hist(ages, bins=20, color='lightblue', normed=True)

结果展示：

Done