在Mac上安装Spark apache-spark-3.5.1

安装spark的步骤较为简单，前提是要安装好Hadoop哦。

这里我使用brew来安装，如果你安装了homebrew直接在终端里输入brew install apache-spark然后等着就可以了（如果没有安装过brew的参考其他博主的内容去安装哦）

brew install apache-spark


==> Auto-updating Homebrew...
Adjust how often this is run with HOMEBREW_AUTO_UPDATE_SECS or disable with
HOMEBREW_NO_AUTO_UPDATE. Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
==> Auto-updated Homebrew!
Updated 2 taps (homebrew/core and homebrew/cask).
==> New Formulae
ffmpeg@6
==> New Casks
boltai

You have 5 outdated formulae installed.

==> Fetching dependencies for apache-spark: libx11 and openjdk@17
==> Fetching libx11
==> Downloading https://mirrors.ustc.edu.cn/homebrew-bottles/libx11-1.8.9.arm64_
######################################################################### 100.0%
==> Fetching openjdk@17
==> Downloading https://mirrors.ustc.edu.cn/homebrew-bottles/openjdk%4017-17.0.1
######################################################################### 100.0%
==> Fetching apache-spark
==> Downloading https://mirrors.ustc.edu.cn/homebrew-bottles/apache-spark-3.5.1.
######################################################################### 100.0%
==> Installing dependencies for apache-spark: libx11 and openjdk@17
==> Installing apache-spark dependency: libx11
==> Pouring libx11-1.8.9.arm64_sonoma.bottle.tar.gz
🍺  /opt/homebrew/Cellar/libx11/1.8.9: 1,042 files, 7.0MB
==> Installing apache-spark dependency: openjdk@17
==> Pouring openjdk@17-17.0.10.arm64_sonoma.bottle.tar.gz
🍺  /opt/homebrew/Cellar/openjdk@17/17.0.10: 635 files, 304.9MB
==> Installing apache-spark
==> Pouring apache-spark-3.5.1.all.bottle.tar.gz
🍺  /opt/homebrew/Cellar/apache-spark/3.5.1: 1,823 files, 423.3MB
==> Running `brew cleanup apache-spark`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).

 上面这个就是用brew安装好的样子

cd /opt/homebrew/Cellar/apache-spark/3.5.1 

切换到spark3.5.1的目录下，然后再切换到 libexec/conf 下面先复制一份 spark-env.sh.template 文件然后重命名（用cp命令）为spark-env.sh

cp spark-env.sh.template spark-env.sh
         
ls

fairscheduler.xml.template	spark-defaults.conf.template	workers.template
log4j2.properties.template	spark-env.sh
metrics.properties.template	spark-env.sh.template

编辑spark-env.sh文件(用：vim spark-env.sh)，在第一行添加以下配置信息:

export SPARK_DIST_CLASSPATH=$(/opt/homebrew/Cellar/hadoop/3.4.0/bin/hadoop classpath)

自己看看自己的Hadoop的位置在哪里哦，就是目录一直要到bin下的Hadoop

我因为也是使用brew下载的Hadoop所以是在Homebrew目录下面，至于怎么用brew怎么装Hadoop在其他博主有讲解

配置完成后就可以直接使用了，就不需要像Hadoop运行启动命令。
通过运行Spark自带的示例，验证Spark是否安装成功。

/opt/homebrew/Cellar/apache-spark/3.5.1/bin

切换到bin目录下执行

run-example SparkPi

如果成功会输出很多这样的运行信息

24/04/11 21:32:32 INFO SparkContext: Running Spark version 3.5.1
24/04/11 21:32:32 INFO SparkContext: OS info Mac OS X, 14.4.1, aarch64
24/04/11 21:32:32 INFO SparkContext: Java version 1.8.0_401
24/04/11 21:32:32 INFO ResourceUtils: ==============================================================
24/04/11 21:32:32 INFO ResourceUtils: No custom resources configured for spark.driver.
24/04/11 21:32:32 INFO ResourceUtils: ==============================================================
24/04/11 21:32:32 INFO SparkContext: Submitted application: Spark Pi
24/04/11 21:32:32 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
24/04/11 21:32:32 INFO ResourceProfile: Limiting resource is cpu
24/04/11 21:32:32 INFO ResourceProfileManager: Added ResourceProfile id: 0

但是那个输出结果不容易找到，这里我使用管道和grep命令进行过滤

命令中的 2>&1 可以将所有的信息都输出到 标志输出（stdout） 中，否则的话由于输出日志的性质（会有WARN之类的stderr），还是会输出到屏幕中

run-example SparkPi 2>&1 | grep "Pi is"


Pi is roughly 3.1475157375786877

如果是输出这样就可以啦

这里我们再启动下Spark Shell试试

我们在还是和上面一样在bin目录下执行spark-shell

启动spark-shell后，会自动创建名为sc的SparkContext对象和名为spark的SparkSession对象

就像这样

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.5.1
      /_/
         
Using Scala version 2.12.18 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_401)
Type in expressions to have them evaluated.
Type :help for more information.

scala>

到这里在Mac上安装Spark就安装完成了