准备工作
为了可以使用start-all.sh脚本启动整个本地集群,需要先配置 ssh 免密。
检查本机是否启动了 ssh 服务,如果出现以下信息则说明服务未启动,需先配置本机的 ssh 服务。
$ ssh localhost
ssh: connect to host localhost port 22: Connection refused
配置 SSH 服务
1. 安装 SSH 服务
sudo apt-get update
sudo apt-get install openssh-server
2. 启动 SSH 服务
sudo systemctl start ssh
3. 启用 SSH 服务开机启动
sudo systemctl enable ssh
配置免密 SSH 登录
首先,需要生成 SSH 密钥对。
ssh-keygen -t rsa
然后,将公钥复制到目标服务器(localhost)。执行以下命令,并按照提示输入密码即可。
$ ssh-copy-id zhangjc@localhost
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
zhangjc@localhost's password:
ERROR: Python with virtualenvwrapper module not found!
Either, install virtualenvwrapper module for the default python3 interpreter
or set VIRTUALENVWRAPPER_PYTHON to the interpreter to use.
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'zhangjc@localhost'"
and check to make sure that only the key(s) you wanted were added.
Spark Standalone启动
首先,从Spark官网(Downloads | Apache Spark)下载安装包。我下载的是“spark-3.5.1-bin-hadoop3.tgz”。
然后,将下载安装包解压至自己喜欢的位置。
执行以下命令时,可能会出现未配置JAVA_HOME的错误,如下:
$ ./sbin/start-all.sh
localhost: JAVA_HOME is not set
这是因为Spark通过SSH登录Worker节点启动服务,但shell登录服务器时不会加载环境变量。解决该问题需要在spark-config.sh中添加以下配置:
export JAVA_HOME=/opt/java
export PATH=$JAVA_HOME/bin:$PATH
配置完成后,再次执行以下命令集群启动成功。
$ ./sbin/start-all.sh
$ jps
99510 Master
99654 Worker