spark学习-hadoop安装与启动

文章连接

1.spark学习-hadoop安装与启动
2.spark学习-spark安装和启动

---

安装前准备

1.首先准备三台服务器.一台master,两台slave.

172.18.101.157 spark-master
172.18.101.162 spark-slave1
172.18.132.162 spark-slave2

2.设置免密登录

1. 生成私钥和公钥

[root@spark-master data]# ssh-keygen -t rsa

一直敲回车,最后生产密钥

Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:xmlOlzYrf0obAn5T5rN+nI39yf+lXTibaETnu72yEN8 root@spark-master
The key's randomart image is:
+---[RSA 2048]----+
|                 |
|                 |
|                 |
|       . . .. .  |
|      . S Bo o   |
|     . * * o+ o. |
|      . * *+ *oEo|
|       . * =Oo=B+|
|         .*=..B*X|
+----[SHA256]-----+
[root@spark-master data]# 

2. 将公钥复制到slave机器上

[root@spark-master data]# scp /root/.ssh/id_rsa.pub root@172.18.101.162:/data/
[root@spark-master data]# scp /root/.ssh/id_rsa.pub root@172.18.132.162:/data/

进入slave机器,如公钥导入授权文件中

[root@spark-slave1 ~]# cd /data/
[root@spark-slave1 data]# cat id_rsa.pub >> /root/.ssh/authorized_keys
#返回master机器,测试是否配置成功
#测试slave1
[root@spark-master ~]# ssh spark-slave1
Last login: Fri Jun 22 20:07:35 2018 from 172.18.101.157
Welcome to Alibaba Cloud Elastic Compute Service !
#测试slave2
[root@spark-master ~]# ssh spark-slave2
Last login: Fri Jun 22 20:07:31 2018 from 172.18.101.157
Welcome to Alibaba Cloud Elastic Compute Service !

3.安装jdk

vim /etc/profile
配置环境变量

JAVA_HOME=/opt/jdk8
CLASSPATH=.:$JAVA_HOME/lib.tools.jar
PAT