下面的例子都是以三台集群为例:
1. 第一步需要三台机器,乌班图或者centos都可以。两者在安装docker上面不同,其他的步骤基本一样。
2. 固定三台机器的ip,把三台机器设置成静态ip,首先上centos8的设置方案:
- centos8
cd /etc/sysconfig/network-scripts
vim ifcfg-ens192
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens192
UUID=3e4aae63-11ad-4213-ac97-1d5685ba9d6a
DEVICE=ens192
ONBOOT=yes
IPADDR=172.16.2.201
NETMASK=255.255.255.0
GATEWAY=172.16.2.1
PREFIX=24
设置成以上保存好,重启网络nmcli c reloadifconfig查看是否设置成功
3. 配置三台机器的hosts,三台都一样
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
172.16.2.201 hadoop-master
172.16.2.202 hadoop-slave1
172.16.2.203 hadoop-slave2
172.16.2.201 spark-master
172.16.2.2 tech.ciwei
另外两台设置一样的scp /etc/hosts root@ip:/etc/hosts
4. 配置三台机器的免密登录
在hadoop-master上执行以下操作
ssh-keygen -t rsa -P '' -f /root/.ssh/id_rsa
cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys
在hadoop-slave1上执行以下操作
ssh-keygen -t rsa -P '' -f /root/.ssh/id_rsa
scp /root/.ssh/id_rsa.pub root@hadoop-master:/root/.ssh/id_rsa.pub01
在hadoop-slave2上执行以下操作
ssh-keygen -t rsa -P '' -f /root/.ssh/id_rsa
scp /root/.ssh/id_rsa.pub root@hadoop-master:/root/.ssh/id_rsa.pub02
在hadoop-master上面执行以下操作
cat /root/.ssh/id_rsa.pub01 >> /root/.ssh/authorized_keys
cat /root/.ssh/id_rsa.pub02 >> /root/.ssh/authorized_keys
scp authorized_keys root@hadoop-slave1:~/roort/ssh/
scp authorized_keys root@hadoop-slave2:~/roort/ssh/
更改三台机器的~/.ssh/文件夹权限改为700, authorized_keys权限改为600
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
至此,三台机器的免密登录过程结束
5. 上传需要安装的hadoop的解压包
6. 配置环境变量
vim ~/.bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATH
export HADOOP_HOME=$HOME/bigdata/hadoop
export JAVA_HOME=$HOME/bigdata/jdk
export HIVE_HOME=$HOME/bigdata/hive
export HBASE_HOME=$HOME/bigdata/hbase
export SPARK_HOME=$HOME/bigdata/spark
export FLUME_HOME=$HOME/bigdata/flume
export KAFKA_HOME=$HOME/bigdata/kafka
export SQOOP_HOME=$HOME/bigdata/sqoop
export ZOOKEEPER_HOME=/root/bigdata/zookeeper
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$JAVA_HOME/bin:$HIVE_HOME/bin:$HBASE_HOME/bin:$SPARK_HOME/bin:$SPARK_HOME/sbin:$FLUME_HOME/bin:/$KAFKA/bin:$SQOOP_HOME/bin:$ZOOKEEPER_HOME/bin:/usr/local/src/git/bin
export HIVE_CONF_DIR=$HIVE_HOME/conf
#export PYSPARK_PYTHON=/miniconda2/envs/py365/bin/python
export PYSPARK_PYTHON=/root/anaconda3/envs/reco_sys/bin/python
#export PYSPARK_DRIVER_PYTHON=/miniconda2/envs/py365/bin/python
export PYSPARK_DRIVER_PYTHON=/root/anaconda3/envs/reco_sys/bin/python
export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.7-src.zip:$PYTHONPATH
export JAVA_LIBRARY_PATH=/root/bigdata/hadoop/lib/native
复制到另外两台机器
scp ~/.bash_profile root@hadoop-slave1:~/.bash_profile
scp ~/.bash_profile root@hadoop-slave2:~/.bash_profile
另外两台机器上执行
source ~/.bash_profile