大数据环境搭建指南

下面的例子都是以三台集群为例：
1. 第一步需要三台机器，乌班图或者centos都可以。两者在安装docker上面不同，其他的步骤基本一样。
2. 固定三台机器的ip，把三台机器设置成静态ip，首先上centos8的设置方案：

• centos8

cd /etc/sysconfig/network-scripts
vim ifcfg-ens192

TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens192
UUID=3e4aae63-11ad-4213-ac97-1d5685ba9d6a
DEVICE=ens192
ONBOOT=yes
IPADDR=172.16.2.201
NETMASK=255.255.255.0
GATEWAY=172.16.2.1
PREFIX=24

设置成以上保存好，重启网络nmcli c reloadifconfig查看是否设置成功
3. 配置三台机器的hosts，三台都一样

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.16.2.201 hadoop-master
172.16.2.202 hadoop-slave1
172.16.2.203 hadoop-slave2
172.16.2.201 spark-master
172.16.2.2 tech.ciwei

另外两台设置一样的scp /etc/hosts root@ip:/etc/hosts
4. 配置三台机器的免密登录
在hadoop-master上执行以下操作

ssh-keygen -t rsa -P '' -f /root/.ssh/id_rsa
cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys
在hadoop-slave1上执行以下操作

ssh-keygen -t rsa -P '' -f /root/.ssh/id_rsa
scp /root/.ssh/id_rsa.pub root@hadoop-master:/root/.ssh/id_rsa.pub01

在hadoop-slave2上执行以下操作

ssh-keygen -t rsa -P '' -f /root/.ssh/id_rsa
scp /root/.ssh/id_rsa.pub root@hadoop-master:/root/.ssh/id_rsa.pub02
在hadoop-master上面执行以下操作

cat /root/.ssh/id_rsa.pub01 >> /root/.ssh/authorized_keys
cat /root/.ssh/id_rsa.pub02 >> /root/.ssh/authorized_keys
scp authorized_keys root@hadoop-slave1:~/roort/ssh/
scp authorized_keys root@hadoop-slave2:~/roort/ssh/

更改三台机器的~/.ssh/文件夹权限改为700， authorized_keys权限改为600

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

至此，三台机器的免密登录过程结束
5. 上传需要安装的hadoop的解压包
6. 配置环境变量

vim ~/.bash_profile

# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATH
export HADOOP_HOME=$HOME/bigdata/hadoop
export JAVA_HOME=$HOME/bigdata/jdk
export HIVE_HOME=$HOME/bigdata/hive
export HBASE_HOME=$HOME/bigdata/hbase
export SPARK_HOME=$HOME/bigdata/spark
export FLUME_HOME=$HOME/bigdata/flume
export KAFKA_HOME=$HOME/bigdata/kafka
export SQOOP_HOME=$HOME/bigdata/sqoop
export ZOOKEEPER_HOME=/root/bigdata/zookeeper
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$JAVA_HOME/bin:$HIVE_HOME/bin:$HBASE_HOME/bin:$SPARK_HOME/bin:$SPARK_HOME/sbin:$FLUME_HOME/bin:/$KAFKA/bin:$SQOOP_HOME/bin:$ZOOKEEPER_HOME/bin:/usr/local/src/git/bin
export HIVE_CONF_DIR=$HIVE_HOME/conf
#export PYSPARK_PYTHON=/miniconda2/envs/py365/bin/python
export PYSPARK_PYTHON=/root/anaconda3/envs/reco_sys/bin/python
#export PYSPARK_DRIVER_PYTHON=/miniconda2/envs/py365/bin/python
export PYSPARK_DRIVER_PYTHON=/root/anaconda3/envs/reco_sys/bin/python
export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.7-src.zip:$PYTHONPATH
export JAVA_LIBRARY_PATH=/root/bigdata/hadoop/lib/native

复制到另外两台机器

scp ~/.bash_profile root@hadoop-slave1:~/.bash_profile
scp ~/.bash_profile root@hadoop-slave2:~/.bash_profile

另外两台机器上执行

source ~/.bash_profile