Hadoop 3.1.3 详细安装教程
Hadoop运行环境搭建
虚拟机环境准备(三台虚拟机都要执行)
1.准备三台虚拟机,虚拟机配置要求如下:
(1)单台虚拟机:内存4G,硬盘50G,安装必要环境
sudo yum install -y epel-release
sudo yum install -y psmisc nc net-tools rsync vim lrzsz ntp libzstd openssl-static tree iotop git
(2) 修改克隆虚拟机的静态IP;
sudo vim /etc/sysconfig/network-scripts/ifcfg-ens33
DEVICE=ens33
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=static
NAME=“ens33”
IPADDR=192.168.43.153
PREFIX=24
GATEWAY=192.168.43.2
DNS1=192.168.43.2
2 . 修改主机名
(1)修改主机名称
sudo hostnamectl --static set-hostname hadoop001
(2)配置主机名称映射,打开/etc/hosts
sudo vim /etc/hosts
192.168.43.153 hadoop001
192.168.43.154 hadoop002
192.168.43.155 hadoop003
(3) 修改window的主机映射文件(hosts文件)
进入C:\Windows\System32\drivers\etc路径
192.168.43.153 hadoop001
192.168.43.154 hadoop002
192.168.43.155 hadoop003
3.关闭防火墙
sudo systemctl stop firewalld
sudo systemctl disable firewalld
4 . 在/opt目录下创建文件夹
cd /opt
sudo mkdir software
5 .安装JDK
(1)卸载现有JDK
rpm -qa | grep -i java | xargs -n1 sudo rpm -e --nodeps
(2)安装JDK (所有tar包上传至/opt/software目录)
cd /opt/software
sudo tar -zxvf jdk-8u201-linux-x64.tar.gz
sudo vi /etc/profile
文件末尾添加
export JAVA_HOME=/opt/software/jdk1.8.0_201
export PATH=$JAVA_HOME/bin:$PATH
使环境变量生效
sudo source /etc/profile
java -version
6 . 安装hadoop
(1) 下载Hadoop安装包
(2)解压 hadoop-3.1.3.tar.gz
tar -zxvf hadoop-3.1.3.tar.gz
(3) 配置环境变量
sudo vi /etc/profile
文件末尾添加
export HADOOP_HOME=/opt/software/hadoop-3.1.3
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin
使环境变量生效
sudo source /etc/profile
hadoop version
配置集群(三台虚拟机都要执行)
1 . 配置ssh免密登录
(1)生成公钥和私钥
然后敲(三个回车),就会生成两个文件id_rsa(私钥)、id_rsa.pub(公钥)
ssh-keygen -t rsa
(2)将公钥拷贝到要免密登录的目标机器上
ssh-copy-id hadoop001
ssh-