基于docker搭建6节点hadoop集群

centos8基于docker搭建6个节点的hadoop集群

yum源配置

配置阿里的centos8 yum源

cd /etc/yum.repos.d
mkdir bak
mv * bak
wget https://mirrors.aliyun.com/repo/Centos-8.repo
yum clean all
yum repolist
yum install -y vim  bash-completion net-tools wget //安装小工具 

安装docker

方法一：

wget -qO- https://get.docker.com/ | sh
yum install https://download.docker.com/linux/fedora/30/x86_64/stable/Packages/containerd.io-1.2.6-3.3.fc30.x86_64.rpm  //出现错误之后执行
wget -qO- https://get.docker.com/ | sh
systemctl start docker
docker -v

方法二：

yum install -y yum-utils  device-mapper-persistent-data  lvm2
yum-config-manager  --add-repo   https://download.docker.com/linux/centos/docker-ce.repo
yum clean all
yum makecache
yum install https://download.docker.com/linux/fedora/30/x86_64/stable/Packages/containerd.io-1.2.6-3.3.fc30.x86_64.rpm
yum install docker-ce docker-ce-cli
systemctl start docker
docker -v

docker拉取centos镜像

创建虚拟网络，方便集群间进行通信

docker network create --driver=bridge hadoop

拉取centos镜像

docker pull centos //默认拉取最稳定的版本

拉取速度慢的话看这里
创建一个centos容器

docker run -itd  --privileged --name Centos8 centos /usr/sbin/init
docker exec -it Centos8 /bin/bash    //进入容器

如果docker内centos连接不上网络，建议：

关闭服务器安全设置，关闭防火墙，关闭Selinux
然后在启动启动docker镜像

setenforce 0//临时修改
//永久改修
vi /etc/selinux/config
SELINUX=Enfocing--->SELINUX=disable

检查命令getenforce   disabled或permissive
//关闭firewalld
systemctl  stop  firewalld
systemctl  disable  firewalld
检查命令：systemctl status firewalld
结果：inactive（dead）

//删除iptables规则
iptables -F;iptables -X;iptables -Z;iptables-save

安装JDK 1.8和Scala

在容器内

yum install -y vim  bash-completion net-tools wget //安装小工具 
cd /etc/yum.repos.d/
mkdir bak
mv * bak/
wget https://mirrors.aliyun.com/repo/Centos-8.repo
yum clean all
yum  makecache
yum install java-1.8.0-openjdk* -y
java -version
yum install scala -y
scala    //测试

配置 SSH

集群间需要使用SSH互相通信

yum install openssh-server openssh-clients -y //安装SSH
cd 
ssh-keygen -t rsa -P "" 	//生成密钥，免密登录
cat .ssh/id_rsa.pub >> .ssh/authorized_keys	//将公钥追加到 authorized_keys 文件中
systemctl start sshd
ssh