大数据集群环境 hadoop集群环境准备

 

 

   Centos7 系统可以通过链接下载Centos下载地址。首先需要准备的软件都有：Centos系统、SecureCRT 8.5、VMware Workstation Pro、jdk-8u172-linux-x64.tar.gz基本上这个软件就是今天的战场。

首先安装VMware Workstation 并配置激活SecureCRT，关于这两个软件的安装可以查看这个教程SecureCRT安装、VMware Workstation 14 中文破解版安装，VMware14安装centos7。按照要求可以准备三台虚拟机。

一、 hadoop环境准备（可以先安装一台虚拟机，在复制两个，统一配置）

1.1查看当前操作系统环境以及主机映射关系

[root@jiang01 ~]# cat /etc/redhat-release 
CentOS Linux release 7.7.1908 (Core)
[root@jiang01 ~]# uname -r
3.10.0-1062.el7.x86_64
[root@jiang01 ~]# free -m
              total        used        free      shared  buff/cache   available
Mem:           2827         196        2323           9         307        2475
Swap:          3071           0        3071
[root@jiang01 ~]# hostname -i

1.2 修改hostname,重启三台机器，主机名称将永久生效

# 1 使用hostnamectl命令
hostnamectl set-hostname xxx	
//删除hostname
hostnamectl set-hostname ""

hostnamectl set-hostname 主机名
# 2
 vi /etc/hostname 

1.3 关闭防火墙

停止firewall         systemctl stop firewalld.service

禁止firewall开机启动   systemctl disable firewalld.service 

1.4 hosts文件设置，只是为了不直接使用ip，使用主机名方便好记，

[root@jiang01 ~]# vi /etc/hosts
[root@jiang01 ~]# cat /etc/hosts | grep jiang
192.168.1.16 jiang01
192.168.1.17 jiang02
192.168.1.17 jiang03

1.5  配置三台机器免密登录 ，网上有很多种方式来配置，我选择了其中最简单的那种。命令依次为

生成私钥并配置禹本纪的免密登录：ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

拷贝私钥 ssh-copy-id 用户名@主机名

查看免秘是否成功  ssh 主机名 

 

 1.6 配置网络镜像源

下载镜像源

curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo

清楚缓存

yum clean all && yum update && yum makecache

1.7 SELINUX关闭

sed -i 's#SELINUX=enforcing#SELINUX=disabled#' /etc/selinux/config 

1.8 安装jdk

[root@jiang01 ~]# mkdir app
[root@jiang01 ~]# 
[root@jiang01 ~]# cd app/
[root@jiang01 app]# 
[root@jiang01 app]# rz
rz waiting to receive.
Starting zmodem transfer.  Press Ctrl+C to cancel.
Transferring installJDK.sh...
  100%     593 bytes  593 bytes/sec 00:00:01       0 Errors  
Transferring jdk-8u172-linux-x64.tar.gz...
  100%  186447 KB    1456 KB/sec    00:02:08       0 Errors  

[root@jiang01 app]# 
[root@jiang01 app]# 
[root@jiang01 app]# source ./installJDK.sh 
/usr/local/app/jdk1.8.0_172
安装完毕！！！\r下面进行测试
Usage: javac <options> <source files>
where possible options include:
  -g                         Generate all debugging info
  -g:none                    Generate no debugging info
  -g:{lines,vars,source}     Generate only some debugging info
  -nowarn                    Generate no warnings
  -verbose                   Output messages about what the compiler is doing
  -deprecation               Output source locations where deprecated APIs are used
  -classpath <path>          Specify where to find user class files and annotation processors
  -cp <path>                 Specify where to find user class files and annotation processors
  -sourcepath <path>         Specify where to find input source files
  -bootclasspath <path>      Override location of bootstrap class files
  -extdirs <dirs>            Override location of installed extensions
  -endorseddirs <dirs>       Override location of endorsed standards path
  -proc:{none,only}          Control whether annotation processing and/or compilation is done.
  -processor <class1>[,<class2>,<class3>...] Names of the annotation processors to run; bypasses default discovery process
  -processorpath <path>      Specify where to find annotation processors
  -parameters                Generate metadata for reflection on method parameters
  -d <directory>             Specify where to place generated class files
  -s <directory>             Specify where to place generated source files
  -h <directory>             Specify where to place generated native header files
  -implicit:{none,class}     Specify whether or not to generate class files for implicitly referenced files
  -encoding <encoding>       Specify character encoding used by source files
  -source <release>          Provide source compatibility with specified release
  -target <release>          Generate class files for specific VM version
  -profile <profile>         Check that API used is available in the specified profile
  -version                   Version information
  -help                      Print a synopsis of standard options
  -Akey[=value]              Options to pass to annotation processors
  -X                         Print a synopsis of nonstandard options
  -J<flag>                   Pass <flag> directly to the runtime system
  -Werror                    Terminate compilation if warnings occur
  @<filename>                Read options and filenames from file

java版本为：
java version "1.8.0_172"
Java(TM) SE Runtime Environment (build 1.8.0_172-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode)
是否删除安装包
1) 是
2) 否
#? 1
脚本执行完毕！！！
[root@jiang01 jdk1.8.0_172]# 

java安装脚本

#!/bin/bash
#author :lizhenjiang
#email:idea_zhenjiang@163.com
mkdir /usr/local/app
tar -zxf jdk* -C /usr/local/app
cd /usr/local/app/jdk*
home=`pwd`
echo $home
echo "export JAVA_HOME=${home}" >> /etc/profile
echo "export PATH=\$PATH:\$JAVA_HOME/bin" >> /etc/profile
echo "安装完毕！！！\r下面进行测试"
#配置完成，下面的是测试
source /etc/profile
javac
echo "java版本为："
java -version
echo "是否删除安装包"
select var in 是 否
do
    break
done
if [ var = 是 ]
then 
    rm jdk-*
    echo "安装包已删除"
fi
echo "脚本执行完毕！！！"

1.9 编写集群管理脚本，这些脚本在其它课程上有很多，都可以拿来使用

批量执行服务脚本命令

#!/bin/bash
#author :lizhenjiang
#email:idea_zhenjiang@163.com
#判断用户是否传参
if [ $# -lt 1 ];then
        echo "请输入参数"
        exit
fi

#获取用户输入的命令
cmd=$@

for (( i=1;i<=3;i++ ))
do
        tput setaf 6
        echo ============= jiang0${i} : $cmd ============
        tput setaf 9
        #远程执行命令
        ssh jiang0${i} "source /etc/profile;$cmd"
        #判断命令是否执行成功
        if [ $? == 0 ];then
                echo "命令执行成功"
        fi
done

[root@jiang01 jdk1.8.0_172]# vi /usr/bin/xcall.sh    
[root@jiang01 jdk1.8.0_172]# chmod +x /usr/bin/xcall.sh   
[root@jiang01 jdk1.8.0_172]#  xcall.sh ls -d /root/
[root@jiang01 jdk1.8.0_172]#  xcall.sh ls -d /root/       
============= jiang01 : ls -d /root/ ============
/root/
命令执行成功
============= jiang02 : ls -d /root/ ============
/root/
命令执行成功
============= jiang03 : ls -d /root/ ============
/root/
命令执行成功
[root@jiang01 jdk1.8.0_172]# 

 

[root@jiang01 jdk1.8.0_172]# xcall.sh "yum -y install rsync"      
============= jiang01 : yum -y install rsync ============
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.aliyun.com
 * extras: mirrors.aliyun.com
 * updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package rsync.x86_64 0:3.1.2-6.el7_6.1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package        Arch            Version                     Repository     Size
================================================================================
Installing:
 rsync          x86_64          3.1.2-6.el7_6.1             base          404 k

Transaction Summary
================================================================================
Install  1 Package

Total download size: 404 k
Installed size: 815 k
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : rsync-3.1.2-6.el7_6.1.x86_64                                 1/1 
  Verifying  : rsync-3.1.2-6.el7_6.1.x86_64                                 1/1 

Installed:
  rsync.x86_64 0:3.1.2-6.el7_6.1                                                

Complete!
命令执行成功
============= jiang02 : yum -y install rsync ============
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.aliyun.com
 * extras: mirrors.aliyun.com
 * updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package rsync.x86_64 0:3.1.2-6.el7_6.1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package        Arch            Version                     Repository     Size
================================================================================
Installing:
 rsync          x86_64          3.1.2-6.el7_6.1             base          404 k

Transaction Summary
================================================================================
Install  1 Package

Total download size: 404 k
Installed size: 815 k
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : rsync-3.1.2-6.el7_6.1.x86_64                                 1/1 
  Verifying  : rsync-3.1.2-6.el7_6.1.x86_64                                 1/1 

Installed:
  rsync.x86_64 0:3.1.2-6.el7_6.1                                                

Complete!
命令执行成功
============= jiang03 : yum -y install rsync ============
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.aliyun.com
 * extras: mirrors.aliyun.com
 * updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package rsync.x86_64 0:3.1.2-6.el7_6.1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package        Arch            Version                     Repository     Size
================================================================================
Installing:
 rsync          x86_64          3.1.2-6.el7_6.1             base          404 k

Transaction Summary
================================================================================
Install  1 Package

Total download size: 404 k
Installed size: 815 k
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : rsync-3.1.2-6.el7_6.1.x86_64                                 1/1 
  Verifying  : rsync-3.1.2-6.el7_6.1.x86_64                                 1/1 

Installed:
  rsync.x86_64 0:3.1.2-6.el7_6.1                                                

Complete!
命令执行成功
[root@jiang01 jdk1.8.0_172]# 

批量同步文件脚本：

#!/bin/bash
#author :lizhenjiang
#email:idea_zhenjiang@163.com

#判断用户是否传参
if [ $# -lt 1 ];then
    echo "请输入参数";
    exit
fi


#获取文件路径
file=$@

#获取子路径
filename=`basename $file`

#获取父路径
dirpath=`dirname $file`

#获取完整路径
cd $dirpath
fullpath=`pwd -P`

#同步文件到DataNode
for (( i=2;i<=3;i++ ))
do
    tput setaf 6
    echo =========== jiang0${i} : $file ===========
    tput setaf 9
    #远程执行命令
    rsync -lr $filename `whoami`@jiang0${i}:$fullpath
    #判断命令是否执行成功
    if [ $? == 0 ];then
        echo "命令执行成功"
    fi
done

[root@jiang01 app]# vi /usr/bin/xrsync.sh
[root@jiang01 app]# 
[root@jiang01 app]# 
[root@jiang01 app]# chmod +x /usr/bin/xrsync.sh 
[root@jiang01 app]# 
[root@jiang01 app]#  xcall.sh cat /etc/hosts 
============= jiang01 : cat /etc/hosts ============
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.16 jiang01
192.168.1.17 jiang02
192.168.1.18 jiang03
命令执行成功
============= jiang02 : cat /etc/hosts ============
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.16 jiang01
192.168.1.17 jiang02
192.168.1.18 jiang03
命令执行成功
============= jiang03 : cat /etc/hosts ============
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.16 jiang01
192.168.1.17 jiang02
192.168.1.18 jiang03
命令执行成功

 

[root@jiang01 app]# xcall.sh mkdir /usr/local/app/
============= jiang01 : mkdir /usr/local/app/ ============
mkdir: cannot create directory ‘/usr/local/app/’: File exists
============= jiang02 : mkdir /usr/local/app/ ============
命令执行成功
============= jiang03 : mkdir /usr/local/app/ ============
命令执行成功
[root@jiang01 app]# ll
total 4
drwxr-xr-x. 8 10 143 4096 Sep 26 11:02 jdk1.8.0_172
[root@jiang01 app]# xrsync.sh jdk1.8.0_172/
=========== jiang02 : jdk1.8.0_172/ ===========
命令执行成功
=========== jiang03 : jdk1.8.0_172/ ===========
命令执行成功
[root@jiang01 app]# 
[root@jiang01 app]# 
[root@jiang01 app]# xrsync.sh /etc/profile
=========== jiang02 : /etc/profile ===========
命令执行成功
=========== jiang03 : /etc/profile ===========
命令执行成功
[root@jiang01 app]# 

2.0 同步网络时钟，集群要保证时钟的一致性，以免集群出现问题

显示时区：
date -R
#安装ntpdate工具
yum install ntpdate -y
#使用ntpdate校时（后面的是ntp服务器）
ntpdate pool.ntp.org
hwclock -w 
定时任务（高可用集群时间同步很重要）
/5 * /usr/sbin/ntpdate pool.ntp.org >/dev/null 2>&1
每五分钟同步一次时间

    

到现在基础环境已经准备就绪，已经完全准备好了安装大数据相关的集群。