Hadoop+Hive安装

大数据软件安装教程

1.虚拟机安装

1.1安装linux (centos7.7)
安装完毕后进入系统页面
1.2配置ip

vi /etc/sysconfig/network-scripts/ifcfg-ens33
...
BOOTPROTO="static"         # 使用静态IP地址，默认为dhcp
IPADDR="192.168.10.101"   # 设置的静态IP地址
NETMASK="255.255.255.0"    # 子网掩码
GATEWAY="192.168.10.2"    # 网关地址
DNS1=8.8.8.8       		  # DNS服务器
DNS2=8.8.4.4
ONBOOT="yes"             #是否开机启用
...

1.3重启网络服务并修改主机名

#重启网络服务
systemctl restart network

#修改主机名
hostnamectl set-hostname xxx
或者
vi /etc/hostname

1.4查看本机ip

ip addr
#查看一下ip地址是否与自己设置的ip一致

#ping 测试是否
#ping外网 ping主机ip
ping www.baidu.com
ping xxx  #主机ip

1.5下载必要的插件

#ifconfig命令
yum -y install ifconfig
yum search ifconfig
yum -y install net-tools.x86_64

#vim
yum -y install vim

#ntp客户端和服务器
yum -y install ntpdate
yum -y install ntp
(ntpdate同步时间：ntpdate -u time.windows.com)

systemctl start ntpd
systemctl enable ntpd

1.6关闭虚拟机防火墙

关闭:systemctl disable firewalld
开启:systemctl enable firewalld
查看状态:systemctl status firewalld

#关闭NetworkManager
常用管理命令：
systemctl status NetworkManager
systemctl start NetworkManager
systemctl stop NetworkManager
systemctl disable NetworkManager
systemctl enable NetworkManager

2.安装Hadoop、JDK

2.1JDK安装

#1.上传jdk安装包到linux上，并解压到指定目录下
tar -zxvf jdk-***.tar.gz -C /opt/apps/
#2.配置环境变量
vi /etc/profile
...
#java environment
export JAVA_HOME=/opt/apps/jdk***
export CLASSPATH=.:$JAVA_HOME/lib

export PATH=$PATH:$JAVA_HOME/bin
...

source /etc/profile
#3.查看版本
java -version 
javac

2.2Hadoop(全分布式)安装

#1.上传并解压安装
tar -zxvf hadoop***.tar.gz -C /opt/apps/
#2.配置环境变量
#hadoop environment
export HADOOP_HOME=/opt/apps/hadoop***
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_YARN_HOME=$HADOOP_HOME

export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

source /etc/profile

2.3配置文件
2.3.1配置hadoop-env.sh

#hadoop-env.sh
vi /opt/apps/hadoop***/etc/hadoop/hadoop-env.sh
...
# The java implementation to use.
export JAVA_HOME=/opt/apps/jdk***
...

2.3.2配置hdfs-site.xml

#hdfs-site.xml
<configuration>
        <property>
                <name>dfs.replication</name>
                <value>3</value>
        </property>
		
        <property>
                <name>dfs.namenode.name.dir</name>
                <value>file:///opt/apps/hadoop/hdpdata/dfs/name</value>
        </property>
        <property>
                <name>dfs.datanode.name.dir</name>
                <value>file:///opt/apps/hadoop/hdpdata/dfs/data</value>
        </property>

        <property>
                <name>dfs.namenode.checkpoint.dir</name>
                <value>file:///opt/apps/hadoop/hdpdata/dfs/caname</value>
        </property>
        <property>
                <name>dfs.namenode.checkpoint.edits.dir</name>
                <value>file:///opt/apps/hadoop/hdpdata/dfs/caname</value>
        </property>
</configuration>

2.3.3配置core-site.xml

<!-- 端口号默认8020 -->
		<property>
                <name>fs.defaultFS</name>
                <value>hdfs://cdh01:9000</value>
        </property>
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/opt/apps/hadoop/hdpdata</value>
        </property>

2.3.4配置mapred-site.xml

mv mapred-site.xml.template  mapred-site.xml

vi mapred-site.xml
		<property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>
		<property>
                <name>mapreduce.jobhistory.address</name>
                <value>cdh01:10020</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.webapp.address</name>
                <value>cdh01:19888</value>
        </property>

2.3.4配置yarn-site.xml

<configuration>
          <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
          </property>
          <property>
                <name>yarn.resourcemanager.hostname</name>
                <value>cdh01</value>
          </property>
</configuration>

2.3.5配置slaves

vi slaves
cdh01
cdh02
cdh03

2.4克隆虚拟机、修改IP、免密登陆

#免密登陆
ssh-keygen

ssh-copy-id cdh02
ssh-copy-id cdh03
ssh-copy-id cdh01

ssh cdh01
ssh localhost
ssh 0.0.0.0

2.5格式化namenode

hdfs namenode -format 

2.6启动测试hadoop

start-all.sh/stop-all.sh

start-dfs.sh
start-yarn.sh

stop-dfs.sh
stop-yarn.sh

#单个启动/关闭nodenode等节点
hadoop-daemon datanode/namenode start/stop

3.安装Hive

3.1安装MySQL（安装到第二台机器）
3.1.0替换aliyun的yum源

#下载wget命令
yum -y install wget
wget -O /etc/yum.repos.d/Centos-7.repo http://mirrors.aliyun.com/repo/Centos-7.repo
cd /etc/yum.repos.d
mv CentOS-Base.repo CentOS-Base.repo.bak
mv Centos-7.repo CentOS-Base.repo
yum clean all
yum repolist

3.1.1rpm安装mysql

#在第二台机器上上传mysql的rpm包
1.yum -y localinstall mysql-community-release-el6-5.noarch.rpm
2.yum -y install mysql-server
#安装完毕
3.service mysqld start
4.mysqladmin -uroot password '123456'
5.mysql -uroot -p123456
6.远程授权
GRANT ALL PRIVILEGES ON *.* TO root@'%' IDENTIFIED BY '123456' with grant option;
flush privileges;
GRANT ALL PRIVILEGES ON *.* TO root@'localhost' IDENTIFIED BY '123456' with grant option;
flush privileges;
7.Navicat远程连接并创建hive库,字符集latin1

3.1.2yum安装mysql

1. 上传安装包：mysql-5.7.28-1.el7.x86_64.rpm-bundle.tar
	说明：tar包里存储了多个rpm包。需要释放
2. 拆包,找到相关的rpm包
tar -xvf mysql-5.7.28-1.el7.x86_64.rpm-bundle.tar  -C ./
    mysql-community-embedded-5.7.28-1.el7.x86_64.rpm
    mysql-community-libs-compat-5.7.28-1.el7.x86_64.rpm
    mysql-community-devel-5.7.28-1.el7.x86_64.rpm
    mysql-community-embedded-compat-5.7.28-1.el7.x86_64.rpm
    mysql-community-libs-5.7.28-1.el7.x86_64.rpm
    mysql-community-test-5.7.28-1.el7.x86_64.rpm
    mysql-community-common-5.7.28-1.el7.x86_64.rpm
    mysql-community-embedded-devel-5.7.28-1.el7.x86_64.rpm
    mysql-community-client-5.7.28-1.el7.x86_64.rpm
    mysql-community-server-5.7.28-1.el7.x86_64.rpm
3. 安装的rpm包：
mysql-community-server-5.7.28-1.el7.x86_64.rpm
步骤：
1. 查询CentOS7.7中是否已经安装了mariadb。如果安装了，请卸载（因为与mysql不兼容） 
rpm -qa | grep mariadb
mariadb-libs-5.5.64-1.el7.x86_64             <==强制卸载
rpm -e mariadb-libs-5.5.64-1.el7.x86_64 --nodeps    
2. 安装mysql-common  因为被mysql-libs依赖
rpm -ivh mysql-community-common-5.7.28-1.el7.x86_64.rpm
3. 安装mysql-libs 因为被mysql-client依赖
rpm -ivh mysql-community-libs-5.7.28-1.el7.x86_64.rpm
4. 安装mysql-client 因为被mysql-server依赖
rpm -ivh mysql-community-client-5.7.28-1.el7.x86_64.rpm
5. 安装依赖插件：perl 和net-tools
yum -y install perl net-tools
6. 安装mysql-server
rpm -ivh mysql-community-server-5.7.28-1.el7.x86_64.rpm
mysql服务端是安装好了，但是服务项没有启动，所以下一步要进行的就是启动服务项
7. 启动mysql的服务项，并检查
systemctl start mysqld
systemctl status mysqld
8. 查找初始密码（启动服务时，会生成/var/log/mysqld.log文件）
cat /var/log/mysqld.log  | grep password
2020-04-08T08:15:04.180438Z 1 [Note] A temporary password is generated for root@localhost: dwtqdjaHn0;p
2020-04-08T08:16:02.523447Z 2 [Note] Access denied for user 'root'@'localhost' (using password: NO)

9. 登陆mysql（mysql命令在rpm安装自动安装到相应path对应的目录下了）,复制临时密码
mysql -uroot -p

10. 修改密码策略级别为low（mysql5.7.28的密码策略比较高，因此要先设置为low）
mysql> set global validate_password_policy=low;
set global validate_password_length=6;

11. 修改密码（此时的密码长度要求时8位以上的字符）
mysql> alter user root@localhost identified by '123456';

12.远程授权
GRANT ALL PRIVILEGES ON *.* TO root@'%' IDENTIFIED BY '123456' with grant option;
flush privileges;
GRANT ALL PRIVILEGES ON *.* TO root@'localhost' IDENTIFIED BY '123456' with grant option;
flush privileges;

13.本机电脑Navicat创建数据库

扩展：策略内容
mysql> show variables like "%validate_password%";

3.2Hive解压安装

tar -zxvf apache-hive***.tar.gz -C /opt/apps/

#配置环境变量
vi /etc/profile
export HIVE_HOME=/opt/apps/hive***
export PATH=$PATH:$HIVE_HOME/bin

source /etc/profile

3.3配置hive-env.sh

mv hive-env.sh.template  hive-env.sh
vi hive-env.sh

export JAVA_HOME=/opt/apps/jdk***
export HADOOP_HOME=/opt/apps/hadoop***
export HIVE_CONF_DIR=/opt/apps/hive***/conf
export HIVE_AUX_JARS_PATH=/opt/apps/hive***/lib

3.4配置hive-site.xml

#hive-site.xml默认的文件内容过多 可以新建hive-site.xml只配置需要的参数即可
vi /opt/apps/hive/conf/hive-site.xml
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!--
   Licensed to the Apache Software Foundation (ASF) under one or more
   contributor license agreements.  See the NOTICE file distributed with
   this work for additional information regarding copyright ownership.
   The ASF licenses this file to You under the Apache License, Version 2.0
   (the "License"); you may not use this file except in compliance with
   the License.  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.
-->
<configuration>
<!--连接mysql的url地址-->
        <property>
                <name>javax.jdo.option.ConnectionURL</name>
                <value>jdbc:mysql://cdh02:3306/hive?createDatabaseIfNotExist=true&amp;characterEncoding=latin1</value>
        </property>
<!--mysql的驱动类-->
        <property>
                <name>javax.jdo.option.ConnectionDriverName</name>
                <value>com.mysql.jdbc.Driver</value>
        </property>
<!--mysql的用户名-->
        <property>
                <name>javax.jdo.option.ConnectionUserName</name>
                <value>root</value>
        </property>
<!--mysql的密码-->
        <property>
                <name>javax.jdo.option.ConnectionPassword</name>
                <value>123456</value>
        </property>
<!--是否在Hive提示中包含当前数据库-->
        <property>
                <name>hive.cli.print.current.db</name>
                <value>true</value>
        </property>
<!--让Hive决定是否在本地模式下自动运行-->
        <property>
                <name>hive.exec.mode.local.auto</name>
                <value>true</value>
        </property>
</configuration>

3.5上传mysql的驱动jar包

将mysql的驱动包 mysql-connector-java-*-bin.jar 上传到$HIVE_HOME/lib下

3.6测试hive

hive

hive --service metastore &
hive --service hiveserver2 &

未完待更