hadoop与hive搭建

一、Hadoop搭建

1、创建hadoop名称的用户

addset hadoop
passwd hadoop

2、切换到root用户，关闭selinux

su - root
关闭selinux：vim /etc/selinux/config
SELINUX=disabled

3、切换到hadoop用户

su - hadoop

4、hadoop集群使用免密登录

cd 进入hadoop用户的家目录
ssh-keygen -t rsa [输入完后连按4个回车]
ssh localhost [yes，输入hadoop用户的密码]
ssh-copy-id localhost [输入hadoop用户的密码]

5、使用xftp上传hadoop、hive、java安装文件

6、解压软件包到/opt/module

cd
tar -zxvf ./jdk-8u181-linux-x64.tar.gz -C /opt/module/
tar -zxvf ./hadoop-2.7.3.tar.gz -C /opt/module/
tar -zxvf ./apache-hive-3.1.1-bin.tar.gz -C /opt/module/

7、编辑环境变量

ls -a #查看隐藏文件
vim ~/.bash_profile ，在文件末尾内容添加如下：

JAVA_HOME=/opt/module/jdk1.8.0_181
HADOOP_HOME=/opt/module/hadoop-2.7.3
HIVE_HOME=/opt/module/apache-hive-3.1.1-bin
PATH=$PATH:$HOME/bin:$JAV{A}_{H}OME/bin:$HADOOP_HOME/bin:$HADOO{P}_{H}OME/sbin:$HIVE_HOME/bin

export JAVA_HOME
export HADOOP_HOME
export HIVE_HOME
export PATH

8、重新加载该文件，使环境变量立即生效

source ~/.bash_profile

9、修改以下文件的主机名、及路径

修改hadoop的配置文件:cd /opt/module/hadoop-2.7.3/etc/hadoop
1.vim ./hadoop-env.sh
export JAVA_HOME=/opt/module/jdk1.8.0_181
2.vim ./mapred-env.sh
export JAVA_HOME=/opt/module/jdk1.8.0_181
3.vim ./yarn-env.sh
export JAVA_HOME=/opt/module/jdk1.8.0_181
4.vim ./core-site.xml

    <!-- 指定HDFS中NameNode的地址 -->
    <property>
            <name>fs.defaultFS</name>
            <value>hdfs://node100:9000</value>
    </property>

    <!-- 指定Hadoop运行时产生文件的存储目录 -->
    <property>
            <name>hadoop.tmp.dir</name>
            <value>/opt/module/hadoopdata</value>
    </property>

    <property>
            <name>hadoop.proxyuser.hadoop.hosts</name>
            <value>*</value>
    </property>
    
    <property>
            <name>hadoop.proxyuser.hadoop.groups</name>
            <value>*</value>
    </property>

5.vim ./hdfs-site.xml

    <!-- 指定HDFS副本的数量 -->
    <property>
            <name>dfs.replication</name>
            <value>1</value>
    </property>

    <!-- 指定Hadoop辅助名称节点主机配置 -->
    <property>
            <name>dfs.namenode.secondary.http-address</name>
            <value>node100:50090</value>
    </property>

6.cp ./mapred-site.xml.template ./mapred-site.xml
vim ./mapred-site.xml

<!-- 指定MR运行在yarn上 -->
<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>

7.vim ./yarn-site.xml

    <!-- Reducer获取数据的方式 -->
    <property>
            <name>yarn.nodemanager.aux-services</name>
            <value>mapreduce_shuffle</value>
    </property>

    <!-- 指定YARN的ResourceManager的地址 -->
    <property>
            <name>yarn.resourcemanager.hostname</name>
            <value>node100</value>
    </property>

    <!-- 关闭虚拟内存检查 -->
    <property>
            <name>yarn.nodemanager.vmem-check-enabled</name>
            <value>false</value>
    </property>

8.vim ./slaves
node100

10、格式化hadoop集群

hdfs namenode -format

11、启动/关闭hadoop集群

start-all.sh
stop-all.sh

12、测试hadoop是否集群开启

查看jps，有6个进程说明hadoop开启
[hadoop@localhost ~]$ jps
21968 SecondaryNameNode
22593 Jps
22248 NodeManager
22138 ResourceManager
21788 DataNode
21646 NameNode

二、hive搭建

1、在HDFS上创建存储Hive数据的目录

hadoop fs -mkdir /tmp
hadoop fs -mkdir -p /user/hive/warehouse
hadoop fs -chmod g+w /tmp
hadoop fs -chmod g+w /user/hive/warehouse
在hive的软件目录(/opt/module/apache-hive-3.1.1-bin/)下执行初始化命令
bin/schematool -dbType derby -initSchema
初始化成功后就会在hive的安装目录下生成derby.log日志文件和metastore_db元数据库

2、查询hadoop文件系统的根目录

hadoop fs -ls /;

3、可以在web查询hadoop目录

登录192.168.5.100:50070

4、启动hive

[hadoop@localhost apache-hive-3.1.1-bin]$ bin/hive
which: no hbase in (/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/hadoop/.local/bin:/home/hadoop/bin:/home/hadoop/bin:/opt/module/jdk1.8.0_181/bin:/opt/module/hadoop-2.7.3/bin:/opt/module/hadoop-2.7.3/sbin:/opt/module/apache-hive-3.1.1-bin/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/apache-hive-3.1.1-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = 4e9849e6-a9b1-4101-9288-6fcf11d0871f

Logging initialized using configuration in jar:file:/opt/module/apache-hive-3.1.1-bin/lib/hive-common-3.1.1.jar!/hive-log4j2.properties Async: true
Hive Session ID = 9992744b-7764-4dd8-aa5b-70777ffc7c3f
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive>