大数据体系架构实践

大数据体系架构实践

Hadoop安装部署

一.网络环境配置

下载安装虚拟机Vbox配置JDK

Cd /etc/sysconfig/network-scripts

Vi ifcfg-enp0s3

二.创建目录

mkdir /opt/tools

mkdir /opt/hadoop

三.Java.JDK

java -version检测是否成功

集群安装过程
安装hive
$ tar -zxvf apache-hive-2.1.1-bin.tar.gz

$ mv apache-hive-2.1.1-bin.tar.gz hive-2.1.1
配置环境变量(同上)

export HIVE_HOME=/home/ranrl/opt/hive-2.1.1

export HIVE_CONF_DIR=$HIVE_HOME/conf

export PATH=$PATH:$HIVE_HOME/bin

在hive的目录下，新建warehouse、tmp、log文件夹。
修改配置文件，下面是hive的配置文件conf目录

-rw-r–r-- 1 ranrl root 1596 Nov 29 05:32 beeline-log4j2.properties.template

-rw-r–r-- 1 ranrl root 229198 Nov 30 03:46 hive-default.xml.template

-rw-r–r-- 1 ranrl root 2378 Nov 29 05:35 hive-env.sh.template

-rwxr-xr-x 1 ranrl root 2274 Feb 7 17:09 hive-exec-
log4j2.properties

-rw-r–r-- 1 ranrl root 2274 Nov 29 05:32 hive-exec-log4j2.properties.template

-rw-r–r-- 1 ranrl root 2925 Nov 29 05:32 hive-log4j2.properties.template

-rw-r–r-- 1 ranrl root 2060 Nov 29 05:32 ivysettings.xml

-rw-r–r-- 1 ranrl root 2719 Nov 29 05:32 llap-cli-log4j2.properties.template

-rw-r–r-- 1 ranrl root 4353 Nov 29 05:35 llap-daemon-log4j2.properties.template
-rw-r–r-- 1 ranrl root 2662 Nov 29 05:32 parquet-logging.properties

$ cp hive-env.sh.template hive-env.sh
$ cp hive-log4j2.properties.template hive-log4j2.properties
$ cp hive-default.xml.template hive-site.xml

修改后配置文件如下：

-rw-r–r-- 1 ranrl root 1596 Nov 29 05:32 beeline-log4j2.properties.template
-rw-r–r-- 1 ranrl root 229198 Nov 30 03:46 hive-default.xml.template
-rwxr-xr-x 1 ranrl root 2440 Feb 7 17:31 hive-env.sh
-rw-r–r-- 1 ranrl root 2378 Nov 29 05:35 hive-env.sh.template
-rwxr-xr-x 1 ranrl root 2274 Feb 7 17:09 hive-exec-log4j2.properties
-rw-r–r-- 1 ranrl root 2274 Nov 29 05:32 hive-exec-log4j2.properties.template
-rwxr-xr-x 1 ranrl root 2925 Feb 7 17:09 hive-log4j2.properties
-rw-r–r-- 1 ranrl root 2925 Nov 29 05:32 hive-log4j2.properties.template
-rwxr-xr-x 1 ranrl root 2514 Feb 7 18:43 hive-site.xml
-rw-r–r-- 1 ranrl root 2060 Nov 29 05:32 ivysettings.xml
-rw-r–r-- 1 ranrl root 2719 Nov 29 05:32 llap-cli-log4j2.properties.template
-rw-r–r-- 1 ranrl root 4353 Nov 29 05:35 llap-daemon-log4j2.properties.template
-rw-r–r-- 1 ranrl root 2662 Nov 29 05:32 parquet-logging.properties

修改hive-env.sh，去掉前面的注释#，加上相应的路径
export HADOOP_HEAPSIZE=1024
HADOOP_HOME=/home/ranrl/opt/hadoop-2.7.3
export HIVE_CONF_DIR=/home/ranrl/opt/hive-2.1.1/conf
export HIVE_AUX_JARS_PATH=/home/ranrl/opt/hive-2.1.1/lib

这儿采用mysql作为hive的元数据库
在自己的mysql中为hive新建一个用户：grant all privileges on . to ‘hive’ @ ‘%’ identified by ‘password’
修改hive-site.xml文件，添加相应的路径和mysql数据库，用户名，密码

hive.metastore.warehouse.dir /home/ranrl/opt/hive-2.1.1/warehouse location of default database for the warehouse hive.querylog.location /home/ranrl/opt/hive-2.1.1/log Location of Hive run time structured log file hive.downloaded.resources.dir /home/ranrl/opt/hive-2.1.1/tmp Temporary local directory for added resources in the remote file system. javax.jdo.option.ConnectionURL jdbc:mysql://localhost:3306/hive JDBC connect string for a JDBC metastore. To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL. For example, jdbc:postgresql://myhost/db?ssl=true for postgres database. javax.jdo.option.ConnectionDriverName com.mysql.jdbc.Driver Driver class name for a JDBC metastore javax.jdo.option.ConnectionUserName hive Username to use against metastore database javax.jdo.option.ConnectionPassword password password to use against metastore database

5. 下载mysql-connector-java
   地址：https://dev.mysql.com/downloads/connector/j/
   解压得到mysql-connector-java-5.1.40.jar,拷贝到hive的lib目录下。

6. 在进入hive目录下的bin目录，运行命令：./schematool -initSchema -dbType mysql
   现在hive已经安装成功，输入hive命令，即可运行hive
   show databases;
   create databases test;
   use test;
   hive> CREATE TABLE test(id int,name string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ‘\t’;
   select * from test ；运行正常说明hive已经安装成功