硬件准备:
系统:CentOS 7.6(1810)
CPU:4核
内存:16G
软件准备:
注意:
在carbondata-1.6.1中,有组件版本限制,具体为:
hadoop支持到2.7.2
hive支持到1.2.1(见carbondata-parent-1.6.1/integration/hive/pom.xml)
spark支持到2.x(见carbondata-parent-1.6.1/integration/spark2/pom.xml)
安装:
1、安装jdk
yum install -y java-1.8.0-openjdk java-devel
配置JAVA_HOME
vim /root/.bashrc
在末尾添加
export JAVA_HOME=/usr/lib/jvm/java
2、配置本机免登陆
ssh-keygen
一路回车
ssh-copy-id root@你的IP
3、安装单机版hadoop
关闭防火墙:
systemctl stop firewalld && systemctl disable firewalld
解压hadoop:
上传hadoop-2.7.2.tar.gz到/root目录下
cd /root && tar -xvf hadoop-2.7.2.tar.gz
新建/data文件夹:
mkdir /data
编辑
vim /root/hadoop-2.7.2/etc/hadoop/core-site.xml
内容为:
fs.defaultFS
hdfs://你的IP:9000
hadoop.tmp.dir
/data/hadoop/tmp
初始化namenode:
cd /root/hadoop-2.7.2/bin && ./hadoop namenode -format
启动hadoop:
cd /root/hadoop-2.7.2/sbin && ./start-all.sh
添加HADOOP_HOME环境变量:
vim /root/.bashrc
在末尾添加
export HADOOP_HOME=/root/hadoop-2.7.2
4、安装单机版hive
上传apache-hive-1.2.1-bin.tar.gz到/root目录下,执行以下命令:
cd /root && tar -xvf apache-hive-1.2.1-bin.tar.gz
安装mysql5.7,配置用户密码(略)
下载myql驱动,拷贝至hive对应目录
cp mysql-connector-java-5.1.47.jar /root/apache-hive-1.2.1-bin/lib/
编辑hive-site.xml
vim /root/apache-hive-1.2.1-bin/conf/hive-site.xml
内容为:
javax.jdo.option.ConnectionURL
jdbc:mysql://127.0.0.1:3306/hive?createDatabaseIfNotExist=true&characterEncoding=UTF-8&useSSL=false
javax.jdo.option.ConnectionUserName
你的数据库用户
javax.jdo.option.ConnectionPassword
你的数据库密码
javax.jdo.option.ConnectionDriverName
com.mysql.jdbc.Driver
hive.metastore.schema.verification
false
初始化hive schema:
cd /root/apache-hive-1.2.1-bin/bin && ./schematool -initSchema -dbType mysql
添加HIVE_HOME环境变量:
vim /root/.bashrc
在末尾添加
export HIVE_HOME=/root/apache-hive-1.2.1-bin
4、安装单机版spark
上传spark-2.2.1-bin-hadoop2.7.tgz到/root目录下,执行以下命令:
cd /root && tar -xvf spark-2.2.1-bin-hadoop2.7.tgz
上传apache-carbondata-1.6.1-bin-spark2.2.1-hadoop2.7.2.jar到/root目录下,执行以下命令:
cd /root && cp apache-carbondata-1.6.1-bin-spark2.2.1-hadoop2.7.2.jar /root/spark-2.2.1-bin-hadoop2.7/jars
上传mysql-connector-java-5.1.47.jar到/root目录下,执行以下命令:
cd /root && cp mysql-connector-java-5.1.47.jar /root/spark-2.2.1-bin-hadoop2.7/jars
拷贝hive-site.xml到spark:
cd /root/spark-2.2.1-bin-hadoop2.7/conf && cp /root/apache-hive-1.2.1-bin/conf/hive-site.xml .
5、carbondata集成spark
新建carbondata测试数据:
cd /root && mkdir carbondata && cd carbondata
cat > sample.csv << EOF
id,name,scale,country,salary
1,yuhai,1.77,china,33000.1
2,runlin,1.70,china,33000.2
EOF
将测试数据sample.csv上传到hadoop的tmp目录:
cd /root/hadoop-2.7.2/bin && ./hadoop fs -put /root/carbondata/sample.csv /tmp
执行spark-shell:
cd /root/spark-2.2.1-bin-hadoop2.7/bin && ./spark-shell
在打开的spark-shell中执行如下命令:
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.CarbonSession._
val rootPath = "hdfs://你的机器IP:9000/user/hadoop/carbon"
val storeLocation = s"$rootPath/store"
val warehouse = s"$rootPath/warehouse"
val metaStoreDB = s"$rootPath/metastore_db"
val carbon = SparkSession.builder().enableHiveSupport().config("spark.sql.warehouse.dir", warehouse).config(org.apache.carbondata.core.constants.CarbonCommonConstants.STORE_LOCATION, storeLocation).getOrCreateCarbonSession(storeLocation, metaStoreDB)
carbon.sql("create table hive_carbon(id int, name string, scale decimal, country string, salary double) STORED BY 'carbondata'")
carbon.sql("LOAD DATA INPATH 'hdfs://你的机器IP:9000/tmp/sample.csv' INTO TABLE hive_carbon")
carbon.sql("SELECT * FROM hive_carbon").show()
可以看到:
scala> carbon.sql("SELECT * FROM hive_carbon").show()
+---+------+-----+-------+-------+
| id| name|scale|country| salary|
+---+------+-----+-------+-------+
| 1| yuhai| 2| china|33000.1|
| 2|runlin| 2| china|33000.2|
+---+------+-----+-------+-------+
至此,carbondata集成spark完成。
6、carbondata集成hive
修改hive-site.xml,添加carbondata配置:
vim /root/apache-hive-1.2.1-bin/conf/hive-site.xml
全部内容如下:
javax.jdo.option.ConnectionURL
jdbc:mysql://127.0.0.1:3306/hive?createDatabaseIfNotExist=true&characterEncoding=UTF-8&useSSL=false
javax.jdo.option.ConnectionUserName
root
javax.jdo.option.ConnectionPassword
123456
javax.jdo.option.ConnectionDriverName
com.mysql.jdbc.Driver
hive.metastore.schema.verification
false
hive.metastore.pre.event.listeners
org.apache.carbondata.hive.CarbonHiveMetastoreListener
添加carbondata-hive-1.6.1.jar到hive:
编译carbondata-parent-1.6.1,获得carbondata-hive-1.6.1.jar(位于carbondata-parent-1.6.1/integration/hive/target目录),上传carbondata-hive-1.6.1.jar到/root目录,执行以下命令:
cp /root/carbondata-hive-1.6.1.jar /root/apache-hive-1.2.1-bin/lib
添加apache-carbondata-1.6.1-bin-spark2.2.1-hadoop2.7.2.jar到hive:
上传apache-carbondata-1.6.1-bin-spark2.2.1-hadoop2.7.2.jar到/root目录下,执行以下命令:
cp /root/apache-carbondata-1.6.1-bin-spark2.2.1-hadoop2.7.2.jar /root/apache-hive-1.2.1-bin/lib
拷贝spark相关jar到hive:
cd /root/apache-hive-1.2.1-bin/lib && cp /root/spark-2.2.1-bin-hadoop2.7/jars/scala-* . && cp /root/spark-2.2.1-bin-hadoop2.7/jars/spark-catalyst_2.11-2.2.1.jar .
启动hive:
cd /root && nohup ~/apache-hive-1.2.1-bin/bin/hive --service metastore 1>metastore.log 2>&1 &
cd /root && nohup ~/apache-hive-1.2.1-bin/bin/hive --service hiveserver2 1>hiveserver2.log 2>&1 &
使用beeline连接hive查询数据:
cd /root/apache-hive-1.2.1-bin/bin && ./beeline -u 'jdbc:hive2://localhost:10000/default' -n root -p '' -e 'set hive.fetch.task.conversion=none;select * from hive_carbon;'
可以看到查询结果:
Connecting to jdbc:hive2://localhost:10000/default
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/root/apache-hive-1.2.1-bin/lib/apache-carbondata-1.6.1-bin-spark2.2.1-hadoop2.7.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Connected to: Apache Hive (version 1.2.1)
Driver: Hive JDBC (version 1.2.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
No rows affected (0.077 seconds)
INFO : Number of reduce tasks is set to 0 since there's no reduce operator
INFO : number of splits:1
INFO : Submitting tokens for job: job_local382345580_0004
INFO : The url to track the job: http://localhost:8080/
INFO : Job running in-process (local Hadoop)
INFO : 2020-02-20 14:48:00,962 Stage-1 map = 100%, reduce = 0%
INFO : Ended Job = job_local382345580_0004
+-----------------+-------------------+--------------------+----------------------+---------------------+--+
| hive_carbon.id | hive_carbon.name | hive_carbon.scale | hive_carbon.country | hive_carbon.salary |
+-----------------+-------------------+--------------------+----------------------+---------------------+--+
| 1 | yuhai | 2 | china | 33000.1 |
| 2 | runlin | 2 | china | 33000.2 |
+-----------------+-------------------+--------------------+----------------------+---------------------+--+
2 rows selected (1.653 seconds)
Beeline version 1.2.1 by Apache Hive
Closing: 0: jdbc:hive2://localhost:10000/default
至此,carbondata集成hive完成。
问题:
1、spark导入数据到表,hive查询没问题,但是hive中建表或者执行insert语句报错:暂时没时间跟踪代码解决。