一、部署(基于CDH 的spark-yarn模式)
下载源码编译(目前官网已经提供编译好的jar包了)【https://dist.apache.org/repos/dist/release/carbondata/】
mvn -DskipTests -Pspark-2.1 -Dspark.version=2.1.0 clean package
mvn -DskipTests -Pspark-2.2 -Dspark.version=2.2.1 clean package
修改配置文件carbon.properties文件
############## System Configuration ##################
#Mandatory. Carbon Store path
carbon.storelocation=hdfs://hostname:8020/carbon
#Base directory for Data files
#carbon.ddl.base.hdfs.url=hdfs://hostname:8020/carbon/data
#Path where the bad records are stored
carbon.badRecords.location=/opt/test
################################################
进入cloudera manager的webUI
点击spark2组件、进入配置 ,在搜索框输入spark-default
在进行配置
spark.master=yarn-client
spark.yarn.dist.files=/opt/cloudera/parcels/CDH-5.10.2-1.cdh5.10.2.p0.5/lib/spark/conf/carbon.properties
spark.yarn.dist.archives=/opt/cloudera/parcels/CDH-5.10.2-1.cdh5.10.2.p0.5/lib/spark/carbonlib/carbondata.tar.gz
spark.executor.extraJavaOptions=-Dcarbon.properties.filepath=carbon.properties
spark.executor.extraClassPath= carbondata.tar.gz/carbonlib/*
spark.driver.extraClassPath=/opt/cloudera/parcels/CDH-5.10.2-1.cdh5.10.2.p0.5/lib/spark/carbonlib/*
spark.driver.extraJavaOptions=-Dcarbon.properties.filepath=/opt/cloudera/parcels/CDH-5.10.2-1.cdh5.10.2.p0.5/lib/spark
/conf/carbon.properties
spark.dynamicAllocation.enabled true
spark.shuffle.service.enabled true
spark.dynamicAllocation.minExecutors 0
spark.dynamicAllocation.maxExecutors 20
保存重启
二、启动:
spark2-shell
三、使用
3.1创建上下文
import org.apache.carbondata.core.util.CarbonProperties
import org.apache.carbondata.core.constants.CarbonCommonConstants
CarbonProperties.getInstance().addProperty(CarbonCommonConstants.LOCK_TYPE, “HDFSLOCK”)
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.CarbonSession._
val carbon = SparkSession.builder().config(sc.getConf).getOrCreateCarbonSession(“hdfs://hostname:/carbon”)
3.2创表
carbon.sql("create table if not exists log_T (IP string," +
"website string," +
"cnid string," +
"uid string," +
"version string," +
"stored by 'carbondata'")
3.3 入库(加载数据)
carbon.sql("load data inpath '/carbon/carbontest.gz' " +
"into table log_T " +
"options('header'='false'," +
"'fileheader'='ip," +
"website," +
"datestr," +
"timestr," +
"cnid," +
"uid," +
"version'," +
"'DELIMITER'='\t'," +
"'MULTILINE'='true'," +
"'ESCAPECHAR'='\t'," +
"'SKIP_EMPTY_LINE'='TRUE'," +
"'SINGLE_PASS'='TRUE')")
DataFrame:
df.write.format(“carbondata").options("tableName", "log_T")) .mode(SaveMode.Overwrite).save()
3.4 查询:
carbon.sql(“select * from log_T”)
3.5 更新和删除
更新一列
UPDATE table1 log_T
SET log_T.REVENUE = log_T.REVENUE - 10 WHERE log_T.PRODUCT = 'uid'
Modify two columns in table1
更新两列
UPDATE table1 log_T
SET (log_T.PRODUCT, log_T.REVENUE) =
(
SELECT PRODUCT, REVENUE
FROM table2 B
WHERE log_B.CITY = log_T.CITY AND log_B.BROKER =log_T.BROKER
)
WHERE log_T.DATE BETWEEN '2017-01-01' AND '2017-01-31'
3.6 删除
carbon.sql("DELETE FROM table1 log_T WHERE log_T.CUSTOMERID =‘123’")
scala> carbon.sql("select * from log_t.carbon_tablename_new").show
+-----+-----------+
| uid | cnid |
+-----+-----------+
| 1 | 2|
| 3 | 4|
+---- +-----------+
scala> carbon.sql("delete from dtwave_dev.carbon_tablename_new a WHERE a.name='1'")
scala> carbon.sql("select * from dtwave_dev.carbon_tablename_new").show
+----+-----------+
|name|PhoneNumber|
+----+-----------+
| 3| 4|
+----+-----------+
carbon.sql("update dtwave_dev.carbon_tablename_new A SET (A.name) =
A.name WHERE A.PhoneNumber = '4'")
carbon.sql("UPDATE dtwave_dev.carbon_tablename_new a SET (a.name, a.PhoneNumber) =
( SELECT '5' as name ,'6' from dtwave_dev.carbon_tablename_new b)")
carbon.sql("UPDATE dtwave_dev.carbon_tablename_new a SET (a.name, a.PhoneNumber) =
( SELECT '5' as name ,'6' as PhoneNumber)")
四、HDFS对应的文件
4.1 HDFS 目录:
/carbon/default/log_tab
Permission
Owner
Group
Size
Last Modified
Replication
Block Size
Name
drwxr-xr-x
root
spark
0 B
Fri Mar 30 19:38:11 +0800 2018
0
0 B
drwxr-xr-x
root
spark
0 B
Fri Mar 30 19:39:05 +0800 2018
0
0 B
4.2 数据文件
/carbon/default/log_tab/Fact/Part0/Segment_0
4.3 元数据文
/carbon/carbonstore/dtwave_dev/carbon_tablename_new/Metadata