![](https://img-blog.csdnimg.cn/20201014180756757.png?x-oss-process=image/resize,m_fixed,h_64,w_64)
Hive
HeMJGaoMM
这个作者很懒,什么都没留下…
展开
-
配置支持Spark操作Hive表数据,使用Intellij
spark2版本使用SparkSession作为统一入口,所以第一步就是给SparkSession增加Hive支持: enableHiveSupport()val spark = SparkSession .builder() .appName("Spark Hive Example").master("local[*]") .enableHiveSuppor...原创 2019-01-15 14:28:53 · 816 阅读 · 0 评论 -
explode hive
select a.dia.ecuid from (select explode(body.serviceData.vehiclestatus.temstatus.diagnostics) as dia from vehicle_dtc_array where body.serviceData.vehiclestatus.temstatus.diagnostics is not null limi...原创 2019-03-06 19:22:51 · 90 阅读 · 0 评论 -
spark shell hive sql
import org.apache.spark.sql.hive.HiveContextval hiveContext = new HiveContext(sc)hiveContext.sql(“select * from …”)原创 2019-03-07 10:24:38 · 93 阅读 · 0 评论 -
对一个字段连续explode hive
SELECT vin, record_date, Latitude,Longitude,dia.ecuid,dtcFROM vehicle_dtc_array_parquetLATERAL VIEW explode(diagnostics) diaTable AS diaLATERAL VIEW explode(dia.dtcs) diaTable AS dtc;原创 2019-03-07 16:10:36 · 376 阅读 · 0 评论 -
查看hive job的log
mapred job -history JOB_ID如:mapred job -history job_1551943436571_0044JOB_ID可在yarn UI中查看到。原创 2019-03-07 18:19:05 · 1951 阅读 · 0 评论 -
top N hive sql
insert into table vin_geo_summaryselect vin,province,city from(select vin,province,cityname as city,sub.c,rank() over (partition by vin order by sub.c desc) as rfrom(select vin,province,cityname,...原创 2019-04-11 19:06:07 · 217 阅读 · 0 评论 -
Spark Sql Read Parquet Files; Number of Partitions.
hive metastore 和 parquet 转化的方式通过 spark.sql.hive.convertMetastoreParquet 控制,默认为 true。如果设置为 true ,会使用 org.apache.spark.sql.execution.FileSourceScanExec ,否则会使用 org.apache.spark.sql.hive.execution.HiveTa...原创 2019-04-18 10:18:31 · 447 阅读 · 1 评论