![](https://img-blog.csdnimg.cn/20201014180756913.png?x-oss-process=image/resize,m_fixed,h_64,w_64)
spark
我是浣熊的微笑
大数据开发
展开
-
spark常用命令
查看报错日志:yarn logs applicationIDspark2-submit --master yarn --class com.hik.ReadHdfs test-1.0-SNAPSHOT.jar进入$SPARK_HOME目录,输入bin/spark-submit --help可以得到该命令的使用帮助。hadoop@wyy :/app/hadoop/spark100$ bin/spark-submit --helpUsage: spark-submit [options] <ap原创 2021-05-17 14:05:42 · 1610 阅读 · 0 评论 -
hadoop常用命令
1、启动hadoop所有进程start-all.sh等价于start-dfs.sh + start-yarn.sh但是一般不推荐使用start-all.sh(因为开源框架中内部命令启动有很多问题)。2、单进程启动。sbin/start-dfs.sh--------------- sbin/hadoop-daemons.sh --config .. --hostname .. start namenode ... sbin/hadoop-daemons.sh --conf...原创 2021-05-17 14:02:01 · 301 阅读 · 0 评论 -
hive常用命令
hive -e "select * from ..." > /tmp/datahive -f 文件INSERT OVERWRITE TABLE employeesPARTITION (country = 'US', state = 'OR')SELECT * FROM staged_employees seWHERE se.cnty = 'US' AND se.st = 'OR';假如需要对多个分区执行,需要扫描表多次FROM staged_employees seINSERT .原创 2021-05-11 11:29:42 · 171 阅读 · 0 评论 -
sparkSQL多条记录列转行
val view = ss.sql( """ | select field1,concat_ws(",",collect_set(field2)) as convert_field | from t1 | where field1 is not null | group by field1 |""".stripMargin)原创 2021-05-10 11:20:30 · 180 阅读 · 0 评论 -
sparkSQL根据一个字段分隔转多行
代码: val row2columnStr = """ |select * from temp_view |lateral view explode(split(field, ',')) tmpTable as result_view """.stripMargin原创 2021-05-10 10:55:19 · 586 阅读 · 0 评论 -
spark读写kudu2
maven依赖<?xml version="1.0" encoding="UTF-8"?><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/ma原创 2021-05-10 10:25:32 · 195 阅读 · 0 评论 -
sparkSQL行转列,列转行
在用spark进行数据处理过程中,避免不了行转列和列传行的操作,特此记录:1.列传行:这里举的例子是certificate_id ,telephone_number 每个身份证号可能对应多个手机号码df.createTempView("tmp") val result = sparkSession.sql( """ | select certifi...原创 2020-04-14 16:01:46 · 1947 阅读 · 0 评论 -
sparkStreaming读kafka
windows环境本地起kafka producer进行测试,windows环境安装启动kafka可参照这篇博客:https://blog.csdn.net/shenyanwei/article/details/90374859代码如下:import org.apache.kafka.clients.consumer.ConsumerConfigimport org.apache.ka...原创 2020-01-10 09:31:15 · 172 阅读 · 0 评论 -
spark读写kudu
package sparkUtilimport org.apache.kudu.spark.kudu.KuduContextimport org.apache.spark.SparkConfimport org.apache.spark.sql.SparkSessionobject SparkKudu { //kuduMasters and tableName val kud...原创 2019-09-24 22:39:25 · 2354 阅读 · 2 评论 -
spark读写Hbase数据
计算结果数据需要写入Hbase,特此记录首先贴出hbase需要的maven依赖<hbase.version>1.2.3</hbase.version> <dependency> <groupId>org.apache.hbase</groupId> <artif...原创 2019-12-05 16:45:34 · 245 阅读 · 0 评论