![](https://img-blog.csdnimg.cn/20201014180756913.png?x-oss-process=image/resize,m_fixed,h_64,w_64)
Spark
秋风小凉鱼
技术无边界!
展开
-
Spark杂记
参考spark2.3.0文档http://spark.apache.org/docs/2.3.0/quick-start.html 1、从spark 2.0开始,官网强烈推荐Dataset ,它比RDD拥有更好的性能. 2、启动spak-shell [spark@big-data-1 home]$ spark-shell Setting default log leve...原创 2018-09-06 17:03:37 · 213 阅读 · 0 评论 -
Spark SQL
1、读取外部配置文件 package com.study; import org.apache.spark.sql.Dataset; import org.apache.spark.sql.Row; import org.apache.spark.sql.SparkSession; /** * @author : yulei * @data : 2018/9/10 11:00 *...原创 2018-09-10 14:27:48 · 193 阅读 · 0 评论 -
Spark Window 代码片段整理
参考地址: http://spark.apache.org/docs/2.3.0/structured-streaming-programming-guide.html#operations-on-streaming-dataframesdatasets Window Operations on Event Time Dataset<Row> words = ... ...原创 2018-09-18 15:33:57 · 254 阅读 · 0 评论 -
Spark 检查点
1、设置检查点的代码基本如下,供参考. public static void main(String[] args) throws Exception { Function0<JavaStreamingContext> contextFactory = new Function0<JavaStreamingContext>() { ...原创 2018-09-14 10:11:09 · 589 阅读 · 0 评论