Spark
秋风小凉鱼
技术无边界!
展开
-
Spark杂记
参考spark2.3.0文档http://spark.apache.org/docs/2.3.0/quick-start.html1、从spark 2.0开始,官网强烈推荐Dataset ,它比RDD拥有更好的性能. 2、启动spak-shell[spark@big-data-1 home]$ spark-shell Setting default log leve...原创 2018-09-06 17:03:37 · 214 阅读 · 0 评论 -
Spark SQL
1、读取外部配置文件package com.study;import org.apache.spark.sql.Dataset;import org.apache.spark.sql.Row;import org.apache.spark.sql.SparkSession;/** * @author : yulei * @data : 2018/9/10 11:00 *...原创 2018-09-10 14:27:48 · 194 阅读 · 0 评论 -
Spark Window 代码片段整理
参考地址:http://spark.apache.org/docs/2.3.0/structured-streaming-programming-guide.html#operations-on-streaming-dataframesdatasetsWindow Operations on Event Time Dataset<Row> words = ... ...原创 2018-09-18 15:33:57 · 255 阅读 · 0 评论 -
Spark 检查点
1、设置检查点的代码基本如下,供参考. public static void main(String[] args) throws Exception { Function0<JavaStreamingContext> contextFactory = new Function0<JavaStreamingContext>() { ...原创 2018-09-14 10:11:09 · 591 阅读 · 0 评论