- 博客(163)
- 收藏
- 关注
RSS订阅
1原创 采集OIDD数据
[ods@master ~]$ mkdir ctyun[ods@master ~]$ lsctyun students.txt[ods@master ~]$ cd ctyun/[ods@master ctyun]$ ls[ods@master ctyun]$ pwd/home/ods/ctyun[ods@master ctyun]$ mkdir oidd[ods@master ctyun]$ cd oidd/[ods@master oidd]$ ls[ods@master oidd].
2022-05-31 20:32:46
695
原创 flume安装配置
[root@master soft]# vim /etc/profilealias soft='cd /usr/local/soft/'[root@master soft]# source /etc/profile[root@master soft]# soft[root@master soft]# cd ~[root@master ~]# pwd/root[root@master ~]# soft[root@master soft]# pwd/usr/local/soft[.
2022-05-31 20:00:54
89
原创 权限的控制
[root@master ~]# cd /usr/local/soft/hadoop-2.7.6/[root@master hadoop-2.7.6]# lsbin include libexec logs README.txt shareetc lib LICENSE.txt NOTICE.txt sbin tmp[root@master hadoop-2.7.6]# cd etc/[root@master etc]# lshado.
2022-05-31 15:47:45
303
原创 人体的指标
<dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-mllib_2.11</artifactId> <version>2.4.5</version></dependency>package com.shujia.mllibimport org.apache.spark.ml.{featur..
2022-05-25 21:08:51
85
原创 StructuredStreaming
package com.shujia.streamingimport org.apache.spark.sql.streaming.OutputModeimport org.apache.spark.sql.{DataFrame, SaveMode, SparkSession}object Demo05StructuredStreaming { def main(args: Array[String]): Unit = { //创建SparkSession val spar.
2022-05-24 10:58:47
85
原创 缉查布控操作
package com.shujia.streamingimport org.apache.spark.broadcast.Broadcastimport org.apache.spark.sql.SparkSessionimport org.apache.spark.streaming.dstream.{DStream, ReceiverInputDStream}import org.apache.spark.streaming.{Durations, StreamingContext}...
2022-05-20 21:11:26
288
原创 滑动窗口操作
package com.shujia.streamingimport org.apache.spark.sql.SparkSessionimport org.apache.spark.streaming.dstream.DStreamimport org.apache.spark.streaming.{Durations, StreamingContext}object Demo03Window { def main(args: Array[String]): Unit = { /.
2022-05-20 19:22:30
262
原创 Action算子、Pi
package com.shujia.coreimport com.shujia.core.Demo10Join.Studentimport org.apache.spark.{SparkConf, SparkContext}import org.apache.spark.rdd.RDDobject Demo16Action { def main(args: Array[String]): Unit = { //常见的Action算子 //foreach take col.
2022-05-19 21:24:00
191
原创 有状态算子
package com.shujia.streamingimport org.apache.spark.sql.SparkSessionimport org.apache.spark.streaming.dstream.{DStream, ReceiverInputDStream}import org.apache.spark.streaming.{Durations, StreamingContext}object Demo01WordCountOnStreaming { d...
2022-05-19 16:28:16
233
原创 SparkStreaming介绍及开发环境搭建
<dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-streaming_2.11</artifactId> <version>2.4.5</version></dependency>package com.shujia.streamingimport org.apache.spa...
2022-05-19 11:17:34
461
原创 GroupByKey VS ReduceByKey
package com.shujia.coreimport org.apache.spark.rdd.RDDimport org.apache.spark.{SparkConf, SparkContext}object Demo11Cartesian { def main(args: Array[String]): Unit = { //创建Spark Context val conf: SparkConf = new SparkConf() conf.setAppN.
2022-05-18 20:48:02
290
原创 决定RDD分区数因素、关联
package com.shujia.coreimport org.apache.spark.{SparkConf, SparkContext}import org.apache.spark.rdd.RDDobject Demo09Union { def main(args: Array[String]): Unit = { //创建Spark Context val conf: SparkConf = new SparkConf() conf.setAppName(.
2022-05-18 19:59:14
297
原创 SparkOnHive
package com.shujia.sqlimport org.apache.spark.sql.expressions.Windowimport org.apache.spark.sql.{DataFrame, SparkSession}object Demo06SparkOnHive { def main(args: Array[String]): Unit = { /** * 通过enableHiveSupport()可以开启Hive的支持 * 需要在po.
2022-05-18 11:03:47
302
原创 Spark SQL写代码的几种方式
package com.shujia.sqlimport org.apache.spark.sql.expressions.Windowimport org.apache.spark.sql.{DataFrame, Dataset, Row, SparkSession}object Demo04DSL { def main(args: Array[String]): Unit = { val spark: SparkSession = SparkSession .buil.
2022-05-18 10:25:56
876
原创 Burks练习题、JD Log练习题
公司代码,年度,1月-------------------------12月的收入金额burk,year,tsl01,tsl02,tsl03,tsl04,tsl05,tsl06,tsl07,tsl08,tsl09,tsl10,tsl11,tsl12853101,2010,100200,25002,19440,20550,14990,17227,40990,28778,19088,29889,10990,20990853101,2011,19446,20556,14996,17233,40996,2..
2022-05-17 17:00:10
222
空空如也
TA创建的收藏夹 TA关注的收藏夹
TA关注的人