Spark-Streaming的最简单使用

最新推荐文章于 2024-08-16 09:21:03 发布

绛门人

最新推荐文章于 2024-08-16 09:21:03 发布

阅读量1.1k

点赞数 1

分类专栏： spark

本文链接：https://blog.csdn.net/wt346326775/article/details/72886207

版权

spark 专栏收录该内容

34 篇文章 0 订阅

订阅专栏

<dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.10</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming-kafka_2.10</artifactId>
            <version>1.6.1</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming-flume_2.10</artifactId>
            <version>${spark.version}</version>
        </dependency>

程序1：

import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.streaming.{Seconds, StreamingContext}
/**
  * Created by Administrator on 2017/6/6.
  */
object StreamingWordCount {
  def main(args: Array[String]) {

    //LoggerLevels.setStreamingLogLevels()
    //StreamingContext
    val conf = new SparkConf().setAppName("StreamingWordCount").setMaster("local[2]")
    val sc = new SparkContext(conf)
    val ssc = new StreamingContext(sc, Seconds(5))
    //接收数据
    val ds = ssc.socketTextStream("192.168.33.62", 9999)
    //DStream是一个特殊的RDD
    //hello tom hello jerry
    val result = ds.flatMap(_.split(" ")).map((_, 1)).reduceByKey(_+_)
    //打印结果
    result.print()
    ssc.start()
    ssc.awaitTermination()
  }
}

再另一个shell中：使用nc命令，不断向端口发送数据

yum -y install nc

nc -lk 9999