SparkStreaming的worldCount实例

1、搭建好spark环境，sparkstreaming不需要什么特殊配置即可以使用

2、SparkStreaming简介

SparkStreaming是实时计算框架，它的数据源可以是socket或kafka等各种消息组件，具体详细的知识不多赘述

3、实例

import org.apache.spark._
import org.apache.spark.storage.StorageLevel
import org.apache.spark.streaming.StreamingContext._
import org.apache.spark.streaming.{Seconds, StreamingContext}

/**
  * Counts words in UTF8 encoded, '\n' delimited text received from the network every second.
  *
  * Usage: NetworkWordCount <hostname> <port>
  * <hostname> and <port> describe the TCP server that Spark Streaming would connect to receive data.
  *
  * To run this on your local machine, you need to first run a Netcat server
  *    `$ nc -lk 9999`
  * and then run the example
  *    `$ bin/run-example org.apache.spark.examples.streaming.NetworkWordCount localhost 9999`
  */
object NetworkWordCount {
  def main(args: Array[String]) {
    if (args.length < 2) {
      System.err.println("Usage: NetworkWordCount <hostname> <port>")
      System.exit(1)
    }

    StreamingExamples.setStreamingLogLevels()

    // Create the context with a 1 second batch size
    val sparkConf = new SparkConf().setAppName("NetworkWordCount")
    val ssc = new StreamingContext(sparkConf, Seconds(1))

    // Create a socket stream on target ip:port and count the
    // words in input stream of \n delimited text (eg. generated by 'nc')
    // Note that no duplication in storage level only for running locally.
    // Replication necessary in distributed scenario for fault tolerance.
    val lines = ssc.socketTextStream(args(0), args(1).toInt, StorageLevel.MEMORY_AND_DISK_SER)
    val words = lines.flatMap(_.split(" "))
    val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _)
    wordCounts.print()
    ssc.start()
    ssc.awaitTermination()
  }
}

4、运行步骤
（1）在一台linux上启动nc -lk 9999，如下所示，此处即生成者
 
（2）消费者即sparkstreaming
到$SPARK_HOME下运行
bin/run-example org.apache.spark.examples.streaming.NetworkWordCount (1)步中的机器IP 9999
可以看到有数据结算打印在屏幕上：格式如下

5、注意事项
请至少安装两台机器的spark集群，否则消费者端看不到输出结果
原因见：下一遍文章