spark streaming 实战入门 （结合github.com/apache/spark的NetworkWordCount.scala案例）

spark解决报错WARN util.Utils: Service 'sparkWorker' could not bind on port 0. Attempting port 1.

 Service 'sparkDriver' could not bind on a random free port. 

这个报错提示是在spark启动从节点时出现的。 
解决的方法是，在spark-env.sh中加入一条 

SPARK_LOCAL_IP=127.0.0.1 

https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/NetworkWordCount.scala参考这个网址的代码进行的实战

使用spak-submit来提交我们的spark应用程序来运行的脚本（生产上面用的比较多）

./spark-submit --master local[2] \
 --class org.apache.spark.examples.streaming.NetworkWordCount \
 --name NetworkWordCount \

 /home/hadoop/app/spark-2.2.0-bin-2.6.0-cdh5.7.0/examples/jars/spark-examples_2.11-2.2.0.jar hadoop000 9999

使用spark shell

./spark-shell --master local[2]

 import org.apache.spark.streaming.{Seconds, StreamingContext}
 val ssc = new StreamingContext(sc, Seconds(1))
 val lines = ssc.socketTextStream("hadoop000", 9999)
 val words = lines.flatMap(_.split(" "))
 val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _)
 wordCounts.print()
 ssc.start()
 ssc.awaitTermination