spark streaming demo

spark streaming demo

1. spark streaming代码

val conf=new SparkConf().setMaster("local[*]").setAppName("sparkStreamingDemo")
...

需要注意的就是上面的master需要设置成local[*]。如果设置成local或者local[1]会报错如下：

WARN BlockManager: Block input-0-1437581601600 replicated to only 0 peer(s) instead of 1 peers

报错的原因参考该链接
将主要原因粘贴如下：

Do not run Spark Streaming programs locally with master configured as local or local[1]. This allocates only one CPU for tasks and if a receiver is running on it, there is no resource left to process the received data. Use at least local[2] to have more cores.

2.netcat命令

nc -l 7777