sparkStream

依赖包

注意要根据spark的版本来选择依赖包

<dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_2.11</artifactId>
      <version>2.4.5</version>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming_2.11</artifactId>
      <version>2.4.5</version>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
      <version>2.4.5</version>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-sql_2.11</artifactId>
      <version>2.4.5</version>
    </dependency>
    <dependency>
      <groupId>com.fasterxml.jackson.core</groupId>
      <artifactId>jackson-databind</artifactId>
      <version>2.6.6</version>
    </dependency>

WordCount

Scala端：

val conf = new SparkConf().setMaster("local[2]").setAppName("demo2")
   //采集周期，指定3秒为每次采集的时间间隔
    val ssc = new StreamingContext(conf,Seconds(3))
    // 指定采集的方法
    val socket = ssc.socketTextStream("hadoop01",7777)
       //将采集来的信息进行处理，统计数据（wordcount)
    val res = socket.flatMap(_.split("\\s+")).map((_,1)).reduceByKey(_+_)
    //打印
    res.print()
    ssc.start()
    ssc.awaitTermination()

Java端：

SparkConf conf = new SparkConf().setMaster("local[*]").setAppName("demo2");
        JavaStreamingContext jsc = new JavaStreamingContext(conf, Durations.seconds(3));
        JavaReceiverInputDStream<String> socket = jsc.socketTextStream("hadoop01", 7777);
        JavaDStream<String> flatMap = socket.flatMap(new FlatMapFunction<String, String>() {
            @Override
            public Iterator<String> call(String s) throws Exception {
                String[] split = s.split("\\s+");
                return Arrays.asList(split).iterator();
            }
        });
        JavaPairDStream<String, Integer> pair = flatMap.mapToPair(new PairFunction<String, String, Integer>() {
            @Override
            public Tuple2<String, Integer> call(String s) throws Exception {
                return new Tuple2<>(s, 1);
            }
        });
        JavaPairDStream<String, Integer> res = pair.reduceByKey(new Function2<Integer, Integer, Integer>() {
            @Override
            public Integer call(Integer integer, Integer integer2) throws Exception {
                return integer + integer2;
            }
        });
        res.print();
        jsc.start();
        jsc.awaitTermination();

虚拟机上连接测试：

nc -lk 7777