kafka 查看待消费数据_Spark Streaming 消费Kafka实时数据

2019年3月份刚刚做的第一个项目，就是基于Kafka的数据流异常检测，当时用C语言写的，编译后的可执行程序大约20M，跑的很6~

Kafka是日志聚合器和发布订阅系统，Kafka表面看起来像是一个消息传递系统，后台是一个日志聚合器。它使用分区预写提交日志来维护所有数据，数据在一个主题下，以语义的方式分组，这只是一个队列的逻辑标签，它是生产者编写的队列，也是消费者读取的队列~

一眨眼快2年过去了，已经退出一线，专心做PPT架构师了，也经历过几个不错的AI项目，将人工智能和安全业务优雅的结合，前前后后翻新了两次AI-安全大数据架构~

本篇文章将会使用Kafka，计算每对站点之间的行程次数，得到共享单车穿梭的热门站点~

• 配置Kafka，并启动~

bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties

• 服务端生产数据~

• 客户端消费数据~
• 可以看到很多都是解锁，又还车~
• 这是因为2013年共享单车在美国刚刚试运行~

核心逻辑分析：

      .map(rec => rec.split(","))
      .map(rec => ((rec(3), rec(7)), 1))
      .reduceByKey(_ + _)
      .repartition(1)
      .map(rec => (rec._2, rec._1))
      .transform(rdd => rdd.sortByKey(ascending = false))

• rec(3)是起点
• rec(7)是终点
• 针对起点和终点，统计出行总次数
• 按照总次数，倒序输出

服务端生产代码：

import java.util.Properties;

import kafka.javaapi.producer.Producer;
import kafka.producer.KeyedMessage;
import kafka.producer.ProducerConfig;

public class KafkaDriver extends AbstractDriver {

	private final String topic;
	private Producer<String, String> producer;

	public KafkaDriver(String path, String topic, Properties props) {
		super(path);
		this.topic = topic;
		ProducerConfig config = new ProducerConfig(props);
		producer = new Producer<String, String>(config);
	}

	@Override
	public void init() throws Exception {
	}

	@Override
	public void close() throws Exception {
		producer.close();
	}

	@Override
	public void sendRecord(String record) throws Exception {
		producer.send(new KeyedMessage<String, String>(topic, record));
	}

	public static void main(String[] args) throws Exception {

		String path = "C:studypro_spark_streaming5data";
		String brokerUrl = "192.168.160.137:9092";
		String topic = "ailx10";

		Properties props = new Properties();
		props.put("metadata.broker.list", brokerUrl);
		props.put("serializer.class", "kafka.serializer.StringEncoder");

		KafkaDriver driver = new KafkaDriver(path, topic, props);
		try {
			driver.execute();
		} finally {
			driver.close();
		}
	}

}

客户端消费代码：

import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.rdd.RDD.rddToOrderedRDDFunctions
import org.apache.spark.streaming.Seconds
import org.apache.spark.streaming.StreamingContext
import org.apache.spark.streaming.dstream.DStream.toPairDStreamFunctions
import kafka.serializer.StringDecoder
import org.apache.spark.streaming.kafka.KafkaUtils

object StationJourneyCountDirectApp {

  def main(args: Array[String]) {

    val checkpointDir = "C:studysparkkafka5ailx10"
    val topic = "ailx10"
    val outputPath = "C:studysparkkafka5output"

    val conf = new SparkConf()
      .setAppName("ailx10")
      .setJars(SparkContext.jarOfClass(this.getClass).toSeq)
      .setMaster("local")

    val ssc = new StreamingContext(conf, Seconds(10))
    ssc.checkpoint(checkpointDir)

    val topics = Set(topic)
    val params = Map[String, String](
      "zookeeper.connect" -> "192.168.160.137:2181",
      "group.id" -> "ailx10",
      "bootstrap.servers" -> "192.168.160.137:9092")
    val line = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, params, topics).map(_._2)
      .map(rec => rec.split(","))
      .map(rec => ((rec(3), rec(7)), 1))
      .reduceByKey(_ + _)
      .repartition(1)
      .map(rec => (rec._2, rec._1))
      .transform(rdd => rdd.sortByKey(ascending = false))

      line.print()

      line.saveAsTextFiles(outputPath)

    ssc.start()
    ssc.awaitTermination()
  }

}

附录ubuntu国内软件源，你已经看到最后辣~

• Kali rolling 中科大源

#中科大
deb http://mirrors.ustc.edu.cn/kali kali-rolling main non-free contrib
deb-src http://mirrors.ustc.edu.cn/kali kali-rolling main non-free contrib

• Kali rolling 清华大学源

#清华大学
deb http://mirrors.tuna.tsinghua.edu.cn/kali kali-rolling main contrib non-free
deb-src https://mirrors.tuna.tsinghua.edu.cn/kali kali-rolling main contrib non-free

本篇完，谢谢大家~