Kafka 学习笔记

1.

 KafkaUtils.createDirectStream[Array[Byte], String, DefaultDecoder, StringDecoder, (Array[Byte], String)](
  ssc, conf.getAll.toMap, fromOffset, messageHandler)

Spark1.3版本后，kafkautil里面提供了两个创建dstream的方法，一个是老版本中有的createStream方法，还有一个是后面新加的createDirectStream方法

createDirectStream性能会更好一点，通过新方法创建出来的dstream的rdd partition和kafka的topic的partition是一一对应的，通过低阶API直接从kafka的topic消费消息，但是它不再往zookeeper中更新consumer offsets，使得基于zk的consumer offsets的监控工具都会失效.

* Create an input stream that directly pulls messages from Kafka Brokers
* without using any receiver. This stream can guarantee that each message
* from Kafka is included in transformations exactly once (see points below).
*
* Points to note:
*  - No receivers: This stream does not use any receiver. It directly queries Kafka
*  - Offsets: This does not use Zookeeper to store offsets. The consumed offsets are tracked
*    by the stream itself. For interoperability with Kafka monitoring tools that depend on
*    Zookeeper, you have to update Kafka/Zookeeper yourself from the streaming application.
*    You can access the offsets used in each batch from the generated RDDs (see
*    [[org.apache.spark.streaming.kafka.HasOffsetRanges]]).
*  - Failure Recovery: To recover from driver failures, you have to enable checkpointing
*    in the [[StreamingContext]]. The information on consumed offset can be
*    recovered from the checkpoint. See the programming guide for details (constraints, etc.).
*  - End-to-end semantics: This stream ensures that every records is effectively received and
*    transformed exactly once, but gives no guarantees on whether the transformed data are
*    outputted exactly once. For end-to-end exactly-once semantics, you have to either ensure
*    that the output operation is idempotent, or use transactions to output records atomically.
*    See the programming guide for more details.

2.

val config = new ProducerConfig(props)
val producer = new Producer[String, String](config)

val data = new KeyedMessage[String, String]("TesthotWords", msg)

msg为key-value形式,key用来对msg进行分区用的,为了散列存储消息,采集器那边key采用的是:topic|加一个随机数的形式,例如:rd_e_pal|20,split by |取0可以拿到对应的topic名字,这样union在一起的消息可以区分出来自哪一个topic

3.