本地 | 远程 | |
---|---|---|
Spark | 1.5.2 | 2.4.3(不过现在还没提交到集群运行) |
Kafka | 0.8.2.1 | 0.8. |
Spark_Streaming:
// 创建Receiver流的一个要点
// “zookeeper.connect” -> “spark1:2181,spark2:2181,spark3:2181,spark4:2181”,
val stream:ReceiverInputDStream[(String, String)] = KafkaUtils.createStream[String,String,
StringDecoder,StringDecoder](ssc,kafkaParams,
topicMap,StorageLevel.MEMORY_AND_DISK_SER)
附上0.8的消费者版本:
class consuemr{
var consumer:kafka.consumer.ConsumerConnector=_
def test(): Unit ={
import kafka.utils.VerifiableProperties
val props = new Properties()
// zookeeper 配置
props.put("zookeeper.connect", "spark1:2181,spark2:2181")
// group 代表一个消费组
props.put("group.id", "jd-group")
// zk连接超时
props.put("zookeeper.session.timeout.ms", "4000")
props.put("zookeeper.sync.time.ms", "200")
props.put("auto.commit.interval.ms", "1000")
props.put("auto.offset.reset", "smallest")
// 序列化类
props.put("serializer.class", "kafka.serializer.StringEncoder")
val config = new ConsumerConfig(props)
consumer = kafka.consumer.Consumer.create(config)
val topicCountMap = new util.HashMap[String,Int]()
topicCountMap.put("",1)
val keyDecoder = new StringDecoder(new VerifiableProperties)
val valueDecoder = new StringDecoder(new VerifiableProperties)
val consumerMap = consumer.createMessageStreams(topicCountMap,keyDecoder,valueDecoder)
val stream:KafkaStream[String,String] = consumerMap("yy-topic").head
val it = stream.iterator()
while(it.hasNext()){
println(it.next().message())
}
}
}
============ 可能问题:
缺少:LKafka/…/Connector:
Spark-Streaming-Kafka 版本不对
NotClassFound:…/Logging
Spark 1.5 以上缺少Logging模块