2021-07-06

标题针对利用SparkStreaming实时消费Kafka，采集不到数据卡住不动 不打印实时时间戳报错问题解说

简单样例代码：定义工具类（存放Kafka的配置信息） ，定义一个执行类（跑SparkStreaming程序）
工具类：MyKafkaUtils

import org.apache.kafka.common.serialization.StringDeserializer
import org.apache.spark.streaming.StreamingContext
import org.apache.spark.streaming.kafka010.ConsumerStrategies.Subscribe
import org.apache.spark.streaming.kafka010.KafkaUtils
import org.apache.spark.streaming.kafka010.LocationStrategies.PreferConsistent
object MyKafkaUtils {
val kafkaParams = Map[String, Object](
“bootstrap.servers” -> “Kafka集群地址”,
“key.deserializer” -> classOf[StringDeserializer],
“value.deserializer” -> classOf[StringDeserializer],
/* “value.deserializer” -> “org.apache.kafka.clients.consumer.ConsumerRecord”,
“key.deserializer” -> “org.apache.kafka.clients.consumer.ConsumerRecord”,/
“group.id” -> “picc_yaoshunqian2”,
“auto.offset.reset” -> “latest”,
“enable.auto.commit” -> (true: java.lang.Boolean)
)
def getkafkaStream(ssc:StreamingContext,topics:String)={
KafkaUtils.createDirectStream[String, String](
ssc,
PreferConsistent,
Subscribe[String, String](topics.toIterable, kafkaParams)
)
// val count = 0

}
}

执行类：
import com.pactera.picc.streaming.project.util.{MyKafkaUtils, RedisUtil}
import org.apache.kafka.clients.consumer.ConsumerRecord
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.streaming.dstream.{DStream, InputDStream}
import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.{SparkConf, SparkContext}
object App {

/* def createNewConnection() = {???

}/
//var count = 0
def main(args: Array[String]): Unit = {
//var count = 0
val conf: SparkConf = new SparkConf().setMaster("local[]").setAppName(“App”)
val ssc = new StreamingContext(conf, Seconds(5))
ssc.checkpoint("./ck1001")
val sourcestream=MyKafkaUtils.getkafkaStream(ssc,“topic名称”)

val sourcevalue =sourcestream.map(record => record.value())
sourcevalue.print()
ssc.start()
ssc.awaitTermination()
}
}
就这么两段儿代码 pom文件按照各自的scala版本，spark版本配好按理说这样去消费是没有问题的，但是还有最最重要容易忽略的一点就是配置本地的host文件 关联消费Kafka地址的目的主机名 不然就是卡住 打印不出时间戳 你还看不懂报错信息。
就是配置这个玩意儿
hosts:
hadoop100 192.168.220.100
hadoop101 192.168.220.101
hadoop102 192.168.220.102
hadoop103 192.168.220.103
hadoop104 192.168.220.104
hadoop105 192.168.220.105
hadoop106 192.168.220.106
hadoop107 192.168.220.107
hadoop108 192.168.220.108