CM添加kafka服务

最新推荐文章于 2023-06-16 14:24:38 发布

ggzone

最新推荐文章于 2023-06-16 14:24:38 发布

阅读量3.3k

点赞数

分类专栏： CM kafka 文章标签： CM kafka

本文链接：https://blog.csdn.net/ggz631047367/article/details/50754408

版权

CM 同时被 2 个专栏收录

4 篇文章 0 订阅

订阅专栏

kafka

2 篇文章 0 订阅

订阅专栏

下载所需的包：
在http://archive.cloudera.com/kafka/parcels/latest/ 选择合适parcel下载：
ubuntu14.04的cdh5.5.1对应包
http://archive.cloudera.com/kafka/parcels/latest/KAFKA-2.0.0-1.kafka2.0.0.p0.12-trusty.parcel
http://archive.cloudera.com/kafka/parcels/latest/KAFKA-2.0.0-1.kafka2.0.0.p0.12-trusty.parcel.sha1
http://archive.cloudera.com/kafka/parcels/latest/manifest.json

在http://archive.cloudera.com/csds/kafka/ 下载
http://archive.cloudera.com/csds/kafka/KAFKA-1.2.0.jar

在cm server主机：

mv KAFKA-1.2.0.jar  /opt/cloudera/csd/KAFKA-1.2.0.jar
mv KAFKA-2.0.0-1.kafka2.0.0.p0.12-trusty.parce* /opt/cloudera/parcel-repo/
mv manifest.json /opt/cloudera/parcel-repo/

进入网页：http://ha0:7180
菜单栏–主机–parcel–检查新的parcel–分配–激活

如果启动时出现内存溢出错误
在kafka配置中修改Java Heap Size of Broker 512即可

Fatal error during KafkaServerStartable startup. Prepare to shutdown
java.lang.OutOfMemoryError: Java heap space
    at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
    at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
    at kafka.log.SkimpyOffsetMap.<init>(OffsetMap.scala:43)
    at kafka.log.LogCleaner$CleanerThread.<init>(LogCleaner.scala:186)
    at kafka.log.LogCleaner$$anonfun$1.apply(LogCleaner.scala:83)
	at kafka.log.LogCleaner$$anonfun$1.apply(LogCleaner.scala:83)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
    at scala.collection.immutable.Range.foreach(Range.scala:166)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
    at scala.collection.AbstractTraversable.map(Traversable.scala:104)
    at kafka.log.LogCleaner.<init>(LogCleaner.scala:83)
    at kafka.log.LogManager.<init>(LogManager.scala:64)
    at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:601)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:180)
    at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
    at kafka.Kafka$.main(Kafka.scala:67)
    at com.cloudera.kafka.wrap.Kafka$.main(Kafka.scala:76)
    at com.cloudera.kafka.wrap.Kafka.main(Kafka.scala)

测试：

kafka-topics --create --zookeeper ha0 --topic ggz --partitions 1 --replication-factor 1
kafka-console-producer --broker-list ha1:9092 --topic ggz
kafka-console-consumer  --zookeeper ha0 --topic ggz --from-beginning