kafka实操笔记

kafka测试

[root@master ~]# cd /usr/local/soft/kafka_2.11-0.10.1.0
[root@master kafka_2.11-0.10.1.0]# bin/zookeeper-server-start.sh config/zookeeper.properties
# ---一堆[info]
ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.ZooKeeperServerMain)
java.net.BindException: 地址已在使用

########### 查看端口以及JPS ###########
[root@master kafka_2.11-0.10.1.0]# netstat -nlp|grep 2181
tcp        0      0 :::2181                     :::*                        LISTEN      75666/java

[root@master kafka_2.11-0.10.1.0]# jps
75731 HMaster
75666 HQuorumPeer
75862 HRegionServer
2903 ResourceManager
93592 Jps
60745 Worker
2586 NameNode
2762 SecondaryNameNode
60653 Master

原因分析是我之前启动的HBase也启动了Zookeeper,所以现在暂时可以先关掉HBase

#### 其实我现在的时间是2020年1月31日11:03:03 ## 集群一直挂起,所以时间不对 ####
[root@master kafka_2.11-0.10.1.0]# stop-hbase.sh 
stopping hbase................
localhost: stopping zookeeper.
[root@master kafka_2.11-0.10.1.0]# jps
2903 ResourceManager
60745 Worker
2586 NameNode
2762 SecondaryNameNode
93996 Jps
60653 Master
[root@master kafka_2.11-0.10.1.0]# bin/zookeeper-server-start.sh config/zookeeper.properties
[2019-12-05 01:09:43,959] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2019-12-05 01:09:43,965] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.DatadirCleanupManager)
[2019-12-05 01:09:43,965] INFO autopurge.purgeInterval set to 0 (org.apache.zookeeper.server.DatadirCleanupManager)
[2019-12-05 01:09:43,965] INFO Purge task is not scheduled. (org.apache.zookeeper.server.DatadirCleanupManager)
[2019-12-05 01:09:43,965] WARN Either no config or no quorum defined in config, running  in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
[2019-12-05 01:09:44,013] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2019-12-05 01:09:44,013] INFO Starting server (org.apache.zookeeper.server.ZooKeeperServerMain)
[2019-12-05 01:09:44,023] INFO Server environment:zookeeper.version=3.4.8--1, built on 02/06/2016 03:18 GMT (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,023] INFO Server environment:host.name=master (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,023] INFO Server environment:java.version=1.8.0_171 (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,023] INFO Server environment:java.vendor=Oracle Corporation (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,023] INFO Server environment:java.home=/usr/local/soft/jdk1.8.0_171/jre (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,023] INFO Server environment:java.class.path=:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/aopalliance-repackaged-2.4.0-b34.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/argparse4j-0.5.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/connect-api-0.10.1.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/connect-file-0.10.1.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/connect-json-0.10.1.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/connect-runtime-0.10.1.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/guava-18.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/hk2-api-2.4.0-b34.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/hk2-locator-2.4.0-b34.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/hk2-utils-2.4.0-b34.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jackson-annotations-2.6.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jackson-core-2.6.3.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jackson-databind-2.6.3.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jackson-jaxrs-base-2.6.3.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jackson-jaxrs-json-provider-2.6.3.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jackson-module-jaxb-annotations-2.6.3.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/javassist-3.18.2-GA.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/javax.annotation-api-1.2.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/javax.inject-1.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/javax.inject-2.4.0-b34.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/javax.servlet-api-3.1.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/javax.ws.rs-api-2.0.1.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jersey-client-2.22.2.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jersey-common-2.22.2.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jersey-container-servlet-2.22.2.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jersey-container-servlet-core-2.22.2.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jersey-guava-2.22.2.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jersey-media-jaxb-2.22.2.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jersey-server-2.22.2.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jetty-continuation-9.2.15.v20160210.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jetty-http-9.2.15.v20160210.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jetty-io-9.2.15.v20160210.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jetty-security-9.2.15.v20160210.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jetty-server-9.2.15.v20160210.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jetty-servlet-9.2.15.v20160210.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jetty-servlets-9.2.15.v20160210.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jetty-util-9.2.15.v20160210.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jopt-simple-4.9.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/kafka_2.11-0.10.1.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/kafka_2.11-0.10.1.0-sources.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/kafka_2.11-0.10.1.0-test-sources.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/kafka-clients-0.10.1.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/kafka-log4j-appender-0.10.1.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/kafka-streams-0.10.1.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/kafka-streams-examples-0.10.1.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/kafka-tools-0.10.1.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/log4j-1.2.17.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/lz4-1.3.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/metrics-core-2.2.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/osgi-resource-locator-1.0.1.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/reflections-0.9.10.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/rocksdbjni-4.9.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/scala-library-2.11.8.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/scala-parser-combinators_2.11-1.0.4.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/slf4j-api-1.7.21.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/slf4j-log4j12-1.7.21.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/snappy-java-1.1.2.6.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/validation-api-1.1.0.Final.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/zkclient-0.9.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/zookeeper-3.4.8.jar (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,024] INFO Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,024] INFO Server environment:java.io.tmpdir=/tmp (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,024] INFO Server environment:java.compiler=<NA> (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,024] INFO Server environment:os.name=Linux (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,024] INFO Server environment:os.arch=amd64 (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,024] INFO Server environment:os.version=2.6.32-431.el6.x86_64 (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,024] INFO Server environment:user.name=root (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,024] INFO Server environment:user.home=/root (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,025] INFO Server environment:user.dir=/usr/local/soft/kafka_2.11-0.10.1.0 (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,035] INFO tickTime set to 3000 (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,035] INFO minSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,035] INFO maxSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,050] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)


之后直接参考子雨老师实验室的教程

Spark连接kafka

[root@master spark-2.1.0-bin-without-hadoop]# spark-shell --jars /usr/lft/spark-2.1.0-bin-without-hadoop/jars/*:/usr/local/soft/spark-2.1.0-bin-without-hadoop/jars/kafka/*
Warning: Local jar /usr/local/soft/spark-2.1.0-bin-without-hadoop/jars/*:/usr/local/soft/spark-2.1.0-bin-without-hadoop/jars/kafka/* does not exist, skipping.
19/12/05 01:44:09 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://192.168.0.11:4040
Spark context available as 'sc' (master = local[*], app id = local-1575528251217).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.1.0
      /_/
         
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_171)
Type in expressions to have them evaluated.
Type :help for more information.

scala> import org.apache.spark.streaming.kafka._
import org.apache.spark.streaming.kafka._

scala> 

前面告诉我说

Warning: Local jar /usr/local/soft/spark-2.1.0-bin-without-hadoop/jars/*:/usr/local/soft/spark-2.1.0-bin-without-hadoop/jars/kafka/* does not exist, skipping.

后面import又成功了,我惊了

scala> import org.apache.spark.streaming.kafka._
import org.apache.spark.streaming.kafka._

我暂时没有配置给每个节点都kafka

Spark编程使用kafka

前提是一定要开着kafka(见上文),然后就可以参考子雨老师实验室的文章了

如果你关掉了kafka和zookeeper,可以这样开启

### 终端1 ###
# 进入kafka所在的目录
cd /usr/local/soft/kafka
bin/zookeeper-server-start.sh config/zookeeper.properties


### 终端2 ###
# 进入kafka所在的目录
cd /usr/local/soft/kafka
bin/kafka-server-start.sh config/server.properties

如果启动kafka报错

kafka.common.KafkaException: Socket server failed to bind to 0.0.0.0:9092: 地址已在使用
那么一般是kafka未正常关闭(发现ctrl+z退出kafka很可能不会彻底退出,所以一般还有自己jps查看进程然后再kill掉)
可以这样操作

lsof -n -i :9092 | grep LISTEN
# java    94330 root   99u  IPv6  844902      0t0  TCP *:XmlIpcRegSvc (LISTEN)
# 其实之前jps也发现 94330 运行着 kafka... 所以也可以直接jps找到对应的线程号,然后关闭
# 之后可以选择
kill -9 94330

### 再次启动
# 进入kafka所在的目录
cd /usr/local/kafka
bin/kafka-server-start.sh config/server.properties

kafka没能中止,会导致Producer产生消息异常,100s产生一次消息,之后修好kafka才恢复正常1秒产生3次消息别问我怎么知道的

发布了33 篇原创文章 · 获赞 4 · 访问量 7285
展开阅读全文

没有更多推荐了,返回首页

©️2019 CSDN 皮肤主题: 技术工厂 设计师: CSDN官方博客

分享到微信朋友圈

×

扫一扫,手机浏览