主机与虚拟机的Kafka通信异常

最新推荐文章于 2023-08-01 18:23:30 发布
最新推荐文章于 2023-08-01 18:23:30 发布
阅读量472 收藏
点赞数
分类专栏： bug修复记录 文章标签： kafka spark big data
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_45973213/article/details/124903552
版权

报错日志：

22/05/21 21:16:21 INFO ReceiverSupervisorImpl: Called receiver onStop
22/05/21 21:16:21 INFO ReceiverSupervisorImpl: Deregistering receiver 0
22/05/21 21:16:21 ERROR ReceiverTracker: Deregistered receiver for stream 0: Error starting receiver 0 - org.I0Itec.zkclient.exception.ZkException: Unable to connect to hadoop000:2181
	at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:71)
	at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1218)
	at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:155)
	at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:129)
	at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:89)
	at kafka.utils.ZkUtils$.apply(ZkUtils.scala:71)
	at kafka.consumer.ZookeeperConsumerConnector.connectZk(ZookeeperConsumerConnector.scala:181)
	at kafka.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:129)
	at kafka.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:146)
	at kafka.consumer.Consumer$.create(ConsumerConnector.scala:109)
	at org.apache.spark.streaming.kafka.KafkaReceiver.onStart(KafkaInputDStream.scala:100)
	at org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:149)
	at org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:131)
	at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:607)
	at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:597)
	at org.apache.spark.SparkContext$$anonfun$34.apply(SparkContext.scala:2173)
	at org.apache.spark.SparkContext$$anonfun$34.apply(SparkContext.scala:2173)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:108)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException: hadoop000
	at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
	at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
	at java.net.InetAddress.getAllByName0(InetAddress.java:1277)
	at java.net.InetAddress.getAllByName(InetAddress.java:1193)
	at java.net.InetAddress.getAllByName(InetAddress.java:1127)
	at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:61)
	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445)
	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380)
	at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:69)
	... 22 more

22/05/21 21:16:21 INFO ReceiverSupervisorImpl: Stopped receiver 0
22/05/21 21:16:21 INFO BlockGenerator: Stopping BlockGenerator
22/05/21 21:16:21 INFO RecurringTimer: Stopped timer for BlockGenerator after time 1653138981600
22/05/21 21:16:21 INFO BlockGenerator: Waiting for block pushing thread to terminate
22/05/21 21:16:21 INFO BlockGenerator: Pushing out the last 0 blocks
22/05/21 21:16:21 INFO BlockGenerator: Stopped block pushing thread
22/05/21 21:16:21 INFO BlockGenerator: Stopped BlockGenerator
22/05/21 21:16:21 INFO ReceiverSupervisorImpl: Waiting for receiver to be stopped
22/05/21 21:16:21 ERROR ReceiverSupervisorImpl: Stopped receiver with error: org.I0Itec.zkclient.exception.ZkException: Unable to connect to hadoop000:2181
22/05/21 21:16:21 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
org.I0Itec.zkclient.exception.ZkException: Unable to connect to hadoop000:2181
	at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:71)
	at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1218)
	at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:155)
	at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:129)
	at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:89)
	at kafka.utils.ZkUtils$.apply(ZkUtils.scala:71)
	at kafka.consumer.ZookeeperConsumerConnector.connectZk(ZookeeperConsumerConnector.scala:181)
	at kafka.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:129)
	at kafka.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:146)
	at kafka.consumer.Consumer$.create(ConsumerConnector.scala:109)
	at org.apache.spark.streaming.kafka.KafkaReceiver.onStart(KafkaInputDStream.scala:100)
	at org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:149)
	at org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:131)
	at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:607)
	at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:597)
	at org.apache.spark.SparkContext$$anonfun$34.apply(SparkContext.scala:2173)
	at org.apache.spark.SparkContext$$anonfun$34.apply(SparkContext.scala:2173)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:108)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException: hadoop000
	at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
	at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
	at java.net.InetAddress.getAllByName0(InetAddress.java:1277)
	at java.net.InetAddress.getAllByName(InetAddress.java:1193)
	at java.net.InetAddress.getAllByName(InetAddress.java:1127)
	at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:61)
	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445)
	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380)
	at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:69)
	... 22 more
22/05/21 21:16:21 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, localhost, executor driver): org.I0Itec.zkclient.exception.ZkException: Unable to connect to hadoop000:2181
	at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:71)
	at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1218)
	at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:155)
	at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:129)
	at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:89)
	at kafka.utils.ZkUtils$.apply(ZkUtils.scala:71)
	at kafka.consumer.ZookeeperConsumerConnector.connectZk(ZookeeperConsumerConnector.scala:181)
	at kafka.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:129)
	at kafka.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:146)
	at kafka.consumer.Consumer$.create(ConsumerConnector.scala:109)
	at org.apache.spark.streaming.kafka.KafkaReceiver.onStart(KafkaInputDStream.scala:100)
	at org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:149)
	at org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:131)
	at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:607)
	at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:597)
	at org.apache.spark.SparkContext$$anonfun$34.apply(SparkContext.scala:2173)
	at org.apache.spark.SparkContext$$anonfun$34.apply(SparkContext.scala:2173)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:108)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException: hadoop000
	at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
	at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
	at java.net.InetAddress.getAllByName0(InetAddress.java:1277)
	at java.net.InetAddress.getAllByName(InetAddress.java:1193)
	at java.net.InetAddress.getAllByName(InetAddress.java:1127)
	at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:61)
	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445)
	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380)
	at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:69)
	... 22 more

总之就是主机中运行scala程序接收虚拟机kafka的消息的时候，主机找不到虚拟机
解决方案：

1在虚拟机的/etc/host里面配虚拟机名称和ip的映射
在这里插入图片描述

2 在主机中的C:\Windows\System32\drivers\etc中配置虚拟机的名称和虚拟机ip的映射
192.168.10.132 hadoop000

BugRecorder
关注
专栏目录
第10单元 Kafka集群部署1
08-08
- **host.name**：设置Kafka服务的主机名，应与实际IP一致。 - **listeners**：定义Kafka监听的端口，如`listeners=PLAINTEXT://192.168.73.200:9092`。 4. **启动Kafka集群** - **启动Zookeeper**：在每个节点...
kafka之主机客户端连接虚拟机上的kafka集群并进行生产者和消费者测试(五)
胖墩的IT博客
07-23 1747
想连接的话需要满足下面的条件 1、关闭虚拟机的防火墙，或者开放kafka和zookeeper的指定端口 centos7关闭防火墙 2、给虚拟机设定静态的IP，当然如果不嫌麻烦就可以不设 CentOS7用NAT模式设置静态IP 3、把kafka文件夹下的配置文件config/server.properties里面的listenters写上虚拟机的IP，不要默认的localhost 下面所有的操作...
03 当前机器的虚拟机[NAT网络]映射出 9200 到宿主机, 局域网的其他机器访问 宿主机+9200 访问不到 kafka 的服务
970655147的专栏
08-14 639
上下文是这样的 : 我开发机器上面安装了一个 虚拟机, 这个虚拟机网络基于 NAT 配置的我本地机器 局域网 ip 为 192.168.0.101, 虚拟机的 ip 为 192.168.220.133。
本机连不上虚拟机的kafka（最全解决方案）
文武傲群雄，彳亍定江山
11-13 3269
第一步:修改虚拟机相关配置，打开2181 9092的端口，并关闭防火墙 防火墙（root） systemctl stop firewalld.service #停止firewall systemctl disable firewalld.service #禁止firewall开机启动 systemctl status firewalld.service #查看firewall的状态 打开相关端口（...
SparkStreaming+Flume出现ERROR ReceiverTracker: Deregistered receiver for stream 0: Error starting rece...
weixin_34161064的博客
01-05 2123
文章发自http://www.cnblogs.com/hark0623/p/4204104.html ，转载请注明 我发现太多太多的坑要趟了… 向yarn提交sparkstreaming了，提交脚本如下，使用的是yarn-client spark-submit --driver-memory 1g --executor-memory 1g --executor-cores 1 --nu...
访问不到本机虚拟机上面的kafka服务
fuck487的博客
07-24 588
参考：https://blog.csdn.net/zyt_1272999061/article/details/79484702 # The address the socket server listens on. It will get the value returned from # java.net.InetAddress.getCanonicalHostName() if not configured. 显式设置kafka的具体地址和端口，如果没设置，默认使用java.net.InetAdd
Kafka集群搭建（3台机）
05-25
为了方便管理，将三台虚拟机的主机名分别设置为Kafka1、Kafka2和Kafka3。确保每台虚拟机的网络连接正常，并且能够互相通信。 接下来是JDK的安装，因为Kafka是用Java编写的，所以需要在所有虚拟机上安装Java开发工具...
CDH5.14安装文档(包含spark2和kafka的安装)
03-30
此外，还需要修改主机名、关闭NetworkManager、禁用防火墙和selinux，以确保集群内的通信畅通无阻。 安装过程大致分为以下几个步骤： 1. 在所有节点上安装JDK，并配置环境变量。 2. 将Cloudera Manager安装包上传到...
Java基于机器学习的电影推荐算法设计与实现项目源码+项目说明.zip
07-20
OS：CentOS 7（项目代码在宿主机上，其余环境都在虚拟机上跑的） JDK 1.8.0 Tomcat 5.5 Mongodb 3.4 Elasticsearch 5.6 Redis 2.9 Kafka 3.4 Spark 2.1 ## 项目配置的时候应该注意的几个问题 1. 所有涉及URL连接...
java连接kafka接收不到数据_关于Java客户端连接虚拟机中的Kafka时，无法发送、接收消息的问题...
weixin_35063366的博客
02-27 762
1.问题排查(１)首先通过在服务器上使用命令行来模拟生产、消费数据，发现生产、消费数据正常。测试如下：＃生产者bin/kafka-console-consumer.sh --bootstrap-server ***.***.***.***:9092 --topic test其中 ***.***.***代表的是 kafka broker的地址.再开一个窗口启动消费者：#消费者bin/kafka-co...
远程kafka通信实例，各种bug解决----虚拟机+本地电脑
weixin_30765319的博客
05-24 250
为了实现远程kafka通信，我可谓是呕心沥血。期间各种bug各种调，太煎熬了 (T.T) 介绍： 我用一台虚拟机作为远程消息的发送方，用本地电脑主机作为消息的接收方 虚拟机：安装java，kafka，zookeeper 主机：eclipse，注意我没有说在主机上也要安装kafka的 1、虚拟机部署 　　1）下载kafka_2.11-2.2.0我用的最新的（当前） 　　2）解压...
使用虚拟机连接本机上的kafka报错Can‘t resolve address: DESKTOP-HL13C2B:9092
fuck487的博客
07-22 1752
在虚拟机上运行java程序访问自己电脑上运行的kafka，发现访问不了kafka，错误日志提示 Can't resolve address: DESKTOP-HL13C2B:9092 DESKTOP-HL13C2B这个不就是我电脑的名字么 难道虚拟机会用我电脑的主机名作为域名访问？既然这样，只需要在虚拟机上面配置域名对应的ip映射即可 vim /etc/hosts 10.0.5.71 DESKTOP-HL13C2B ...
2021windows虚拟机搭建kafka集群详细步骤+可能出现的bug+重新部署(带图,超详细,亲测有效)
ManCxyster的博客
06-30 1590
第一步:安装zookeeper 这里使用的是zookeeper-3.6.3,直接去管网下载即可 第二步:配置zookeeper （1）下载解压完成后，来到conf文件夹下，有一个 zoo_sample.cfg 官方默认的配置文件。复制一份，重命名为 zoo.cfg （2）配置，打开zoo.cfg 修改配置信息 dataDir=../zkData dataLogDir=../zkLog clientPort=2181 修改完新增下列属性: admin.serverPort=8000 server.1=12
踩坑（5）整合kafka 报错 java.net.UnknownHostException: 不知道这样的主机
最新发布
SuperrWatermelon的博客
08-01 2636
整合kafka 报错 java.net.UnknownHostException: 不知道这样的主机
SparkStream中updateStateByKey报错：
XIAOMO__的博客
08-24 1401
错误1： ERROR StreamingContext: Error starting the context, marking it as stopped requirement failed: No output operations registered, so nothing to execute 意思是：StreamingContext错误：启动上下文时出错，将其标记为已停止的需求失败：没有注册输出操作，因此没有要执行的内容 报错原因是:检查点的时间至少是10s以上，一般设置时间是ba.
Windows环境下Kafka操作中启动的报错问题和解决方法
qq_41144930的博客
10-16 6427
一，启动Kafka报错： 报错的一些情况如下图所示： ERROR Error while loading log dir D:\Download\kafka_2.11-1.0\kafka_2.11-1.0\kafka_2.11-1.0.0\Downloadkafka_2.11-1.0kafka_2.11-1.0kafka_2.11-1.0.0logs (kafka.log.LogManager)...
宿主机telnet不通虚拟机kafka的问题解决
bo1988313的博客
01-22 3075
环境宿主机： win7 虚拟机： centos7 telnet 已安装 在虚拟机上部署好zookeeper、kafka后在宿主机上使用telnet命令。死活掉不通。浪费了好长时间才搞定。希望对大家有帮助。 1.防火墙问题 检查防火墙状态，已关闭。 而且zookeeper服务宿主机可以通过telnet命令调通。排除。 2.kafka服务自身问题 检查kafka服务是否正常启动。 已正常启动。 那就是kafka自身的问题了。 打开kafka的配置文件 结合网上查到资料觉着是liste.
解决远程kafka本地无法连接问题
热门推荐
weixin_43751248的博客
04-19 1万+
解决远程kafka本地无法连接问题
VMware虚拟机间通信实现——知识库模块
仅主机模式则限制了虚拟机仅能与物理主机或其他在同一网络上的虚拟机通信。虚拟机间的通信还需要正确配置IP地址、子网掩码和网关，确保它们在同一网络段内或者通过路由可达。 此外，论文还提到了在线客服聊天后台...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值