k8s中dubbo注册IP和消费的问题

最新推荐文章于 2023-12-01 14:09:48 发布

warrah

最新推荐文章于 2023-12-01 14:09:48 发布

阅读量5.5k

点赞数

分类专栏：岁月云——Web系统最佳实践岁月云——运维文章标签： dubbo

本文链接：https://blog.csdn.net/warrah/article/details/106923454

版权

岁月云——Web系统最佳实践同时被 2 个专栏收录

117 篇文章 0 订阅

订阅专栏

岁月云——运维

36 篇文章 0 订阅

订阅专栏

在系统迁移过程要考虑到平稳过度，先将部分应用迁移到k8s上，但是发现并没有那么简单

从dubbo-admin中可以看到注册服务的ip是容器组ip，如果所有应用都在k8s中自然是没有问题，但如果k8s与虚拟机中的服务相互调用，就存在问题了，因为找不到ip。

参考Docker 容器内运行 Dubbo 服务，启动容器,但是dubbo中还是容器的IP，并不是主机ip

docker run --name ab99755feae7 -d -p 9010:9010 -p 21716:21716 -e Dubbo_IP_TO_REGISTRY=10.101.7.66  ab99755feae7
# 再查看容器的环境变量，
[root@worker3 ~]# docker exec 1f559cde8604 env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=1f559cde8604
DUBBO_IP_TO_REGISTRY=10.101.7.66
LANG=C.UTF-8
JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
JAVA_VERSION=8u111
JAVA_DEBIAN_VERSION=8u111-b14-2~bpo8+1
CA_CERTIFICATES_JAVA_VERSION=20140324
HOME=/root

接着翻阅Spring cloud dubbo docker部署IP注册问题，但我查看dubbo-2.8.4 的源码，没看到读取环境变量的地方，不知道他说的源码是哪里的。查看MavenRepository，最新的版本也才2.6.8，估计是当初使用当当的dubbox导致的，应为我的dubbo是2016年就可以使使用了。
通过docker cp f8f041b98c79:/eztserver.jar /appdata，下载查看后发现里面确实没有环境变量，而官方2.6.7版本确实有这个。于是想到了替换版本。
但并不是直接替换就可以了会出问题，Unable to locate Spring NamespaceHandler for XML schema namespace排错记录

spring.schemas的内容如下：

http\://dubbo.apache.org/schema/dubbo/dubbo.xsd=META-INF/dubbo.xsd
http\://code.alibabatech.com/schema/dubbo/dubbo.xsd=META-INF/compat/dubbo.xsd

启动服务，提示下面的错误，

\target\classes\dubbo\dubbo.xml] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 9; columnNumber: 42; cvc-complex-type.2.4.c: 通配符的匹配很全面, 但无法找到元素 'dubbo:application' 的声明。
	at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:398)
	at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:335)

检查这些地方，都是正确的。为啥还提示这个错误呢？

如果放弃dubbo.xml的方式，采取springboot的方式，再调用docker run --name b448720570ed -d -p 9010:9010 -p 21716:21716 -e DUBBO_IP_TO_REGISTRY=10.101.7.65 b448720570ed,这个时候ip就是主机ip了

<properties>
		<dubbo.version>2.7.5</dubbo.version>
		<zookeeper.version>3.4.14</zookeeper.version>
	</properties>
	<!-- dubbo依赖 -->
		<dependency>
			<groupId>org.apache.dubbo</groupId>
			<artifactId>dubbo</artifactId>
			<version>${dubbo.version}</version>
			<exclusions>
				<exclusion>
					<groupId>org.springframework</groupId>
					<artifactId>spring</artifactId>
				</exclusion>
				<exclusion>
					<groupId>javax.servlet</groupId>
					<artifactId>servlet-api</artifactId>
				</exclusion>
				<exclusion>
					<groupId>log4j</groupId>
					<artifactId>log4j</artifactId>
				</exclusion>
			</exclusions>
		</dependency>

		<!-- zookeeper依赖 -->
		<dependency>
			<groupId>org.apache.zookeeper</groupId>
			<artifactId>zookeeper</artifactId>
			<version>${zookeeper.version}</version>
			<exclusions>
				<exclusion>
					<artifactId>slf4j-log4j12</artifactId>
					<groupId>org.slf4j</groupId>
				</exclusion>
			</exclusions>
		</dependency>
		<dependency>
			<groupId>com.github.sgroschupf</groupId>
			<artifactId>zkclient</artifactId>
			<version>0.1</version>
		</dependency>
		<dependency>
			<groupId>org.apache.curator</groupId>
			<artifactId>curator-recipes</artifactId>
			<version>4.2.0</version>
		</dependency>
		<dependency>
			<groupId>org.apache.curator</groupId>
			<artifactId>curator-framework</artifactId>
			<version>4.2.0</version>
		</dependency>

在springboot的启动类设置DubboComponentScan和EnableDubboConfig

import lombok.extern.slf4j.Slf4j;
import org.apache.dubbo.config.spring.context.annotation.DubboComponentScan;
import org.apache.dubbo.config.spring.context.annotation.EnableDubboConfig;
import org.mybatis.spring.annotation.MapperScan;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
import org.springframework.cloud.netflix.eureka.EnableEurekaClient;
import org.springframework.cloud.openfeign.EnableFeignClients;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.scheduling.annotation.EnableScheduling;

@Slf4j
@SpringBootApplication
@ComponentScan(basePackages = { "com.test" })
@MapperScan("com.bwjf.**.dao")
@EnableScheduling
@EnableEurekaClient
@EnableFeignClients(basePackages = {"com.test"})
@EnableDubboConfig
@DubboComponentScan("com.test.bzt.dubbo.service.impl")
@EnableDiscoveryClient
public class Application {
	
	private static Logger log = LoggerFactory.getLogger(Application.class);

    public static void main(String[] args) {
        try {
            System.setProperty("es.set.netty.runtime.available.processors", "false");
            SpringApplication.run(Application.class, args);
        } catch (Exception e) {
            log.info(e.getMessage(), e);
        } finally {
            log.info("server start finish");
        }
    }
}

而在yaml中设置dubbo

#Dubbbo
domain: bztserver  
dubbo:
  owner: zhengbangshun
  organization: bwjf
  registry:
    address: zookeeper://192.168.5.128:2181?backup=192.168.5.128:2182,192.168.5.128:2183
  protocol:
    name: dubbo
    port: 21716
    accesslog: dubbo-access.log
  service:
    validation: true
    version: 1.0.0
    timeout: 10000
    retries: 2
    loadbalance: random
    actives: 150
  reference:
    check: false
    version: 1.0.0

在k8s中并不能直接应用，因为你不知道他会将程序不出到那台机器上，故可参考Dubbo与Kubernetes集成,通过status.hostIP来制定主机ip

---
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: '3'
    k8s.eip.work/ingress: 'false'
    k8s.eip.work/service: none
    k8s.eip.work/workload: eztserver
    k8s.kuboard.cn/ingress: 'false'
    k8s.kuboard.cn/service: none
    k8s.kuboard.cn/workload: eztserver
  creationTimestamp: '2020-06-28T05:01:08Z'
  generation: 3
  labels:
    eztserver: eztserver
    k8s.eip.work/layer: ''
    k8s.eip.work/name: eztserver
  name: eztserver
  namespace: java
  resourceVersion: '2785134'
  selfLink: /apis/apps/v1/namespaces/java/deployments/eztserver
  uid: 1470af0b-0b71-4faf-bdbb-4a3534c1795a
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      eztserver: eztserver
      k8s.eip.work/layer: ''
      k8s.eip.work/name: eztserver
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        eztserver: eztserver
        k8s.eip.work/layer: ''
        k8s.eip.work/name: eztserver
    spec:
      containers:
        - env:
            - name: DUBBO_IP_TO_REGISTRY
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.hostIP
          image: '192.168.5.101/java/eztserver:1.0.0'
          imagePullPolicy: Always
          lifecycle: {}
          name: eztserver
          ports:
            - containerPort: 9010
              hostPort: 9010
              protocol: TCP
            - containerPort: 21716
              hostPort: 21716
              protocol: TCP
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
      dnsConfig: {}
      dnsPolicy: ClusterFirst
      imagePullSecrets:
        - name: harbor192.168.5.101
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        seLinuxOptions: {}
      terminationGracePeriodSeconds: 30

容器的dubbo端口也无需要开放出来

注册问题算是解决了，新的问题来，k8s容器内部的应用反而无法调用了

但是在dubbo-admin中看到消费者是主机ip，而不是容器ip，所以消费失败

接着下载dubbo-demo,做简单的配置

<dependency>
            <groupId>javax.validation</groupId>
            <artifactId>validation-api</artifactId>
            <version>1.1.0.Final</version>
        </dependency>
        <dependency>
            <groupId>org.hibernate</groupId>
            <artifactId>hibernate-validator</artifactId>
            <version>5.1.0.Final</version>
        </dependency>

否则可能会有下面的错误,因为接口做了校验处理

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'defaultValidator' defined in class path resource [org/springframework/boot/autoconfigure/validation/ValidationAutoConfiguration.class]: Invocation of init method failed; nested exception is java.lang.AbstractMethodError: org.hibernate.validator.engine.ConfigurationImpl.getDefaultParameterNameProvider()Ljavax/validation/ParameterNameProvider;
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1694) ~[spring-beans-5.0.10.RELEASE.jar:5.0.10.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:573) ~[spring-beans-5.0.10.RELEASE.jar:5.0.10.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:495) ~[spring-beans-5.0.10.RELEASE.jar:5.0.10.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317) ~[spring-beans-5.0.10.RELEASE.jar:5.0.10.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory$$Lambda$118/631673932.getObject(Unknown Source)

经过测试，在本地是可以调用到docker中发布的dubbo服务，但是docker内部的应用却不能访问，那么就可以断定是消费者注册ip的问题

跟踪UrlUtils中可以看到dubbo会将consumer和provider进行比对

继续跟踪，参照spring工程dubbo docker迁移做改造之后，在springboot工程中可以获取到接口返回，但是springmvc却找不到提供者
入下图，springboot工程和springmvc就可以成为对应的消费者，订阅了这个服务

springmvc工程就提示下面的异常

com.alibaba.dubbo.rpc.RpcException: No provider available in [invoker :interface com.bwjf.bdk.base.service.BdkWxService -> zookeeper://192.168.5.157:2181/com.alibaba.dubbo.registry.RegistryService?actives=150&anyhost=true&application=BWN&bean.name=com.bwjf.bdk.base.service.BdkWxService&check=false&dubbo=2.0.2&generic=false&interface=com.bwjf.bdk.base.service.BdkWxService&loadbalance=random&methods=getCardInfo,getStoreCard,getCoupon,getStoreMemberInfo,getMemberCouponInfo,getMemberCardList,getMemberSerialOrderInfo,getMemberSerialOrderList,bindMember,getMemberInfo,updateMemberInfo,getStoreInfo,getStoreCoupon,getCard,getCouponInfo,getMemberCoupon,registerMember,saveMemberTitle,getMemberStoreList,memberTitleDetail&organization=bwjf&owner=liuqiao&pid=24632&register.ip=10.101.5.102&remote.timestamp=1594188092802&retries=2&revision=1.0.0&side=consumer&timeout=60000&timestamp=1594189017615&version=1.0.0,directory: com.alibaba.dubbo.registry.integration.RegistryDirectory@3a917017, invoker :interface com.bwjf.bdk.base.service.BdkWxService -> zookeeper://192.168.5.157:2181/com.alibaba.dubbo.registry.RegistryService?actives=150&anyhost=true&application=BWN&bean.name=com.bwjf.bdk.base.service.BdkWxService&check=false&dubbo=2.0.2&generic=false&interface=com.bwjf.bdk.base.service.BdkWxService&loadbalance=random&methods=getCardInfo,getStoreCard,getCoupon,getStoreMemberInfo,getMemberCouponInfo,getMemberCardList,getMemberSerialOrderInfo,getMemberSerialOrderList,bindMember,getMemberInfo,updateMemberInfo,getStoreInfo,getStoreCoupon,getCard,getCouponInfo,getMemberCoupon,registerMember,saveMemberTitle,getMemberStoreList,memberTitleDetail&organization=bwjf&owner=liuqiao&pid=24632&register.ip=10.101.5.102&remote.timestamp=1594188092802&retries=2&revision=1.0.0&side=consumer&timeout=60000&timestamp=1594189017615&version=1.0.0,directory: com.alibaba.dubbo.registry.integration.RegistryDirectory@53fc870f]
	at com.alibaba.dubbo.rpc.cluster.support.AvailableCluster$1.doInvoke(AvailableCluster.java:48)
	at com.alibaba.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:244)
	at com.alibaba.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:75)
	at com.alibaba.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:52)
	at com.alibaba.dubbo.common.bytecode.proxy0.getMemberInfo(proxy0.java)

Dubbo：com.alibaba.dubbo.rpc.RpcException,实际netty-all并不会导致这个问题，不过没有会出现Dubbo 2.6.5启动报java.lang.NoClassDefFoundError: io/netty/channel/EventLoopGroup,springmvc需要这个jar，但是springboot工程并需要

java.lang.NoClassDefFoundError: io/netty/channel/EventLoopGroup
	at com.alibaba.dubbo.qos.protocol.QosProtocolWrapper.startQosServer(QosProtocolWrapper.java:96)
	at com.alibaba.dubbo.qos.protocol.QosProtocolWrapper.refer(QosProtocolWrapper.java:68)
	at com.alibaba.dubbo.rpc.Protocol$Adaptive.refer(Protocol$Adaptive.java)
	at com.alibaba.dubbo.config.ReferenceConfig.createProxy(ReferenceConfig.java:401)
	at com.alibaba.dubbo.config.ReferenceConfig.init(ReferenceConfig.java:335)
	at com.alibaba.dubbo.config.ReferenceConfig.get(ReferenceConfig.java:164)

将这个jar放在启动工程即可

<dependency>
			<groupId>io.netty</groupId>
			<artifactId>netty-all</artifactId>
			<version>4.1.32.Final</version>
		</dependency>

如果dubbo的日志没有打印出来，那么有些日志也就看不出来，log4j-over-slf4j这个不要漏了，否则dubbo的日志就没有

!-- log -->
		<dependency>
			<groupId>org.slf4j</groupId>
			<artifactId>slf4j-api</artifactId>
		</dependency>
		<dependency>
			<groupId>ch.qos.logback</groupId>
			<artifactId>logback-classic</artifactId>
		</dependency>
		<dependency>
			<groupId>ch.qos.logback</groupId>
			<artifactId>logback-access</artifactId>
		</dependency>
		<dependency>
			<groupId>org.slf4j</groupId>
			<artifactId>log4j-over-slf4j</artifactId>
		</dependency>

打开日志后发现，在服务启动的时候抛出了异常,虽然在dubbo-admin中看到没有问题，但是已经出错了

18:03:06.026 ERROR com.alibaba.dubbo.qos.server.Server -  [DUBBO] qos-server can not bind localhost:22222, dubbo version: 2.6.7, current host: 10.101.5.102
java.net.BindException: Address already in use: bind
	at sun.nio.ch.Net.bind0(Native Method)
	at sun.nio.ch.Net.bind(Net.java:437)
	at sun.nio.ch.Net.bind(Net.java:429)
	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
	at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:130)
	at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:562)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1358)
	at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501)
	at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486)
	at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:1019)
	at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:258)
	at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:366)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:474)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:745)
18:03:06.027 WARN  com.alibaba.dubbo.qos.protocol.QosProtocolWrapper -  [DUBBO] Fail to start qos server: , dubbo version: 2.6.7, current host: 10.101.5.102
java.net.BindException: Address already in use: bind

知道问题了就好解决了，(3)Dubbo启动时qos-server can not bind localhost:22222错误解决,使用不同的qos.port就可以了，

<dubbo:application name="${domain}" owner="${dubbo.owner}"
                       organization="${dubbo.organization}" >
        <dubbo:parameter key="qos.enable" value="true"/>
        <dubbo:parameter key="qos.accept.foreign.ip" value="false"/>
        <dubbo:parameter key="qos.port" value="22200"/>
    </dubbo:application>

参考springboot+dubbo2.6.0关闭QOS服务，将qos.enable禁用掉也可以

<dubbo:application name="${domain}" owner="${dubbo.owner}"
                       organization="${dubbo.organization}" >
        <dubbo:parameter key="qos.enable" value="false"/>
        <dubbo:parameter key="qos.accept.foreign.ip" value="false"/>
        <!--<dubbo:parameter key="qos.port" value="22200"/>-->
    </dubbo:application>

warrah

关注

0
点赞
踩
9

收藏

觉得还不错? 一键收藏
打赏
3
评论
k8s中dubbo注册IP和消费的问题

在系统迁移过程要考虑到平稳过度，先将部分应用迁移到k8s上，但是发现并没有那么简单从dubbo-admin中可以看到注册服务的ip是容器组ip，如果所有应用都在k8s中自然是没有问题，但如果k8s与虚拟机中的服务相互调用，就存在问题了，因为找不到ip。参考Docker 容器内运行 Dubbo 服务，启动容器,但是dubbo中还是容器的IP，并不是主机ipdocker run --name ab99755feae7 -d -p 9010:9010 -p 21716:21716 -e Dubbo_IP
复制链接

扫一扫