k8s中dubbo注册IP和消费的问题

深碍√TFBOYSˉ_ 2023-02-20 04:56 89阅读 0赞

在系统迁移过程要考虑到平稳过度，先将部分应用迁移到k8s上，但是发现并没有那么简单
1
从dubbo-admin中可以看到注册服务的ip是容器组ip，如果所有应用都在k8s中自然是没有问题，但如果k8s与虚拟机中的服务相互调用，就存在问题了，因为找不到ip。
1
参考Docker 容器内运行 Dubbo 服务，启动容器,但是dubbo中还是容器的IP，并不是主机ip

  1. docker run --name ab99755feae7 -d -p 9010:9010 -p 21716:21716 -e Dubbo_IP_TO_REGISTRY=10.101.7.66  ab99755feae7
  2. # 再查看容器的环境变量，
  3. [root@worker3 ~]# docker exec 1f559cde8604 env
  4. PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
  5. HOSTNAME=1f559cde8604
  6. DUBBO_IP_TO_REGISTRY=10.101.7.66
  7. LANG=C.UTF-8
  8. JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
  9. JAVA_VERSION=8u111
  10. JAVA_DEBIAN_VERSION=8u111-b14-2~bpo8+1
  11. CA_CERTIFICATES_JAVA_VERSION=20140324
  12. HOME=/root

接着翻阅Spring cloud dubbo docker部署IP注册问题，但我查看dubbo-2.8.4 的源码，没看到读取环境变量的地方，不知道他说的源码是哪里的。查看MavenRepository，最新的版本也才2.6.8，估计是当初使用当当的dubbox导致的，应为我的dubbo是2016年就可以使使用了。
通过docker cp f8f041b98c79:/eztserver.jar /appdata，下载查看后发现里面确实没有环境变量，而官方2.6.7版本确实有这个。于是想到了替换版本。
但并不是直接替换就可以了会出问题，Unable to locate Spring NamespaceHandler for XML schema namespace排错记录
1
spring.schemas的内容如下：

  1. http\://dubbo.apache.org/schema/dubbo/dubbo.xsd=META-INF/dubbo.xsd
  2. http\://code.alibabatech.com/schema/dubbo/dubbo.xsd=META-INF/compat/dubbo.xsd

启动服务，提示下面的错误，

  1. \target\classes\dubbo\dubbo.xml] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 9; columnNumber: 42; cvc-complex-type.2.4.c: 通配符的匹配很全面, 但无法找到元素 'dubbo:application' 的声明。
  2.     at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:398)
  3.     at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:335)

检查这些地方，都是正确的。为啥还提示这个错误呢？
1
如果放弃dubbo.xml的方式，采取springboot的方式，再调用docker run --name b448720570ed -d -p 9010:9010 -p 21716:21716 -e DUBBO_IP_TO_REGISTRY=10.101.7.65 b448720570ed,这个时候ip就是主机ip了

  1. <properties>
  2.         <dubbo.version>2.7.5</dubbo.version>
  3.         <zookeeper.version>3.4.14</zookeeper.version>
  4.     </properties>
  5.     <!-- dubbo依赖 -->
  6.         <dependency>
  7.             <groupId>org.apache.dubbo</groupId>
  8.             <artifactId>dubbo</artifactId>
  9.             <version>${dubbo.version}</version>
  10.             <exclusions>
  11.                 <exclusion>
  12.                     <groupId>org.springframework</groupId>
  13.                     <artifactId>spring</artifactId>
  14.                 </exclusion>
  15.                 <exclusion>
  16.                     <groupId>javax.servlet</groupId>
  17.                     <artifactId>servlet-api</artifactId>
  18.                 </exclusion>
  19.                 <exclusion>
  20.                     <groupId>log4j</groupId>
  21.                     <artifactId>log4j</artifactId>
  22.                 </exclusion>
  23.             </exclusions>
  24.         </dependency>
  26.         <!-- zookeeper依赖 -->
  27.         <dependency>
  28.             <groupId>org.apache.zookeeper</groupId>
  29.             <artifactId>zookeeper</artifactId>
  30.             <version>${zookeeper.version}</version>
  31.             <exclusions>
  32.                 <exclusion>
  33.                     <artifactId>slf4j-log4j12</artifactId>
  34.                     <groupId>org.slf4j</groupId>
  35.                 </exclusion>
  36.             </exclusions>
  37.         </dependency>
  38.         <dependency>
  39.             <groupId>com.github.sgroschupf</groupId>
  40.             <artifactId>zkclient</artifactId>
  41.             <version>0.1</version>
  42.         </dependency>
  43.         <dependency>
  44.             <groupId>org.apache.curator</groupId>
  45.             <artifactId>curator-recipes</artifactId>
  46.             <version>4.2.0</version>
  47.         </dependency>
  48.         <dependency>
  49.             <groupId>org.apache.curator</groupId>
  50.             <artifactId>curator-framework</artifactId>
  51.             <version>4.2.0</version>
  52.         </dependency>

在springboot的启动类设置DubboComponentScan和EnableDubboConfig

  1. import lombok.extern.slf4j.Slf4j;
  2. import org.apache.dubbo.config.spring.context.annotation.DubboComponentScan;
  3. import org.apache.dubbo.config.spring.context.annotation.EnableDubboConfig;
  4. import org.mybatis.spring.annotation.MapperScan;
  5. import org.slf4j.Logger;
  6. import org.slf4j.LoggerFactory;
  7. import org.springframework.boot.SpringApplication;
  8. import org.springframework.boot.autoconfigure.SpringBootApplication;
  9. import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
  10. import org.springframework.cloud.netflix.eureka.EnableEurekaClient;
  11. import org.springframework.cloud.openfeign.EnableFeignClients;
  12. import org.springframework.context.annotation.ComponentScan;
  13. import org.springframework.scheduling.annotation.EnableScheduling;
  15. @Slf4j
  16. @SpringBootApplication
  17. @ComponentScan(basePackages = {  "com.test" })
  18. @MapperScan("com.bwjf.**.dao")
  19. @EnableScheduling
  20. @EnableEurekaClient
  21. @EnableFeignClients(basePackages = { "com.test"})
  22. @EnableDubboConfig
  23. @DubboComponentScan("com.test.bzt.dubbo.service.impl")
  24. @EnableDiscoveryClient
  25. public class Application { 
  27.     private static Logger log = LoggerFactory.getLogger(Application.class);
  29.     public static void main(String[] args) { 
  30.         try { 
  31.             System.setProperty("es.set.netty.runtime.available.processors", "false");
  32.             SpringApplication.run(Application.class, args);
  33.         } catch (Exception e) { 
  34.             log.info(e.getMessage(), e);
  35.         } finally { 
  36.             log.info("server start finish");
  37.         }
  38.     }
  39. }

而在yaml中设置dubbo

  1. #Dubbbo
  2. domain: bztserver  
  3. dubbo:
  4.   owner: zhengbangshun
  5.   organization: bwjf
  6.   registry:
  7.     address: zookeeper://192.168.5.128:2181?backup=192.168.5.128:2182,192.168.5.128:2183
  8.   protocol:
  9.     name: dubbo
  10.     port: 21716
  11.     accesslog: dubbo-access.log
  12.   service:
  13.     validation: true
  14.     version: 1.0.0
  15.     timeout: 10000
  16.     retries: 2
  17.     loadbalance: random
  18.     actives: 150
  19.   reference:
  20.     check: false
  21.     version: 1.0.0

在k8s中并不能直接应用，因为你不知道他会将程序不出到那台机器上，故可参考Dubbo与Kubernetes集成,通过status.hostIP来制定主机ip

  1. ---
  2. apiVersion: apps/v1
  3. kind: Deployment
  4. metadata:
  5.   annotations:
  6.     deployment.kubernetes.io/revision: '3'
  7.     k8s.eip.work/ingress: 'false'
  8.     k8s.eip.work/service: none
  9.     k8s.eip.work/workload: eztserver
  10.     k8s.kuboard.cn/ingress: 'false'
  11.     k8s.kuboard.cn/service: none
  12.     k8s.kuboard.cn/workload: eztserver
  13.   creationTimestamp: '2020-06-28T05:01:08Z'
  14.   generation: 3
  15.   labels:
  16.     eztserver: eztserver
  17.     k8s.eip.work/layer: ''
  18.     k8s.eip.work/name: eztserver
  19.   name: eztserver
  20.   namespace: java
  21.   resourceVersion: '2785134'
  22.   selfLink: /apis/apps/v1/namespaces/java/deployments/eztserver
  23.   uid: 1470af0b-0b71-4faf-bdbb-4a3534c1795a
  24. spec:
  25.   progressDeadlineSeconds: 600
  26.   replicas: 1
  27.   revisionHistoryLimit: 10
  28.   selector:
  29.     matchLabels:
  30.       eztserver: eztserver
  31.       k8s.eip.work/layer: ''
  32.       k8s.eip.work/name: eztserver
  33.   strategy:
  34.     rollingUpdate:
  35.       maxSurge: 25%
  36.       maxUnavailable: 25%
  37.     type: RollingUpdate
  38.   template:
  39.     metadata:
  40.       creationTimestamp: null
  41.       labels:
  42.         eztserver: eztserver
  43.         k8s.eip.work/layer: ''
  44.         k8s.eip.work/name: eztserver
  45.     spec:
  46.       containers:
  47.         - env:
  48.             - name: DUBBO_IP_TO_REGISTRY
  49.               valueFrom:
  50.                 fieldRef:
  51.                   apiVersion: v1
  52.                   fieldPath: status.hostIP
  53.           image: '192.168.5.101/java/eztserver:1.0.0'
  54.           imagePullPolicy: Always
  55.           lifecycle: { }
  56.           name: eztserver
  57.           ports:
  58.             - containerPort: 9010
  59.               hostPort: 9010
  60.               protocol: TCP
  61.             - containerPort: 21716
  62.               hostPort: 21716
  63.               protocol: TCP
  64.           resources: { }
  65.           terminationMessagePath: /dev/termination-log
  66.           terminationMessagePolicy: File
  67.       dnsConfig: { }
  68.       dnsPolicy: ClusterFirst
  69.       imagePullSecrets:
  70.         - name: harbor192.168.5.101
  71.       restartPolicy: Always
  72.       schedulerName: default-scheduler
  73.       securityContext:
  74.         seLinuxOptions: { }
  75.       terminationGracePeriodSeconds: 30

容器的dubbo端口也无需要开放出来
1
注册问题算是解决了，新的问题来，k8s容器内部的应用反而无法调用了
1
但是在dubbo-admin中看到消费者是主机ip，而不是容器ip，所以消费失败
1
接着下载dubbo-demo,做简单的配置

  1. <dependency>
  2.             <groupId>javax.validation</groupId>
  3.             <artifactId>validation-api</artifactId>
  4.             <version>1.1.0.Final</version>
  5.         </dependency>
  6.         <dependency>
  7.             <groupId>org.hibernate</groupId>
  8.             <artifactId>hibernate-validator</artifactId>
  9.             <version>5.1.0.Final</version>
  10.         </dependency>

否则可能会有下面的错误,因为接口做了校验处理

  1. org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'defaultValidator' defined in class path resource [org/springframework/boot/autoconfigure/validation/ValidationAutoConfiguration.class]: Invocation of init method failed; nested exception is java.lang.AbstractMethodError: org.hibernate.validator.engine.ConfigurationImpl.getDefaultParameterNameProvider()Ljavax/validation/ParameterNameProvider;
  2.     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1694) ~[spring-beans-5.0.10.RELEASE.jar:5.0.10.RELEASE]
  3.     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:573) ~[spring-beans-5.0.10.RELEASE.jar:5.0.10.RELEASE]
  4.     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:495) ~[spring-beans-5.0.10.RELEASE.jar:5.0.10.RELEASE]
  5.     at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317) ~[spring-beans-5.0.10.RELEASE.jar:5.0.10.RELEASE]
  6.     at org.springframework.beans.factory.support.AbstractBeanFactory$$Lambda$118/631673932.getObject(Unknown Source)

经过测试，在本地是可以调用到docker中发布的dubbo服务，但是docker内部的应用却不能访问，那么就可以断定是消费者注册ip的问题
1
跟踪UrlUtils中可以看到dubbo会将consumer和provider进行比对
1
继续跟踪，参照spring工程dubbo docker迁移做改造之后，在springboot工程中可以获取到接口返回，但是springmvc却找不到提供者
入下图，springboot工程和springmvc就可以成为对应的消费者，订阅了这个服务
1
springmvc工程就提示下面的异常

  1. com.alibaba.dubbo.rpc.RpcException: No provider available in [invoker :interface com.bwjf.bdk.base.service.BdkWxService -> zookeeper://192.168.5.157:2181/com.alibaba.dubbo.registry.RegistryService?actives=150&anyhost=true&application=BWN&bean.name=com.bwjf.bdk.base.service.BdkWxService&check=false&dubbo=2.0.2&generic=false&interface=com.bwjf.bdk.base.service.BdkWxService&loadbalance=random&methods=getCardInfo,getStoreCard,getCoupon,getStoreMemberInfo,getMemberCouponInfo,getMemberCardList,getMemberSerialOrderInfo,getMemberSerialOrderList,bindMember,getMemberInfo,updateMemberInfo,getStoreInfo,getStoreCoupon,getCard,getCouponInfo,getMemberCoupon,registerMember,saveMemberTitle,getMemberStoreList,memberTitleDetail&organization=bwjf&owner=liuqiao&pid=24632&register.ip=10.101.5.102&remote.timestamp=1594188092802&retries=2&revision=1.0.0&side=consumer&timeout=60000&timestamp=1594189017615&version=1.0.0,directory: com.alibaba.dubbo.registry.integration.RegistryDirectory@3a917017, invoker :interface com.bwjf.bdk.base.service.BdkWxService -> zookeeper://192.168.5.157:2181/com.alibaba.dubbo.registry.RegistryService?actives=150&anyhost=true&application=BWN&bean.name=com.bwjf.bdk.base.service.BdkWxService&check=false&dubbo=2.0.2&generic=false&interface=com.bwjf.bdk.base.service.BdkWxService&loadbalance=random&methods=getCardInfo,getStoreCard,getCoupon,getStoreMemberInfo,getMemberCouponInfo,getMemberCardList,getMemberSerialOrderInfo,getMemberSerialOrderList,bindMember,getMemberInfo,updateMemberInfo,getStoreInfo,getStoreCoupon,getCard,getCouponInfo,getMemberCoupon,registerMember,saveMemberTitle,getMemberStoreList,memberTitleDetail&organization=bwjf&owner=liuqiao&pid=24632&register.ip=10.101.5.102&remote.timestamp=1594188092802&retries=2&revision=1.0.0&side=consumer&timeout=60000&timestamp=1594189017615&version=1.0.0,directory: com.alibaba.dubbo.registry.integration.RegistryDirectory@53fc870f]
  2.     at com.alibaba.dubbo.rpc.cluster.support.AvailableCluster$1.doInvoke(AvailableCluster.java:48)
  3.     at com.alibaba.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:244)
  4.     at com.alibaba.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:75)
  5.     at com.alibaba.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:52)
  6.     at com.alibaba.dubbo.common.bytecode.proxy0.getMemberInfo(proxy0.java)

Dubbo：com.alibaba.dubbo.rpc.RpcException,实际netty-all并不会导致这个问题 ，不过没有会出现Dubbo 2.6.5启动报java.lang.NoClassDefFoundError: io/netty/channel/EventLoopGroup,springmvc需要这个jar，但是springboot工程并需要

  1. java.lang.NoClassDefFoundError: io/netty/channel/EventLoopGroup
  2.     at com.alibaba.dubbo.qos.protocol.QosProtocolWrapper.startQosServer(QosProtocolWrapper.java:96)
  3.     at com.alibaba.dubbo.qos.protocol.QosProtocolWrapper.refer(QosProtocolWrapper.java:68)
  4.     at com.alibaba.dubbo.rpc.Protocol$Adaptive.refer(Protocol$Adaptive.java)
  5.     at com.alibaba.dubbo.config.ReferenceConfig.createProxy(ReferenceConfig.java:401)
  6.     at com.alibaba.dubbo.config.ReferenceConfig.init(ReferenceConfig.java:335)
  7.     at com.alibaba.dubbo.config.ReferenceConfig.get(ReferenceConfig.java:164)

将这个jar放在启动工程即可

  1. <dependency>
  2.             <groupId>io.netty</groupId>
  3.             <artifactId>netty-all</artifactId>
  4.             <version>4.1.32.Final</version>
  5.         </dependency>

如果dubbo的日志没有打印出来，那么有些日志也就看不出来，log4j-over-slf4j这个不要漏了，否则dubbo的日志就没有

  1. !-- log -->
  2.         <dependency>
  3.             <groupId>org.slf4j</groupId>
  4.             <artifactId>slf4j-api</artifactId>
  5.         </dependency>
  6.         <dependency>
  7.             <groupId>ch.qos.logback</groupId>
  8.             <artifactId>logback-classic</artifactId>
  9.         </dependency>
  10.         <dependency>
  11.             <groupId>ch.qos.logback</groupId>
  12.             <artifactId>logback-access</artifactId>
  13.         </dependency>
  14.         <dependency>
  15.             <groupId>org.slf4j</groupId>
  16.             <artifactId>log4j-over-slf4j</artifactId>
  17.         </dependency>

打开日志后发现，在服务启动的时候抛出了异常,虽然在dubbo-admin中看到没有问题，但是已经出错了

  1. 18:03:06.026 ERROR com.alibaba.dubbo.qos.server.Server -  [DUBBO] qos-server can not bind localhost:22222, dubbo version: 2.6.7, current host: 10.101.5.102
  2. java.net.BindException: Address already in use: bind
  3.     at sun.nio.ch.Net.bind0(Native Method)
  4.     at sun.nio.ch.Net.bind(Net.java:437)
  5.     at sun.nio.ch.Net.bind(Net.java:429)
  6.     at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
  7.     at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:130)
  8.     at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:562)
  9.     at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1358)
  10.     at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501)
  11.     at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486)
  12.     at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:1019)
  13.     at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:258)
  14.     at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:366)
  15.     at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:163)
  16.     at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
  17.     at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
  18.     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:474)
  19.     at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
  20.     at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
  21.     at java.lang.Thread.run(Thread.java:745)
  22. 18:03:06.027 WARN  com.alibaba.dubbo.qos.protocol.QosProtocolWrapper -  [DUBBO] Fail to start qos server: , dubbo version: 2.6.7, current host: 10.101.5.102
  23. java.net.BindException: Address already in use: bind

知道问题了就好解决了，(3)Dubbo启动时qos-server can not bind localhost:22222错误解决,使用不同的qos.port就可以了，

  1. <dubbo:application name="${domain}" owner="${dubbo.owner}" organization="${dubbo.organization}" >
  2.         <dubbo:parameter key="qos.enable" value="true"/>
  3.         <dubbo:parameter key="qos.accept.foreign.ip" value="false"/>
  4.         <dubbo:parameter key="qos.port" value="22200"/>
  5.     </dubbo:application>

参考springboot+dubbo2.6.0关闭QOS服务，将qos.enable禁用掉也可以

  1. <dubbo:application name="${domain}" owner="${dubbo.owner}" organization="${dubbo.organization}" >
  2.         <dubbo:parameter key="qos.enable" value="false"/>
  3.         <dubbo:parameter key="qos.accept.foreign.ip" value="false"/>
  4.         <!--<dubbo:parameter key="qos.port" value="22200"/>-->
  5.     </dubbo:application>

发表评论

表情：
评论列表 （有 0 条评论，89人围观）

还没有评论，来说两句吧...

相关阅读

    相关 【k8s】——k8s的GC机制

    一、前言       一说到GC，很多人(包括我在内)第一反应是内存的垃圾回收，尤其是javaer们，面试最爱考的八股文就是java的垃圾回收。不过今天要说的这个k8s的

    迷南。迷南。/ 0 赞/ 576 阅读