k8s中dubbo注册IP和消费的问题

深碍√TFBOYSˉ_ 2023-02-20 04:56 7阅读 0赞

在系统迁移过程要考虑到平稳过度，先将部分应用迁移到k8s上，但是发现并没有那么简单  
![1][]  
从dubbo-admin中可以看到注册服务的ip是容器组ip，如果所有应用都在k8s中自然是没有问题，但如果k8s与虚拟机中的服务相互调用，就存在问题了，因为找不到ip。  
![1][1 1]  
参考[Docker 容器内运行 Dubbo 服务][Docker _ Dubbo]，启动容器,但是dubbo中还是容器的IP，并不是主机ip

docker run --name ab99755feae7 -d -p 9010:9010 -p 21716:21716 -e Dubbo_IP_TO_REGISTRY=10.101.7.66  ab99755feae7
    # 再查看容器的环境变量，
    [root@worker3 ~]# docker exec 1f559cde8604 env
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    HOSTNAME=1f559cde8604
    DUBBO_IP_TO_REGISTRY=10.101.7.66
    LANG=C.UTF-8
    JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
    JAVA_VERSION=8u111
    JAVA_DEBIAN_VERSION=8u111-b14-2~bpo8+1
    CA_CERTIFICATES_JAVA_VERSION=20140324
    HOME=/root

接着翻阅[Spring cloud dubbo docker部署IP注册问题][Spring cloud dubbo docker_IP]，但我查看`dubbo-2.8.4` 的源码，没看到读取环境变量的地方，不知道他说的源码是哪里的。查看[MavenRepository][]，最新的版本也才`2.6.8`，估计是当初使用当当的`dubbox`导致的，应为我的dubbo是2016年就可以使使用了。  
通过`docker cp f8f041b98c79:/eztserver.jar /appdata`，下载查看后发现里面确实没有环境变量，而官方`2.6.7`版本确实有这个。于是想到了替换版本。  
但并不是直接替换就可以了会出问题，[Unable to locate Spring NamespaceHandler for XML schema namespace排错记录][Unable to locate Spring NamespaceHandler for XML schema namespace]  
![1][1 2]  
`spring.schemas`的内容如下：

http\://dubbo.apache.org/schema/dubbo/dubbo.xsd=META-INF/dubbo.xsd
    http\://code.alibabatech.com/schema/dubbo/dubbo.xsd=META-INF/compat/dubbo.xsd

启动服务，提示下面的错误，

\target\classes\dubbo\dubbo.xml] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 9; columnNumber: 42; cvc-complex-type.2.4.c: 通配符的匹配很全面, 但无法找到元素 'dubbo:application' 的声明。
    	at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:398)
    	at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:335)

检查这些地方，都是正确的。为啥还提示这个错误呢？  
![1][1 3]  
如果放弃`dubbo.xml`的方式，采取springboot的方式，再调用`docker run --name b448720570ed -d -p 9010:9010 -p 21716:21716 -e DUBBO_IP_TO_REGISTRY=10.101.7.65 b448720570ed`,这个时候ip就是主机ip了

<properties>
    		<dubbo.version>2.7.5</dubbo.version>
    		<zookeeper.version>3.4.14</zookeeper.version>
    	</properties>
    	
    		<dependency>
    			<groupId>org.apache.dubbo</groupId>
    			<artifactId>dubbo</artifactId>
    			<version>${dubbo.version}</version>
    			<exclusions>
    				<exclusion>
    					<groupId>org.springframework</groupId>
    					<artifactId>spring</artifactId>
    				</exclusion>
    				<exclusion>
    					<groupId>javax.servlet</groupId>
    					<artifactId>servlet-api</artifactId>
    				</exclusion>
    				<exclusion>
    					<groupId>log4j</groupId>
    					<artifactId>log4j</artifactId>
    				</exclusion>
    			</exclusions>
    		</dependency>
    
    		
    		<dependency>
    			<groupId>org.apache.zookeeper</groupId>
    			<artifactId>zookeeper</artifactId>
    			<version>${zookeeper.version}</version>
    			<exclusions>
    				<exclusion>
    					<artifactId>slf4j-log4j12</artifactId>
    					<groupId>org.slf4j</groupId>
    				</exclusion>
    			</exclusions>
    		</dependency>
    		<dependency>
    			<groupId>com.github.sgroschupf</groupId>
    			<artifactId>zkclient</artifactId>
    			<version>0.1</version>
    		</dependency>
    		<dependency>
    			<groupId>org.apache.curator</groupId>
    			<artifactId>curator-recipes</artifactId>
    			<version>4.2.0</version>
    		</dependency>
    		<dependency>
    			<groupId>org.apache.curator</groupId>
    			<artifactId>curator-framework</artifactId>
    			<version>4.2.0</version>
    		</dependency>

在springboot的启动类设置`DubboComponentScan`和`EnableDubboConfig`

import lombok.extern.slf4j.Slf4j;
    import org.apache.dubbo.config.spring.context.annotation.DubboComponentScan;
    import org.apache.dubbo.config.spring.context.annotation.EnableDubboConfig;
    import org.mybatis.spring.annotation.MapperScan;
    import org.slf4j.Logger;
    import org.slf4j.LoggerFactory;
    import org.springframework.boot.SpringApplication;
    import org.springframework.boot.autoconfigure.SpringBootApplication;
    import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
    import org.springframework.cloud.netflix.eureka.EnableEurekaClient;
    import org.springframework.cloud.openfeign.EnableFeignClients;
    import org.springframework.context.annotation.ComponentScan;
    import org.springframework.scheduling.annotation.EnableScheduling;
    
    @Slf4j
    @SpringBootApplication
    @ComponentScan(basePackages = {  "com.test" })
    @MapperScan("com.bwjf.**.dao")
    @EnableScheduling
    @EnableEurekaClient
    @EnableFeignClients(basePackages = { "com.test"})
    @EnableDubboConfig
    @DubboComponentScan("com.test.bzt.dubbo.service.impl")
    @EnableDiscoveryClient
    public class Application { 
    	
    	private static Logger log = LoggerFactory.getLogger(Application.class);
    
        public static void main(String[] args) { 
            try { 
                System.setProperty("es.set.netty.runtime.available.processors", "false");
                SpringApplication.run(Application.class, args);
            } catch (Exception e) { 
                log.info(e.getMessage(), e);
            } finally { 
                log.info("server start finish");
            }
        }
    }

而在yaml中设置dubbo

#Dubbbo
    domain: bztserver  
    dubbo:
      owner: zhengbangshun
      organization: bwjf
      registry:
        address: zookeeper://192.168.5.128:2181?backup=192.168.5.128:2182,192.168.5.128:2183
      protocol:
        name: dubbo
        port: 21716
        accesslog: dubbo-access.log
      service:
        validation: true
        version: 1.0.0
        timeout: 10000
        retries: 2
        loadbalance: random
        actives: 150
      reference:
        check: false
        version: 1.0.0

在k8s中并不能直接应用，因为你不知道他会将程序不出到那台机器上，故可参考[Dubbo与Kubernetes集成][Dubbo_Kubernetes],通过`status.hostIP`来制定主机ip

---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      annotations:
        deployment.kubernetes.io/revision: '3'
        k8s.eip.work/ingress: 'false'
        k8s.eip.work/service: none
        k8s.eip.work/workload: eztserver
        k8s.kuboard.cn/ingress: 'false'
        k8s.kuboard.cn/service: none
        k8s.kuboard.cn/workload: eztserver
      creationTimestamp: '2020-06-28T05:01:08Z'
      generation: 3
      labels:
        eztserver: eztserver
        k8s.eip.work/layer: ''
        k8s.eip.work/name: eztserver
      name: eztserver
      namespace: java
      resourceVersion: '2785134'
      selfLink: /apis/apps/v1/namespaces/java/deployments/eztserver
      uid: 1470af0b-0b71-4faf-bdbb-4a3534c1795a
    spec:
      progressDeadlineSeconds: 600
      replicas: 1
      revisionHistoryLimit: 10
      selector:
        matchLabels:
          eztserver: eztserver
          k8s.eip.work/layer: ''
          k8s.eip.work/name: eztserver
      strategy:
        rollingUpdate:
          maxSurge: 25%
          maxUnavailable: 25%
        type: RollingUpdate
      template:
        metadata:
          creationTimestamp: null
          labels:
            eztserver: eztserver
            k8s.eip.work/layer: ''
            k8s.eip.work/name: eztserver
        spec:
          containers:
            - env:
                - name: DUBBO_IP_TO_REGISTRY
                  valueFrom:
                    fieldRef:
                      apiVersion: v1
                      fieldPath: status.hostIP
              image: '192.168.5.101/java/eztserver:1.0.0'
              imagePullPolicy: Always
              lifecycle: { }
              name: eztserver
              ports:
                - containerPort: 9010
                  hostPort: 9010
                  protocol: TCP
                - containerPort: 21716
                  hostPort: 21716
                  protocol: TCP
              resources: { }
              terminationMessagePath: /dev/termination-log
              terminationMessagePolicy: File
          dnsConfig: { }
          dnsPolicy: ClusterFirst
          imagePullSecrets:
            - name: harbor192.168.5.101
          restartPolicy: Always
          schedulerName: default-scheduler
          securityContext:
            seLinuxOptions: { }
          terminationGracePeriodSeconds: 30

容器的dubbo端口也无需要开放出来  
![1][1 4]  
注册问题算是解决了，新的问题来，k8s容器内部的应用反而无法调用了  
![1][1 5]  
但是在dubbo-admin中看到消费者是主机ip，而不是容器ip，所以消费失败  
![1][1 6]  
接着下载[dubbo-demo][],做简单的配置

<dependency>
                <groupId>javax.validation</groupId>
                <artifactId>validation-api</artifactId>
                <version>1.1.0.Final</version>
            </dependency>
            <dependency>
                <groupId>org.hibernate</groupId>
                <artifactId>hibernate-validator</artifactId>
                <version>5.1.0.Final</version>
            </dependency>

否则可能会有下面的错误,因为接口做了校验处理

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'defaultValidator' defined in class path resource [org/springframework/boot/autoconfigure/validation/ValidationAutoConfiguration.class]: Invocation of init method failed; nested exception is java.lang.AbstractMethodError: org.hibernate.validator.engine.ConfigurationImpl.getDefaultParameterNameProvider()Ljavax/validation/ParameterNameProvider;
    	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1694) ~[spring-beans-5.0.10.RELEASE.jar:5.0.10.RELEASE]
    	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:573) ~[spring-beans-5.0.10.RELEASE.jar:5.0.10.RELEASE]
    	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:495) ~[spring-beans-5.0.10.RELEASE.jar:5.0.10.RELEASE]
    	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317) ~[spring-beans-5.0.10.RELEASE.jar:5.0.10.RELEASE]
    	at org.springframework.beans.factory.support.AbstractBeanFactory$$Lambda$118/631673932.getObject(Unknown Source)

经过测试，在本地是可以调用到docker中发布的dubbo服务，但是docker内部的应用却不能访问，那么就可以断定是消费者注册ip的问题  
![1][1 7]  
跟踪`UrlUtils`中可以看到dubbo会将consumer和provider进行比对  
![1][1 8]  
继续跟踪，参照[spring工程dubbo docker迁移][spring_dubbo docker]做改造之后，在springboot工程中可以获取到接口返回，但是springmvc却找不到提供者  
入下图，springboot工程和springmvc就可以成为对应的消费者，订阅了这个服务  
![1][1 9]  
springmvc工程就提示下面的异常

com.alibaba.dubbo.rpc.RpcException: No provider available in [invoker :interface com.bwjf.bdk.base.service.BdkWxService -> zookeeper://192.168.5.157:2181/com.alibaba.dubbo.registry.RegistryService?actives=150&anyhost=true&application=BWN&bean.name=com.bwjf.bdk.base.service.BdkWxService&check=false&dubbo=2.0.2&generic=false&interface=com.bwjf.bdk.base.service.BdkWxService&loadbalance=random&methods=getCardInfo,getStoreCard,getCoupon,getStoreMemberInfo,getMemberCouponInfo,getMemberCardList,getMemberSerialOrderInfo,getMemberSerialOrderList,bindMember,getMemberInfo,updateMemberInfo,getStoreInfo,getStoreCoupon,getCard,getCouponInfo,getMemberCoupon,registerMember,saveMemberTitle,getMemberStoreList,memberTitleDetail&organization=bwjf&owner=liuqiao&pid=24632&register.ip=10.101.5.102&remote.timestamp=1594188092802&retries=2&revision=1.0.0&side=consumer&timeout=60000&timestamp=1594189017615&version=1.0.0,directory: com.alibaba.dubbo.registry.integration.RegistryDirectory@3a917017, invoker :interface com.bwjf.bdk.base.service.BdkWxService -> zookeeper://192.168.5.157:2181/com.alibaba.dubbo.registry.RegistryService?actives=150&anyhost=true&application=BWN&bean.name=com.bwjf.bdk.base.service.BdkWxService&check=false&dubbo=2.0.2&generic=false&interface=com.bwjf.bdk.base.service.BdkWxService&loadbalance=random&methods=getCardInfo,getStoreCard,getCoupon,getStoreMemberInfo,getMemberCouponInfo,getMemberCardList,getMemberSerialOrderInfo,getMemberSerialOrderList,bindMember,getMemberInfo,updateMemberInfo,getStoreInfo,getStoreCoupon,getCard,getCouponInfo,getMemberCoupon,registerMember,saveMemberTitle,getMemberStoreList,memberTitleDetail&organization=bwjf&owner=liuqiao&pid=24632&register.ip=10.101.5.102&remote.timestamp=1594188092802&retries=2&revision=1.0.0&side=consumer&timeout=60000&timestamp=1594189017615&version=1.0.0,directory: com.alibaba.dubbo.registry.integration.RegistryDirectory@53fc870f]
    	at com.alibaba.dubbo.rpc.cluster.support.AvailableCluster$1.doInvoke(AvailableCluster.java:48)
    	at com.alibaba.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:244)
    	at com.alibaba.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:75)
    	at com.alibaba.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:52)
    	at com.alibaba.dubbo.common.bytecode.proxy0.getMemberInfo(proxy0.java)

[Dubbo：com.alibaba.dubbo.rpc.RpcException][Dubbo_com.alibaba.dubbo.rpc.RpcException],实际`netty-all`并不会导致这个问题 ，不过没有会出现[Dubbo 2.6.5启动报java.lang.NoClassDefFoundError: io/netty/channel/EventLoopGroup][Dubbo 2.6.5_java.lang.NoClassDefFoundError_ io_netty_channel_EventLoopGroup],springmvc需要这个jar，但是springboot工程并需要

java.lang.NoClassDefFoundError: io/netty/channel/EventLoopGroup
    	at com.alibaba.dubbo.qos.protocol.QosProtocolWrapper.startQosServer(QosProtocolWrapper.java:96)
    	at com.alibaba.dubbo.qos.protocol.QosProtocolWrapper.refer(QosProtocolWrapper.java:68)
    	at com.alibaba.dubbo.rpc.Protocol$Adaptive.refer(Protocol$Adaptive.java)
    	at com.alibaba.dubbo.config.ReferenceConfig.createProxy(ReferenceConfig.java:401)
    	at com.alibaba.dubbo.config.ReferenceConfig.init(ReferenceConfig.java:335)
    	at com.alibaba.dubbo.config.ReferenceConfig.get(ReferenceConfig.java:164)

将这个jar放在启动工程即可

<dependency>
    			<groupId>io.netty</groupId>
    			<artifactId>netty-all</artifactId>
    			<version>4.1.32.Final</version>
    		</dependency>

如果dubbo的日志没有打印出来，那么有些日志也就看不出来，`log4j-over-slf4j`这个不要漏了，否则dubbo的日志就没有

!-- log -->
    		<dependency>
    			<groupId>org.slf4j</groupId>
    			<artifactId>slf4j-api</artifactId>
    		</dependency>
    		<dependency>
    			<groupId>ch.qos.logback</groupId>
    			<artifactId>logback-classic</artifactId>
    		</dependency>
    		<dependency>
    			<groupId>ch.qos.logback</groupId>
    			<artifactId>logback-access</artifactId>
    		</dependency>
    		<dependency>
    			<groupId>org.slf4j</groupId>
    			<artifactId>log4j-over-slf4j</artifactId>
    		</dependency>

打开日志后发现，在服务启动的时候抛出了异常,虽然在`dubbo-admin`中看到没有问题，但是已经出错了

18:03:06.026 ERROR com.alibaba.dubbo.qos.server.Server -  [DUBBO] qos-server can not bind localhost:22222, dubbo version: 2.6.7, current host: 10.101.5.102
    java.net.BindException: Address already in use: bind
    	at sun.nio.ch.Net.bind0(Native Method)
    	at sun.nio.ch.Net.bind(Net.java:437)
    	at sun.nio.ch.Net.bind(Net.java:429)
    	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
    	at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:130)
    	at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:562)
    	at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1358)
    	at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501)
    	at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486)
    	at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:1019)
    	at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:258)
    	at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:366)
    	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:163)
    	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
    	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
    	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:474)
    	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
    	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    	at java.lang.Thread.run(Thread.java:745)
    18:03:06.027 WARN  com.alibaba.dubbo.qos.protocol.QosProtocolWrapper -  [DUBBO] Fail to start qos server: , dubbo version: 2.6.7, current host: 10.101.5.102
    java.net.BindException: Address already in use: bind

知道问题了就好解决了，[(3)Dubbo启动时qos-server can not bind localhost:22222错误解决][3_Dubbo_qos-server can not bind localhost_22222],使用不同的`qos.port`就可以了，

<dubbo:application name="${domain}" owner="${dubbo.owner}" organization="${dubbo.organization}" >
            <dubbo:parameter key="qos.enable" value="true"/>
            <dubbo:parameter key="qos.accept.foreign.ip" value="false"/>
            <dubbo:parameter key="qos.port" value="22200"/>
        </dubbo:application>

参考[springboot+dubbo2.6.0关闭QOS服务][springboot_dubbo2.6.0_QOS]，将`qos.enable`禁用掉也可以

<dubbo:application name="${domain}" owner="${dubbo.owner}" organization="${dubbo.organization}" >
            <dubbo:parameter key="qos.enable" value="false"/>
            <dubbo:parameter key="qos.accept.foreign.ip" value="false"/>
            
        </dubbo:application>

[1]: https://img-blog.csdnimg.cn/20200623145323621.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA==,size_16,color_FFFFFF,t_70
[1 1]: https://img-blog.csdnimg.cn/20200623144915626.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA==,size_16,color_FFFFFF,t_70
[Docker _ Dubbo]: https://zhuanlan.zhihu.com/p/66805023
[Spring cloud dubbo docker_IP]: https://blog.csdn.net/liubingyu12345/article/details/93464587
[MavenRepository]: https://mvnrepository.com/artifact/com.alibaba/dubbo
[Unable to locate Spring NamespaceHandler for XML schema namespace]: https://blog.csdn.net/qq_31772441/article/details/82533864
[1 2]: https://img-blog.csdnimg.cn/20200624115938888.jpg?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA==,size_16,color_FFFFFF,t_70
[1 3]: https://img-blog.csdnimg.cn/20200624125123768.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA==,size_16,color_FFFFFF,t_70
[Dubbo_Kubernetes]: https://www.cnblogs.com/nxlhero/archive/2019/11/04/11789006.html
[1 4]: https://img-blog.csdnimg.cn/20200708081957878.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA==,size_16,color_FFFFFF,t_70
[1 5]: https://img-blog.csdnimg.cn/20200707155442647.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA==,size_16,color_FFFFFF,t_70
[1 6]: https://img-blog.csdnimg.cn/20200707155727508.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA==,size_16,color_FFFFFF,t_70
[dubbo-demo]: https://github.com/nzomkxia/dubbo-demo
[1 7]: https://img-blog.csdnimg.cn/20200708091717584.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA==,size_16,color_FFFFFF,t_70
[1 8]: https://img-blog.csdnimg.cn/20200708104122144.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA==,size_16,color_FFFFFF,t_70
[spring_dubbo docker]: https://blog.csdn.net/warrah/article/details/107157257
[1 9]: https://img-blog.csdnimg.cn/2020070814182688.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhcnJhaA==,size_16,color_FFFFFF,t_70
[Dubbo_com.alibaba.dubbo.rpc.RpcException]: https://blog.csdn.net/a_842297171/article/details/80819202
[Dubbo 2.6.5_java.lang.NoClassDefFoundError_ io_netty_channel_EventLoopGroup]: https://blog.csdn.net/q283614346/article/details/85042036
[3_Dubbo_qos-server can not bind localhost_22222]: https://blog.csdn.net/u012988901/article/details/84503672
[springboot_dubbo2.6.0_QOS]: https://blog.csdn.net/xdkprosperous/article/details/99583204