Sentinal动态配置规则之zookeeper配置规则

刺骨的言语ヽ痛彻心扉 2023-01-05 01:24 221阅读 0赞

Sentinel之所以需要配置动态规则，是因为默认情况下，流控、熔断等规则只保存在内存中，一旦客户端重启，规则就全消失了。而动态规则可以将将这些信息持久化到像Nacos、Zookeeper、Apollo等平台中去。一般在生产环境中推荐使用动态规则。

本人在配置动态规则的过程中，碰到了一些坑，然后又解决掉了，现在把它们记录下来，也可以给其他使用Sentinel的人一些参考。

这里测试的是sentinel与dubbo的整合。

## 一、Sentinel dashboard服务端 ##

因为sentinel dashboard默认是不支持持久化的，所以我们需要这个功能的时候需要自己进行相应的改造，官网上面有一些改造的步骤。

首先在github上下载sentinel的源码，找到sentinel-dashboard的那部分代码

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3_size_16_color_FFFFFF_t_70][]

进入com.alibaba.csp.sentinel.dashboard.controller.v2包下，找到FlowControllerV2这个类

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3_size_16_color_FFFFFF_t_70 1][]

将类中的指定的bean替换成zookeeper相关的bean。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3_size_16_color_FFFFFF_t_70 2][]

再将原来在src/test/java目录下的com.alibaba.csp.sentinel.dashboard.rule.zookeeper包剪切到src/main/java下面去

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3_size_16_color_FFFFFF_t_70 3][]

然后找到main方法，运行dashboard，同时要加上启动参数-Dserver.port=9352 -Dcsp.sentinel.dashboard.server=localhost:9352 -Dproject.name=sentinel-dashboard

![20191102144403540.png][]

启动成功的画面

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3_size_16_color_FFFFFF_t_70 4][]

浏览器上输入[http://localhost:9352][http_localhost_9352]进入控制台

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3_size_16_color_FFFFFF_t_70 5][]

**注意，目前为止，dashboard连接的zookeeper地址是写死在代码中的(如图)，我们需要将它提取到启动参数中，还要做一些修改。**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3_size_16_color_FFFFFF_t_70 6][]

首先，我们找到DashboardConfig这个类，在其中加入以下代码：

![20191102145150934.png][]

/**
         * Zookeeper address
         */
        public static final String CONFIG_ZK_ADDR = "sentinel.dashboard.zk.addr";
     
        public static String getZkAddr() {
            return getConfigStr(CONFIG_ZK_ADDR);
        }

然后在前面的那个ZookeeperConfig类中，将zookeeper的地址替换掉

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3_size_16_color_FFFFFF_t_70 7][]

然后，在启动参数中加入-Dsentinel.dashboard.zk.addr=192.168.10.15:2181即可指定zookeeper的地址了。

## 二、Sentinel dashboard客户端 ##

sentinel既可以对dubbo的提供者，也可以对消费者进行流控，不过官网上建议最好对消费者进行流控。本次测试中，sentinel是对于dubbo的消费端进行流量控制。dubbo服务端不需要额外配置，消费端的pom.xml文件需要导入以下内容：

pom.xml

<properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <sentinel.version>1.6.2</sentinel.version>
      </properties>
     
        
    	
    	<dependency>
    	    <groupId>com.alibaba.csp</groupId>
    	    <artifactId>sentinel-core</artifactId>
    	    <version>${sentinel.version}</version>
    	</dependency>
    	
    	
    	<dependency>
    	    <groupId>com.alibaba.csp</groupId>
    	    <artifactId>sentinel-transport-simple-http</artifactId>
    	    <version>${sentinel.version}</version>
    	</dependency>
    	
    	
    	<dependency>
    	    <groupId>com.alibaba.csp</groupId>
    	    <artifactId>sentinel-dubbo-adapter</artifactId>
    	    <version>${sentinel.version}</version>
    	</dependency>
    	
    	
    	 
    	<dependency> 
    	   <groupId>com.alibaba.csp</groupId> 
    	   <artifactId>sentinel-datasource-zookeeper</artifactId> 
    	   <version>${sentinel.version}</version> 
    	</dependency> 
    	
    	<dependency> 
    	      <groupId>com.alibaba.csp</groupId> 
    	      <artifactId>sentinel-datasource-extension</artifactId> 
    	      <version>${sentinel.version}</version> 
    	</dependency>

然后启动的时候监听zookeeper的某个节点，并实时同步到流控的规则中。只需要加入以下代码：

//接入zookeeper的规则源
    		final String remoteAddress = "192.168.10.15:2181";
            final String path = "/sentinel_rule_config/com.zhuyun.test.Main";
     
            ReadableDataSource<String, List<FlowRule>> flowRuleDataSource = new ZookeeperDataSource<>(remoteAddress, path,
                    source -> JSON.parseObject(source, new TypeReference<List<FlowRule>>() {}));
            FlowRuleManager.register2Property(flowRuleDataSource.getProperty());

当然，其中的变量可以写成配置文件或者启动参数的形式，这里就不演示了。

接下来，分别启动dubbo的提供者与消费者，然后在sentinel-dashboard控制台上可以看到他们了。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3_size_16_color_FFFFFF_t_70 8][]

接下来我们在控制台上设置流控规则，对run()这个方法进行限速

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3_size_16_color_FFFFFF_t_70 9][]

限速后，再来看看统计的结果

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3_size_16_color_FFFFFF_t_70 10][]

run()方法已经限制成功了，另一个方法依然是原来的速度。

而且，在zookeeper中，也已经存在了该条规则，下次重启的时候规则仍然在。

![20191102151439429.png][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3_size_16_color_FFFFFF_t_70 11][]

## 三、遇到的坑 ##

下面是调试过程中遇到过的一些坑。

**1、pom.xml**

<dependency>
                <groupId>org.apache.curator</groupId>
                <artifactId>curator-recipes</artifactId>
                <version>${curator.version}</version>
                
            </dependency>

刚开始将test中的zookeeper包拷贝到main下运行的时候，出现了以下的错误

java.lang.IllegalStateException: Error processing condition on org.springframework.boot.autoconfigure.context.PropertyPlaceholderAutoConfiguration.propertySourcesPlaceholderConfigurer
    	at org.springframework.boot.autoconfigure.condition.SpringBootCondition.matches(SpringBootCondition.java:64) ~[spring-boot-autoconfigure-2.0.5.RELEASE.jar:2.0.5.RELEASE]
    	at org.springframework.context.annotation.ConditionEvaluator.shouldSkip(ConditionEvaluator.java:108) ~[spring-context-5.0.9.RELEASE.jar:5.0.9.RELEASE]
    	at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitionsForBeanMethod(ConfigurationClassBeanDefinitionReader.java:180) ~[spring-context-5.0.9.RELEASE.jar:5.0.9.RELEASE]
    	at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitionsForConfigurationClass(ConfigurationClassBeanDefinitionReader.java:141) ~[spring-context-5.0.9.RELEASE.jar:5.0.9.RELEASE]
    	at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitions(ConfigurationClassBeanDefinitionReader.java:117) ~[spring-context-5.0.9.RELEASE.jar:5.0.9.RELEASE]
    	at org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(ConfigurationClassPostProcessor.java:328) ~[spring-context-5.0.9.RELEASE.jar:5.0.9.RELEASE]
    	at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(ConfigurationClassPostProcessor.java:233) ~[spring-context-5.0.9.RELEASE.jar:5.0.9.RELEASE]
    	at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors(PostProcessorRegistrationDelegate.java:271) ~[spring-context-5.0.9.RELEASE.jar:5.0.9.RELEASE]
    	at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:91) ~[spring-context-5.0.9.RELEASE.jar:5.0.9.RELEASE]
    	at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:694) ~[spring-context-5.0.9.RELEASE.jar:5.0.9.RELEASE]
    	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:532) ~[spring-context-5.0.9.RELEASE.jar:5.0.9.RELEASE]
    	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:140) ~[spring-boot-2.0.5.RELEASE.jar:2.0.5.RELEASE]
    	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:780) [spring-boot-2.0.5.RELEASE.jar:2.0.5.RELEASE]
    	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:412) [spring-boot-2.0.5.RELEASE.jar:2.0.5.RELEASE]
    	at org.springframework.boot.SpringApplication.run(SpringApplication.java:333) [spring-boot-2.0.5.RELEASE.jar:2.0.5.RELEASE]
    	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1277) [spring-boot-2.0.5.RELEASE.jar:2.0.5.RELEASE]
    	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1265) [spring-boot-2.0.5.RELEASE.jar:2.0.5.RELEASE]
    	at com.alibaba.csp.sentinel.dashboard.DashboardApplication.main(DashboardApplication.java:33) [classes/:na]
    Caused by: java.lang.IllegalStateException: Failed to introspect Class [com.alibaba.csp.sentinel.dashboard.rule.zookeeper.ZookeeperConfig] from ClassLoader [sun.misc.Launcher$AppClassLoader@2a139a55]
    	at org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:659) ~[spring-core-5.0.9.RELEASE.jar:5.0.9.RELEASE]
    	at org.springframework.util.ReflectionUtils.doWithMethods(ReflectionUtils.java:556) ~[spring-core-5.0.9.RELEASE.jar:5.0.9.RELEASE]
    	at org.springframework.util.ReflectionUtils.doWithMethods(ReflectionUtils.java:541) ~[spring-core-5.0.9.RELEASE.jar:5.0.9.RELEASE]
    	at org.springframework.util.ReflectionUtils.getUniqueDeclaredMethods(ReflectionUtils.java:599) ~[spring-core-5.0.9.RELEASE.jar:5.0.9.RELEASE]
    	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.getTypeForFactoryMethod(AbstractAutowireCapableBeanFactory.java:718) ~[spring-beans-5.0.9.RELEASE.jar:5.0.9.RELEASE]
    	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.determineTargetType(AbstractAutowireCapableBeanFactory.java:659) ~[spring-beans-5.0.9.RELEASE.jar:5.0.9.RELEASE]
    	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.predictBeanType(AbstractAutowireCapableBeanFactory.java:627) ~[spring-beans-5.0.9.RELEASE.jar:5.0.9.RELEASE]
    	at org.springframework.beans.factory.support.AbstractBeanFactory.isFactoryBean(AbstractBeanFactory.java:1489) ~[spring-beans-5.0.9.RELEASE.jar:5.0.9.RELEASE]
    	at org.springframework.beans.factory.support.AbstractBeanFactory.isFactoryBean(AbstractBeanFactory.java:1012) ~[spring-beans-5.0.9.RELEASE.jar:5.0.9.RELEASE]
    	at org.springframework.boot.autoconfigure.condition.BeanTypeRegistry.addBeanTypeForNonAliasDefinition(BeanTypeRegistry.java:194) ~[spring-boot-autoconfigure-2.0.5.RELEASE.jar:2.0.5.RELEASE]
    	at org.springframework.boot.autoconfigure.condition.BeanTypeRegistry.addBeanTypeForNonAliasDefinition(BeanTypeRegistry.java:170) ~[spring-boot-autoconfigure-2.0.5.RELEASE.jar:2.0.5.RELEASE]
    	at org.springframework.boot.autoconfigure.condition.BeanTypeRegistry.addBeanType(BeanTypeRegistry.java:163) ~[spring-boot-autoconfigure-2.0.5.RELEASE.jar:2.0.5.RELEASE]
    	at org.springframework.boot.autoconfigure.condition.BeanTypeRegistry.lambda$updateTypesIfNecessary$2(BeanTypeRegistry.java:150) ~[spring-boot-autoconfigure-2.0.5.RELEASE.jar:2.0.5.RELEASE]
    	at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[na:1.8.0_91]
    	at org.springframework.boot.autoconfigure.condition.BeanTypeRegistry.updateTypesIfNecessary(BeanTypeRegistry.java:148) ~[spring-boot-autoconfigure-2.0.5.RELEASE.jar:2.0.5.RELEASE]
    	at org.springframework.boot.autoconfigure.condition.BeanTypeRegistry.getNamesForType(BeanTypeRegistry.java:114) ~[spring-boot-autoconfigure-2.0.5.RELEASE.jar:2.0.5.RELEASE]
    	at org.springframework.boot.autoconfigure.condition.OnBeanCondition.collectBeanNamesForType(OnBeanCondition.java:265) ~[spring-boot-autoconfigure-2.0.5.RELEASE.jar:2.0.5.RELEASE]
    	at org.springframework.boot.autoconfigure.condition.OnBeanCondition.getBeanNamesForType(OnBeanCondition.java:254) ~[spring-boot-autoconfigure-2.0.5.RELEASE.jar:2.0.5.RELEASE]
    	at org.springframework.boot.autoconfigure.condition.OnBeanCondition.getMatchingBeans(OnBeanCondition.java:196) ~[spring-boot-autoconfigure-2.0.5.RELEASE.jar:2.0.5.RELEASE]
    	at org.springframework.boot.autoconfigure.condition.OnBeanCondition.getMatchOutcome(OnBeanCondition.java:116) ~[spring-boot-autoconfigure-2.0.5.RELEASE.jar:2.0.5.RELEASE]
    	at org.springframework.boot.autoconfigure.condition.SpringBootCondition.matches(SpringBootCondition.java:47) ~[spring-boot-autoconfigure-2.0.5.RELEASE.jar:2.0.5.RELEASE]
    	... 17 common frames omitted
    Caused by: java.lang.NoClassDefFoundError: org/apache/curator/RetryPolicy
    	at java.lang.Class.getDeclaredMethods0(Native Method) ~[na:1.8.0_91]
    	at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) ~[na:1.8.0_91]
    	at java.lang.Class.getDeclaredMethods(Class.java:1975) ~[na:1.8.0_91]
    	at org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:641) ~[spring-core-5.0.9.RELEASE.jar:5.0.9.RELEASE]
    	... 37 common frames omitted
    Caused by: java.lang.ClassNotFoundException: org.apache.curator.RetryPolicy
    	at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[na:1.8.0_91]
    	at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_91]
    	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) ~[na:1.8.0_91]
    	at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_91]
    	... 41 common frames omitted

后来发现是zookeeper的依赖包中有<scope>test</scope>这个选项，导致依赖包只能在跑test测试程序的时候才生效，正式程序中无效，要注释掉它。

**2、/v2/flow 浏览器缓存**

在改造成功之前，在浏览器中访问过了很多次[http://localhost:9352][http_localhost_9352]，进入到“流控规则”界面时，url本来应该是这样的

![20191102151659101.png][]

![20191102151714374.png][]

但是实际上我每次进入这个界面时，url是这样的：

![20191102151746291.png][]

其中少了一个/v2，我还一直以为是我的配置有问题，找了很久的原因，后来换了一个浏览器就好了，才知道是缓存导致的。看来以后用到浏览器测试的时候，一定要多想想缓存的问题。

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3_size_16_color_FFFFFF_t_70]: /images/20221119/71f3739f9b274640b4dcc24564e9811c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20191102143717500.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20191102143805385.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/2019110214414837.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3,size_16,color_FFFFFF,t_70
[20191102144403540.png]: https://img-blog.csdnimg.cn/20191102144403540.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20191102144506862.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3,size_16,color_FFFFFF,t_70
[http_localhost_9352]: http://localhost:9352/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20191102144640377.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/20191102144939917.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3,size_16,color_FFFFFF,t_70
[20191102145150934.png]: https://img-blog.csdnimg.cn/20191102145150934.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3_size_16_color_FFFFFF_t_70 7]: https://img-blog.csdnimg.cn/20191102145448353.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3_size_16_color_FFFFFF_t_70 8]: https://img-blog.csdnimg.cn/20191102150419278.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3_size_16_color_FFFFFF_t_70 9]: https://img-blog.csdnimg.cn/20191102150544279.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3_size_16_color_FFFFFF_t_70 10]: https://img-blog.csdnimg.cn/20191102150628595.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3,size_16,color_FFFFFF,t_70
[20191102151439429.png]: https://img-blog.csdnimg.cn/20191102151439429.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3_size_16_color_FFFFFF_t_70 11]: https://img-blog.csdnimg.cn/20191102151453952.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTIzNTg3,size_16,color_FFFFFF,t_70
[20191102151659101.png]: https://img-blog.csdnimg.cn/20191102151659101.png
[20191102151714374.png]: https://img-blog.csdnimg.cn/20191102151714374.png
[20191102151746291.png]: https://img-blog.csdnimg.cn/20191102151746291.png