熔断超时不生效及动态加载配置不生效问题

àì夳堔傛蜴生んèń 2022-10-12 04:13 373阅读 0赞

1、问题描述

在本地环境测试调用南向服务超时会记录失败次数，达到阈值熔断，服务器上却不生效。

2、定位过程

经过远程debug，发现是由于在服务器上的executionTimeoutEnabled配置是false导致的，感觉很奇怪，因为默认值配置是true，也没有指定为false，只能在代码里传入这个配置项，置为true，具体代码跟踪流程如下：

Hystrix是在com.netflix.hystrix.AbstractCommand\#executeCommandAndObserve方法中进行是否开启超时熔断的判断的

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Rvbmd5dXh1MzQyNzE5_size_16_color_FFFFFF_t_70][]

而这个executionTimeoutEnabled配置是从properties属性中取的，可以看到这个属性的值来源于com.netflix.hystrix.HystrixCommandProperties\#executionTimeoutEnabled。

com.netflix.hystrix.HystrixCommandProperties\#executionTimeoutEnabled又是在构造HystrixCommandProperties对象时通过构造函数初始化的

this.executionTimeoutEnabled = getProperty(propertyPrefix, key, "execution.timeout.enabled", builder.getExecutionTimeoutEnabled(), default\_executionTimeoutEnabled);

// known that we're using deprecated HystrixPropertiesChainedServoProperty until ChainedDynamicProperty exists in Archaius
        protected HystrixCommandProperties(HystrixCommandKey key, HystrixCommandProperties.Setter builder, String propertyPrefix) {
            this.key = key;
            this.circuitBreakerEnabled = getProperty(propertyPrefix, key, "circuitBreaker.enabled", builder.getCircuitBreakerEnabled(), default_circuitBreakerEnabled);
            this.circuitBreakerRequestVolumeThreshold = getProperty(propertyPrefix, key, "circuitBreaker.requestVolumeThreshold", builder.getCircuitBreakerRequestVolumeThreshold(), default_circuitBreakerRequestVolumeThreshold);
            this.circuitBreakerSleepWindowInMilliseconds = getProperty(propertyPrefix, key, "circuitBreaker.sleepWindowInMilliseconds", builder.getCircuitBreakerSleepWindowInMilliseconds(), default_circuitBreakerSleepWindowInMilliseconds);
            this.circuitBreakerErrorThresholdPercentage = getProperty(propertyPrefix, key, "circuitBreaker.errorThresholdPercentage", builder.getCircuitBreakerErrorThresholdPercentage(), default_circuitBreakerErrorThresholdPercentage);
            this.circuitBreakerForceOpen = getProperty(propertyPrefix, key, "circuitBreaker.forceOpen", builder.getCircuitBreakerForceOpen(), default_circuitBreakerForceOpen);
            this.circuitBreakerForceClosed = getProperty(propertyPrefix, key, "circuitBreaker.forceClosed", builder.getCircuitBreakerForceClosed(), default_circuitBreakerForceClosed);
            this.executionIsolationStrategy = getProperty(propertyPrefix, key, "execution.isolation.strategy", builder.getExecutionIsolationStrategy(), default_executionIsolationStrategy);
            //this property name is now misleading.  //TODO figure out a good way to deprecate this property name
            this.executionTimeoutInMilliseconds = getProperty(propertyPrefix, key, "execution.isolation.thread.timeoutInMilliseconds", builder.getExecutionIsolationThreadTimeoutInMilliseconds(), default_executionTimeoutInMilliseconds);
            this.executionTimeoutEnabled = getProperty(propertyPrefix, key, "execution.timeout.enabled", builder.getExecutionTimeoutEnabled(), default_executionTimeoutEnabled);

因此就能知道，这个属性的赋值是由我们代码中通过继承com.netflix.hystrix.HystrixObservableCommand类，传入com.netflix.hystrix.HystrixCommandProperties.Setter对象初始化的

protected AbstractCommand(HystrixCommandGroupKey group, HystrixCommandKey key, HystrixThreadPoolKey threadPoolKey, HystrixCircuitBreaker circuitBreaker, HystrixThreadPool threadPool,
                HystrixCommandProperties.Setter commandPropertiesDefaults, HystrixThreadPoolProperties.Setter threadPoolPropertiesDefaults,
                HystrixCommandMetrics metrics, TryableSemaphore fallbackSemaphore, TryableSemaphore executionSemaphore,
                HystrixPropertiesStrategy propertiesStrategy, HystrixCommandExecutionHook executionHook) {
    
            this.commandGroup = initGroupKey(group);
            this.commandKey = initCommandKey(key, getClass());
            this.properties = initCommandProperties(this.commandKey, propertiesStrategy, commandPropertiesDefaults);

所以在代码中通过setter.withExecutionTimeoutEnabled(true);来打开超时熔断的开关。

**Hystrix判断超时的逻辑**

Hystrix是通过com.netflix.hystrix.AbstractCommand.HystrixObservableTimeoutOperator\#call方法中的com.netflix.hystrix.util.HystrixTimer.TimerListener对象实现的，这个对象是通过ScheduledExecutor启动一个定时器java.util.concurrent.ScheduledThreadPoolExecutor\#scheduleAtFixedRate，通过定时器判断调用服务是否超时。其中传入的超时时间是由下面的代码获取的。

![20210703095316790.png][]

跟踪properties属性的赋值是由com.netflix.hystrix.strategy.properties.HystrixPropertiesFactory\#getCommandProperties方法得到

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Rvbmd5dXh1MzQyNzE5_size_16_color_FFFFFF_t_70 1][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Rvbmd5dXh1MzQyNzE5_size_16_color_FFFFFF_t_70 2][]

com.netflix.hystrix.strategy.properties.HystrixPropertiesFactory\#getCommandProperties方法中会先从缓存取，如果缓存取不到才会根据传入的com.netflix.hystrix.HystrixCommandProperties.Setter对象赋值

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Rvbmd5dXh1MzQyNzE5_size_16_color_FFFFFF_t_70 3][]

这就涉及到另一个问题了，就是为什么创建的HystrixCommand属性没有生效的问题，因为优先使用了缓存的值，官方提供的动态更新配置的方法就是使用archaius来动态加载。

官方提供的动态更新hystrix配置的示例：[https://github.com/Netflix/archaius/wiki/Users-Guide][https_github.com_Netflix_archaius_wiki_Users-Guide]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Rvbmd5dXh1MzQyNzE5_size_16_color_FFFFFF_t_70]: /images/20221005/5b4fc92c24374d5bb28b2749c4a1ee6d.png
[20210703095316790.png]: /images/20221005/df79798f97e54477a8edfcd35e5668c7.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Rvbmd5dXh1MzQyNzE5_size_16_color_FFFFFF_t_70 1]: /images/20221005/6bc9f0d75c4548ef8b8528de9679cca3.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Rvbmd5dXh1MzQyNzE5_size_16_color_FFFFFF_t_70 2]: /images/20221005/a5b3138e4941400aadf571a162fa7d8d.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Rvbmd5dXh1MzQyNzE5_size_16_color_FFFFFF_t_70 3]: /images/20221005/b4a18add1277475da31a57b68f64b23a.png
[https_github.com_Netflix_archaius_wiki_Users-Guide]: https://github.com/Netflix/archaius/wiki/Users-Guide