记一次Hystrix报错HystrixRuntimeException: HystrixCommandKey fallback execution rejected.

痛定思痛。 2022-12-22 11:27 362阅读 0赞

定位代码  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0VsbGVuX1Rhbmd4aWFuZw_size_16_color_FFFFFF_t_70_pic_center]

/com/netflix/hystrix/AbstractCommand.java:851

if (fallbackSemaphore.tryAcquire()) { 
                        try { 
                            if (isFallbackUserDefined()) { 
                                executionHook.onFallbackStart(this);
                                fallbackExecutionChain = getFallbackObservable();
                            } else { 
                                //same logic as above without the hook invocation
                                fallbackExecutionChain = getFallbackObservable();
                            }
                        } catch (Throwable ex) { 
                            //If hook or user-fallback throws, then use that as the result of the fallback lookup
                            fallbackExecutionChain = Observable.error(ex);
                        }
    
                        return fallbackExecutionChain
                                .doOnEach(setRequestContext)
                                .lift(new FallbackHookApplication(_cmd))
                                .lift(new DeprecatedOnFallbackHookApplication(_cmd))
                                .doOnNext(markFallbackEmit)
                                .doOnCompleted(markFallbackCompleted)
                                .onErrorResumeNext(handleFallbackError)
                                .doOnTerminate(singleSemaphoreRelease)
                                .doOnUnsubscribe(singleSemaphoreRelease);
                    } else { 
                       return handleFallbackRejectionByEmittingError();
                    }

fallbackSemaphore.tryAcquire()  
如果是信号量隔离那么走  
com.netflix.hystrix.AbstractCommand.TryableSemaphoreActual

@Override
            public boolean tryAcquire() { 
                int currentCount = count.incrementAndGet();
                if (currentCount > numberOfPermits.get()) { 
                    count.decrementAndGet();
                    return false;
                } else { 
                    return true;
                }
            }

如果数量大于允许数量那么会返回false降级失败  
也就是降级也有信号量的限制

数量初始化

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0VsbGVuX1Rhbmd4aWFuZw_size_16_color_FFFFFF_t_70_pic_center 1]

通过构造函数进行设置

public TryableSemaphoreActual(HystrixProperty<Integer> numberOfPermits) { 
                this.numberOfPermits = numberOfPermits;
            }

fallbackSemaphorePerCircuit.putIfAbsent(commandKey.name(), new TryableSemaphoreActual(properties.fallbackIsolationSemaphoreMaxConcurrentRequests()));

数量是由HystrixCommandProperties的参数fallbackIsolationSemaphoreMaxConcurrentRequests决定

protected TryableSemaphore getFallbackSemaphore() { 
            if (fallbackSemaphoreOverride == null) { 
                TryableSemaphore _s = fallbackSemaphorePerCircuit.get(commandKey.name());
                if (_s == null) { 
                    // we didn't find one cache so setup
                    fallbackSemaphorePerCircuit.putIfAbsent(commandKey.name(), new TryableSemaphoreActual(properties.fallbackIsolationSemaphoreMaxConcurrentRequests()));
                    // assign whatever got set (this or another thread)
                    return fallbackSemaphorePerCircuit.get(commandKey.name());
                } else { 
                    return _s;
                }
            } else { 
                return fallbackSemaphoreOverride;
            }
        }

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0VsbGVuX1Rhbmd4aWFuZw_size_16_color_FFFFFF_t_70_pic_center 2]

/com/netflix/hystrix/HystrixCommandProperties.java:128

this.fallbackIsolationSemaphoreMaxConcurrentRequests = getProperty(propertyPrefix, key, "fallback.isolation.semaphore.maxConcurrentRequests", builder.getFallbackIsolationSemaphoreMaxConcurrentRequests(), default_fallbackIsolationSemaphoreMaxConcurrentRequests);

如果不设置失败信号量，那么失败信号量的数量为default\_fallbackIsolationSemaphoreMaxConcurrentRequests

private static final Integer default\_fallbackIsolationSemaphoreMaxConcurrentRequests = 10;

可以通过withFallbackIsolationSemaphoreMaxConcurrentRequests方法对参数进行设置

HystrixCommandProperties
                            .Setter()
                            .withExecutionIsolationStrategy(HystrixCommandProperties.ExecutionIsolationStrategy.SEMAPHORE)
                            .withExecutionIsolationSemaphoreMaxConcurrentRequests(1)
                            .withFallbackIsolationSemaphoreMaxConcurrentRequests(100)

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0VsbGVuX1Rhbmd4aWFuZw_size_16_color_FFFFFF_t_70_pic_center]: /images/20221120/a7148706344a4eb78cf6a2edcee47fea.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0VsbGVuX1Rhbmd4aWFuZw_size_16_color_FFFFFF_t_70_pic_center 1]: /images/20221120/7b8ac2bbae68411a8099b3e1d8fb6080.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0VsbGVuX1Rhbmd4aWFuZw_size_16_color_FFFFFF_t_70_pic_center 2]: /images/20221120/2e572019ada744cebc8444575d1517f8.png