Java: Hystrix And ThreadLocals

转 https://medium.com/@saurav24081996/java-hystrix-and-threadlocals-95ea9e194e83

ThreadLocal is a java class which provide thread-local variables, i.e. variables specific to thread. Popularly used a lot for storing request context, for example requestId (in MDC for error reporting to sentry), or storing New Relic token for monitoring.

Hystrix

For those who don’t know much about hystrix, it is open source library with circuit breaker functionality. You can read more about it here.

Those of us who are used to distributed systems (for example Aggregation layer, or micro-service architecture) know that it is very much possible for a down stream to go down, or respond at higher latency than expected.

So how to prevent our service from deviating from its latency. This is where need for a circuit breaker pattern arises. It not only saves our service from becoming unavailable, also it gives the downstream services some breathing space.

Hystrix has two strategies, Semaphore and Thread. In this blog we focus mainly on Thread Strategy. So first of all lets discuss working of hystrix in short. So hystrix has 2 functions

• run() : contains primary execution code
• getFallback() : contains code to be run in case of fallback. There can be three reasons when this method is triggered, listed below.

In Thread strategy, hystrix runs the primary execution in different thread. Lets name that thread hystrix-command-*. Now there is a timer thread (named as hystrix-timer-*) this thread waits for t ms (specified timeout) for run() to execute. The thread on which fallback is executed depends on the reason due to which fallback was triggered

 

Now that we have discussed about how hystrix works with its thread, the problem that comes to mind is that how do we make sure that the threadLocal variables are transferred safely to these threads.

First thing that comes to mind is using InheritableThreadLocal(). Now I will give a short advice about its usage, DO NOT USE InheritableThreadLocal!!!

Why???? Because hystrix internally maintains thread pool, which allows thread to be reused. So the thread locals inherited first time in the hystrix thread will be there for all other calling threads, no matter what.

So now what?

Well hystrix developers had already thought about this problem. Hence they provided several plugins to tackle this. I will discuss two approach for this, the first one is a short one, with a little downsides, however the second one is best way for doing it.

Using HystrixConcurrencyStrategy

This plugin can simply be used to wrap the callable for hystrix run().

 

public class ThreadLocalUtil {
    private static ThreadLocal<Integer> idTL = new ThreadLocal<>();

    public static void setId(Integer id) {
        idTL.set(id);
    }

    public static void getId() {
        idTL.get();
    }

    public static void clear() {
        idTL.remove();
    }
}

public class ConcurrencyStrategy extends HystrixConcurrencyStrategy {
    @Override
    public <T> Callable<T> wrapCallable(Callable<T> callable) {
        Integer id = ThreadLocalUtil.getId();
        try {
            return () -> {
                ThreadLocalUtil.setId(id);
                return callable.call();
            };
        } finally {
            // Remember to clear thee threadLocals
            ThreadLocalUtil.clear()
        }
    }
}

// Register the concurrency strategy
HystrixPlugins.getInstance().registerConcurrencyStrategy(new ConcurrencyStrategy());

This works in a similar way like TaskDecorator. We extract the threadLocal id from caller thread and set it into the new thread invoked by hystrix.

So if this is so easy then whats the problem?

The issue with HystrixConcurrencyStrategy is that it wraps only the run() function. In simpler words if you use HystrixConcurrencyStrategy, ThreadLocal variables won’t be available in fallback methods thread(in case the thread for fallback is different, i.e. timeout).

If you don’t need threadLocal variables in fallback, then you are good to go. But If you need it there, then you need to use HystrixCommandExecutionHook.

Using HystrixCommandExecutionHook

HystrixCommandExecutionHook allows you to override several functions like onExecutionStart, onFallbackStart, etc. This gives you full control over the hystrix execution flow. I will now describe briefly some of the functions we need. If you want more about them, you can read it here.

• onStart(): executes when the hystrixCommand is called. Runs on the caller thread itself
• onSuccess(): executes if hystrixCommand.execute() is successful, i.e. either run() or getFallback() function have been successful. Runs in caller thread
• onError(): executes when hystrixCommand.execute() fails, i.e. both run() and getFallback() functions have failed. Runs in caller thread
• onExecutionStart(): executes just before run() code. Runs on the hystrix-command-thread
• onExecutionSuccess(): executes whenever the run() method executes successfully (without error or timeout). Runs on the hystrix-command-thread
• onExecutionError(): executes when run() method fails (timeout or any exception). Runs on the hystrix-command-thread
• onFallbackStart(): executes when fallback starts. Runs on the hystrix-command-thread OR hystrix-timer-* or caller thread.
• onFallbackSuccess(): executes when fallback ends successfully. Runs on the hystrix-command-thread OR hystrix-timer-* or caller thread.
• onFallbackError(): executes when fallback ends with any exception. Runs on the hystrix-command-thread OR hystrix-timer-* or caller thread.

So now that we have knowledge about full flow, we just need to pull out the threadLocals from calling thread and set it in any of the subsequent thread.

// Buggy code
public class HystrixHook extends HystrixCommandExecutionHook {
    
    private Integer id;

    @Override
    public <T> void onStart(HystrixInvokable<T> commandInstance) {
        getThreadLocals();
    }

    @Override
    public <T> void onExecutionStart(HystrixInvokable<T> commandInstance) {
        setThreadLocals();
    }

    @Override
    public <T> void onFallbackStart(HystrixInvokable<T> commandInstance) {
        setThreadLocals();
    }
    
    @Override
    public <T> void onExecutionSuccess(HystrixInvokable<T> commandInstance) {
        ThreadLocalUtil.clear();
        super.onExecutionSuccess(commandInstance);
    }

    @Override
    public <T> Exception onExecutionError(HystrixInvokable<T> commandInstance, Exception e) {
        ThreadLocalUtil.clear();
        return super.onExecutionError(commandInstance, e);
    }
    
     @Override
    public <T> Exception onFallbackError(HystrixInvokable<T> commandInstance, Exception e) {
        ThreadLocalUtil.clear();
        return super.onFallbackError(commandInstance, e);
    }

    @Override
    public <T> void onFallbackSuccess(HystrixInvokable<T> commandInstance) {
        ThreadLocalUtil.clear();
        super.onFallbackSuccess(commandInstance);
    }
    
     private void getThreadLocals() {
        id = ThreadLocalUtil.getId();
    }

    private void setThreadLocals() {
        ThreadLocalUtil.setId(id);
    }
}

// Register hystrix hook plugin
HystrixPlugins.getInstance().registerCommandExecutionHook(new HystrixHook());

 

Simple isn’t? Well nice guess NO! :D There are several issues with the about code.

• Same HystrixHook object is shared across all the threads. So the value set to idTL can be changed if any other threads executes the hystrixCommand at the same time.
• As I mentioned before the fallback can run in calling thread as well, so onFallbackError()/onFallbackSuccess() methods can run in the caller thread, and hence the threadLocal variables will be cleared from the calling thread which is not required.

Final Solution

So what now? Don’t worry this is the final solution, promise ;)

Same HystrixHook object is shared across all the threads.

HystrixRequestVariableDefault is a kind of threadLocal, provided by hystrix for such usage. But unlike threadLocals, it has an scope to a hystrix request. . Since I will skip details about it, so you can read more about it here.

So this is how the final code for HystrixHook will look like

public class HystrixHook extends HystrixCommandExecutionHook {

    private HystrixRequestVariableDefault<Integer> hrv = new HystrixRequestVariableDefault<>();

    @Override
    public <T> void onStart(HystrixInvokable<T> commandInstance) {
        HystrixRequestContext.initializeContext();
        getThreadLocals();
    }

    @Override
    public <T> void onExecutionStart(HystrixInvokable<T> commandInstance) {
        setThreadLocals();
    }


    @Override
    public <T> void onFallbackStart(HystrixInvokable<T> commandInstance) {
        setThreadLocals();
    }
  
  
    @Override
    public <T> void onSuccess(HystrixInvokable<T> commandInstance) {
        HystrixRequestContext.getContextForCurrentThread().shutdown();
        super.onSuccess(commandInstance);
    }

    @Override
    public <T> Exception onError(HystrixInvokable<T> commandInstance, HystrixRuntimeException.FailureType failureType, Exception e) {
        HystrixRequestContext.getContextForCurrentThread().shutdown();
        return super.onError(commandInstance, failureType, e);
    }
  
    private void getThreadLocals() {
        hrv.set(ThreadLocalUtil.getId());
    }

    private void setThreadLocals() {
        ThreadLocalUtil.setId(hrv.get());
    }
}

// Register hystrix hook plugin
HystrixPlugins.getInstance().registerCommandExecutionHook(new HystrixHook());