目录
2、NacosExecuteTaskExecuteEngine——nacos立即执行引擎
2.3、NacosExecuteTaskExecuteEngine
2.5、NacosExecuteTaskExecuteEngine总结
3、NacosDelayTaskExecuteEngine——nacos延迟任务执行引擎
3.2、NacosDelayTaskExecuteEngine
3.4、NacosDelayTaskExecuteEngine总结
1、任务执行引擎的结构
源码中相关包结构如下图所示:
直接看源码包中的类关系不够清晰,我们可以利用idea查看UML类图
如下图所示:
分析上图可知,AbstractNacosTaskExecuteEngine聚合了NacosTaskProcessor,其中有两条线表示聚合关系,一条是一对多,还有一条是一对一的关系。
下面我们来查看AbstractNacosTaskExecuteEngine源码
在 AbstractNacosTaskExecuteEngine中有两个相关成员变量,由名字可以得出,taskProcessors是任务处理器的Map集合,defaultTaskProcessor是默认的任务处理器。而NacosExecuteTaskExecuteEngine和NacosDelayTaskExecuteEngine继承了AbstractNacosTaskExecuteEngine,表示也继承了其这两个属性。
2、NacosExecuteTaskExecuteEngine——nacos立即执行引擎
NacosExecuteTaskExecuteEngine只会执行继承了AbstractExecuteTask的任务。下面我们来看看AbstractExecuteTask。
2.1、AbstractExecuteTask
/**
* Abstract task which should be executed immediately.
* 立即执行的抽象任务
*
* @author xiweng.yy
*/
public abstract class AbstractExecuteTask implements NacosTask, Runnable {
protected static final long INTERVAL = 3000L;
@Override
public boolean shouldProcess() {
return true;
}
}
AbstractExecuteTask实现了Runnable,实现AbstractExecuteTask则必须实现run方法,执行任务可以调用run执行,一般是由TaskExcuteWorker来执行的。
2.2、TaskExecuteWorker
TaskExcuteWorker实际上实现了NacosTaskProcessor,是一个任务处理器,在NacosExecuteTaskExecuteEngine中组合了TaskExcuteWorker。如下图示例
下面我们来分析一下TaskExecuteWorker的源码
/**
* Nacos execute task execute worker.
* Nacos执行任务执行工作者
*
* @author xiweng.yy
*/
public final class TaskExecuteWorker implements NacosTaskProcessor, Closeable {
/**
* Max task queue size 32768.
* 最大的队列长度,也是默认长度
*/
private static final int QUEUE_CAPACITY = 1 << 15;
private final Logger log;
private final String name;
private final BlockingQueue<Runnable> queue;
private final AtomicBoolean closed;
public TaskExecuteWorker(final String name, final int mod, final int total) {
this(name, mod, total, null);
}
public TaskExecuteWorker(final String name, final int mod, final int total, final Logger logger) {
this.name = name + "_" + mod + "%" + total;
this.queue = new ArrayBlockingQueue<Runnable>(QUEUE_CAPACITY);
// 默认创建就运行
this.closed = new AtomicBoolean(false);
this.log = null == logger ? LoggerFactory.getLogger(TaskExecuteWorker.class) : logger;
// 实际上的执行者是内部的InnerWorker
new InnerWorker(name).start();
}
public String getName() {
return name;
}
@Override
public boolean process(NacosTask task) {
if (task instanceof AbstractExecuteTask) {
putTask((Runnable) task);
}
return true;
}
/**
* 添加任务到队列
* @param task 任务
*/
private void putTask(Runnable task) {
try {
queue.put(task);
} catch (InterruptedException ire) {
log.error(ire.toString(), ire);
}
}
/**
* 获取队列长度
*/
public int pendingTaskCount() {
return queue.size();
}
/**
* Worker status.
*/
public String status() {
return name + ", pending tasks: " + pendingTaskCount();
}
@Override
public void shutdown() throws NacosException {
queue.clear();
closed.compareAndSet(false, true);
}
/**
* Inner execute worker.
*/
private class InnerWorker extends Thread {
InnerWorker(String name) {
setDaemon(false);
setName(name);
}
@Override
public void run() {
while (!closed.get()) {
try {
// 取出task
Runnable task = queue.take();
long begin = System.currentTimeMillis();
// 运行task
task.run();
// 计算task执行时间
long duration = System.currentTimeMillis() - begin;
if (duration > 1000L) {
log.warn("task {} takes {}ms", task, duration);
}
} catch (Throwable e) {
log.error("[TASK-FAILED] " + e.toString(), e);
}
}
}
}
}
在TaskExecuteWorker被创建的时候,默认会去执行它里面阻塞队列中的任务的。下面我们看看
NacosExecuteTaskExecuteEngine源码。
2.3、NacosExecuteTaskExecuteEngine
/**
* Nacos execute task execute engine.
* Nacos执行任务执行引擎
*
* @author xiweng.yy
*/
public class NacosExecuteTaskExecuteEngine extends AbstractNacosTaskExecuteEngine<AbstractExecuteTask> {
private final TaskExecuteWorker[] executeWorkers;
public NacosExecuteTaskExecuteEngine(String name, Logger logger) {
this(name, logger, ThreadUtils.getSuitableThreadCount(1));
}
public NacosExecuteTaskExecuteEngine(String name, Logger logger, int dispatchWorkerCount) {
super(logger);
executeWorkers = new TaskExecuteWorker[dispatchWorkerCount];
for (int mod = 0; mod < dispatchWorkerCount; ++mod) {
executeWorkers[mod] = new TaskExecuteWorker(name, mod, dispatchWorkerCount, getEngineLog());
}
}
@Override
public int size() {
int result = 0;
for (TaskExecuteWorker each : executeWorkers) {
result += each.pendingTaskCount();
}
return result;
}
@Override
public boolean isEmpty() {
return 0 == size();
}
@Override
public void addTask(Object tag, AbstractExecuteTask task) {
// 如果没有添加过对应标签的任务执行器,则使用默认的任务执行器
NacosTaskProcessor processor = getProcessor(tag);
if (null != processor) {
processor.process(task);
return;
}
// 如果该key没有对应的处理器并且没有设置默认处理器,则使用TaskExecuteWorker来处理
TaskExecuteWorker worker = getWorker(tag);
worker.process(task);
}
private TaskExecuteWorker getWorker(Object tag) {
int idx = (tag.hashCode() & Integer.MAX_VALUE) % workersCount();
return executeWorkers[idx];
}
private int workersCount() {
return executeWorkers.length;
}
@Override
public AbstractExecuteTask removeTask(Object key) {
throw new UnsupportedOperationException("ExecuteTaskEngine do not support remove task");
}
@Override
public Collection<Object> getAllTaskKeys() {
throw new UnsupportedOperationException("ExecuteTaskEngine do not support get all task keys");
}
@Override
public void shutdown() throws NacosException {
for (TaskExecuteWorker each : executeWorkers) {
each.shutdown();
}
}
/**
* Get workers status.
*
* @return workers status string
*/
public String workersStatus() {
StringBuilder sb = new StringBuilder();
for (TaskExecuteWorker worker : executeWorkers) {
sb.append(worker.status()).append('\n');
}
return sb.toString();
}
}
由上分析可以得出,NacosExecuteTaskExecuteEngine在创建的时候也会创建出多个TaskExecuteWorker,在向NacosExecuteTaskExecuteEngine添加执行任务的时候(addTask方法),如果存在有tag对应的处理器,则获取并执行处理器的processor方法,如果没有tag对应的任务处理器,则使用默认的任务处理器,如果没设置默认处理器,则使用TaskExecuteWorker来处理。这里可以自定义NacosTaskProcessor,可以用来处理对应的任务,因为TaskExecuteWorker里面是一个阻塞队列,如果有一个需要立即执行的任务,在任务量大的时候TaskExecuteWorker是无法立即执行的,所以这个时候可以自定义默认的NacosTaskProcessor来处理。
2.4、使用案例
public class MyExecuteTask extends AbstractExecuteTask {
@Override
public void run() {
System.out.println("MyExecuteTask...");
}
}
public class MyNacosTaskProcessor implements NacosTaskProcessor {
@Override
public boolean process(NacosTask task) {
System.out.println("MyNacosTaskProcessor start...");
Runnable runnable = (Runnable) task;
runnable.run();
System.out.println("MyNacosTaskProcessor end...");
return false;
}
}
@RunWith(MockitoJUnitRunner.class)
public class MyNacosExecuteTaskExecuteEngineTest {
private NacosExecuteTaskExecuteEngine executeTaskExecuteEngine;
@Before
public void setUp() {
executeTaskExecuteEngine = new NacosExecuteTaskExecuteEngine("TaskExecuteEngine", null);
}
@After
public void tearDown() throws NacosException {
executeTaskExecuteEngine.shutdown();
}
@Test
public void testAddTask() throws InterruptedException {
// 直接添加
executeTaskExecuteEngine.addTask("myTest", new MyExecuteTask());
// 用对应的处理器执行
executeTaskExecuteEngine.addProcessor(MyExecuteTask.class, new MyNacosTaskProcessor());
executeTaskExecuteEngine.addTask(MyExecuteTask.class, new MyExecuteTask());
Thread.sleep(1000);
}
}
最后输出的结果如下
2.5、NacosExecuteTaskExecuteEngine总结
该执行引擎实际上是一个立即执行任务的引擎,在创建引擎的时候会创建TaskExecuteWorker,该引擎中只能添加执行继承了AbstractExecuteTask的任务,并且可以指定自定义的处理器对任务进行处理,如果不指定,则是默认使用TaskExecuteWorker处理。
3、NacosDelayTaskExecuteEngine——nacos延迟任务执行引擎
NacosDelayTaskExecuteEngine只会执行继承了AbstractDelayTask的任务。下面我们来看看AbstractDelayTask。
3.1、AbstractDelayTask
/**
* Abstract task which can delay and merge.
* 可以延迟、合并的抽象任务
*
* @author huali
* @author xiweng.yy
*/
public abstract class AbstractDelayTask implements NacosTask {
/**
* Task time interval between twice processing, unit is millisecond.
* 任务处理的时间间隔
*/
private long taskInterval;
/**
* The time which was processed at last time, unit is millisecond.
* 最近一次的任务处理时间
*/
private long lastProcessTime;
/**
* The default time interval, in milliseconds, between tasks.
*/
protected static final long INTERVAL = 1000L;
/**
* merge task.
* 合并任务
*
* @param task task
*/
public abstract void merge(AbstractDelayTask task);
public void setTaskInterval(long interval) {
this.taskInterval = interval;
}
public long getTaskInterval() {
return this.taskInterval;
}
public void setLastProcessTime(long lastProcessTime) {
this.lastProcessTime = lastProcessTime;
}
public long getLastProcessTime() {
return this.lastProcessTime;
}
// 判断是否可以执行
@Override
public boolean shouldProcess() {
return (System.currentTimeMillis() - this.lastProcessTime >= this.taskInterval);
}
AbstractDelayTask 是一个延时任务,并且任务可以被合并,我们可以看看NacosDelayTaskExecuteEngine是如何启动,如何处理延时任务的。
NacosDelayTaskExecuteEngine和AbstractDelayTask关系如下图:
3.2、NacosDelayTaskExecuteEngine
/**
* Nacos delay task execute engine.
* nacos延时任务执行引擎
*
* @author xiweng.yy
*/
public class NacosDelayTaskExecuteEngine extends AbstractNacosTaskExecuteEngine<AbstractDelayTask> {
private final ScheduledExecutorService processingExecutor;
protected final ConcurrentHashMap<Object, AbstractDelayTask> tasks;
protected final ReentrantLock lock = new ReentrantLock();
public NacosDelayTaskExecuteEngine(String name) {
this(name, null);
}
public NacosDelayTaskExecuteEngine(String name, Logger logger) {
this(name, 32, logger, 100L);
}
public NacosDelayTaskExecuteEngine(String name, Logger logger, long processInterval) {
this(name, 32, logger, processInterval);
}
public NacosDelayTaskExecuteEngine(String name, int initCapacity, Logger logger) {
this(name, initCapacity, logger, 100L);
}
public NacosDelayTaskExecuteEngine(String name, int initCapacity, Logger logger, long processInterval) {
super(logger);
tasks = new ConcurrentHashMap<>(initCapacity);
processingExecutor = ExecutorFactory.newSingleScheduledExecutorService(new NameThreadFactory(name));
processingExecutor
.scheduleWithFixedDelay(new ProcessRunnable(), processInterval, processInterval, TimeUnit.MILLISECONDS);
}
@Override
public int size() {
lock.lock();
try {
return tasks.size();
} finally {
lock.unlock();
}
}
@Override
public boolean isEmpty() {
lock.lock();
try {
return tasks.isEmpty();
} finally {
lock.unlock();
}
}
@Override
public AbstractDelayTask removeTask(Object key) {
lock.lock();
try {
AbstractDelayTask task = tasks.get(key);
// 如果可以执行,则删除
if (null != task && task.shouldProcess()) {
return tasks.remove(key);
} else {
return null;
}
} finally {
lock.unlock();
}
}
@Override
public Collection<Object> getAllTaskKeys() {
Collection<Object> keys = new HashSet<Object>();
lock.lock();
try {
keys.addAll(tasks.keySet());
} finally {
lock.unlock();
}
return keys;
}
@Override
public void shutdown() throws NacosException {
tasks.clear();
processingExecutor.shutdown();
}
@Override
public void addTask(Object key, AbstractDelayTask newTask) {
lock.lock();
try {
AbstractDelayTask existTask = tasks.get(key);
if (null != existTask) {
newTask.merge(existTask);
}
// 如果没有实现merge方法,则直接使用新的task替换旧的
tasks.put(key, newTask);
} finally {
lock.unlock();
}
}
/**
* process tasks in execute engine.
*/
protected void processTasks() {
// 获取task所有的key
Collection<Object> keys = getAllTaskKeys();
for (Object taskKey : keys) {
// 根据key获取task,并且删除tasks对应的task,判断是否可以执行了
AbstractDelayTask task = removeTask(taskKey);
// task不能执行直接跳过
if (null == task) {
continue;
}
// 获取对应的处理器来处理task,如果没有则跳过
NacosTaskProcessor processor = getProcessor(taskKey);
if (null == processor) {
getEngineLog().error("processor not found for task, so discarded. " + task);
continue;
}
try {
// ReAdd task if process failed
// 使用processor执行,执行失败则尝试重新延迟执行
if (!processor.process(task)) {
retryFailedTask(taskKey, task);
}
} catch (Throwable e) {
getEngineLog().error("Nacos task execute error : " + e.toString(), e);
retryFailedTask(taskKey, task);
}
}
}
private void retryFailedTask(Object key, AbstractDelayTask task) {
task.setLastProcessTime(System.currentTimeMillis());
addTask(key, task);
}
private class ProcessRunnable implements Runnable {
@Override
public void run() {
try {
processTasks();
} catch (Throwable e) {
getEngineLog().error(e.toString(), e);
}
}
}
}
NacosDelayTaskExecuteEngine在创建的时候会创建一个定时任务去执行processTasks方法,拿到对应的task并且去拿task所对应的processor去执行任务,每次执行都会判断task是否可以执行了,如果可以执行则删除并执行task,以便下次不会重复执行该task,如果任务执行失败,则会重新放入,待下次重新执行。
3.3、使用案例
public class MyDelayTask extends AbstractDelayTask {
private String name;
private Integer number;
public void setName(String name) {
this.name = name;
}
public String getName() {
return name;
}
public Integer getNumber() {
return number;
}
public void setNumber(Integer number) {
this.number = number;
}
@Override
public void merge(AbstractDelayTask task) {
MyDelayTask oldMyDelayTask = (MyDelayTask) task;
// 需要保留旧的number,这里可以进行合并
this.number = oldMyDelayTask.getNumber();
}
}
public class MyNacosDelayTaskFailProcessor implements NacosTaskProcessor {
@Override
public boolean process(NacosTask task) {
System.out.println("MyNacosDelayTaskFailProcessor start...");
MyDelayTask delayTask = (MyDelayTask) task;
System.out.println("name=" + delayTask.getName() + ";number=" + delayTask.getNumber());
System.out.println("MyNacosDelayTaskFailProcessor end...");
return false;
}
}
public class MyNacosDelayTaskProcessor implements NacosTaskProcessor {
@Override
public boolean process(NacosTask task) {
System.out.println("MyNacosDelayTaskProcessor start...");
MyDelayTask delayTask = (MyDelayTask) task;
System.out.println("name=" + delayTask.getName() + ";number=" + delayTask.getNumber());
System.out.println("MyNacosDelayTaskProcessor end...");
return true;
}
}
@RunWith(MockitoJUnitRunner.class)
public class MyNacosDelayTaskExecuteEngineTest {
private NacosDelayTaskExecuteEngine nacosDelayTaskExecuteEngine;
@Before
public void setUp() throws Exception {
nacosDelayTaskExecuteEngine = new NacosDelayTaskExecuteEngine(MyNacosDelayTaskExecuteEngineTest.class.getName());
}
@After
public void tearDown() throws Exception {
nacosDelayTaskExecuteEngine.shutdown();
}
@Test
public void testAddTask() throws InterruptedException {
nacosDelayTaskExecuteEngine.addProcessor(MyDelayTask.class, new MyNacosDelayTaskProcessor());
MyDelayTask delayTask = new MyDelayTask();
// 延迟1秒后执行
delayTask.setTaskInterval(1000L);
nacosDelayTaskExecuteEngine.addTask(MyDelayTask.class, delayTask);
Thread.sleep(10000);
}
@Test
public void testRetryTaskAfterFail() throws InterruptedException {
// 执行任务失败,则会一直重试执行
nacosDelayTaskExecuteEngine.addProcessor(MyDelayTask.class, new MyNacosDelayTaskFailProcessor());
MyDelayTask delayTask = new MyDelayTask();
// 延迟1秒后执行
delayTask.setTaskInterval(1000L);
nacosDelayTaskExecuteEngine.addTask(MyDelayTask.class, delayTask);
Thread.sleep(10000);
}
@Test
public void testTaskMerge() throws InterruptedException {
nacosDelayTaskExecuteEngine.addProcessor(MyDelayTask.class, new MyNacosDelayTaskProcessor());
MyDelayTask delayTaskA = new MyDelayTask();
// 延迟1秒后执行
delayTaskA.setTaskInterval(1000L);
delayTaskA.setName("taskA");
delayTaskA.setNumber(1);
MyDelayTask delayTaskB = new MyDelayTask();
// 延迟2秒后执行
delayTaskB.setTaskInterval(2000L);
delayTaskB.setName("taskB");
delayTaskB.setNumber(2);
nacosDelayTaskExecuteEngine.addTask(MyDelayTask.class, delayTaskA);
Thread.sleep(10000); // 中间做一些处理会使merge方法失效
nacosDelayTaskExecuteEngine.addTask(MyDelayTask.class, delayTaskB);
Thread.sleep(3000);
}
@Test
public void testTaskLoopMerge() throws InterruptedException {
nacosDelayTaskExecuteEngine.addProcessor(MyDelayTask.class, new MyNacosDelayTaskFailProcessor());
MyDelayTask delayTaskA = new MyDelayTask();
// 延迟1秒后执行
delayTaskA.setTaskInterval(1000L);
delayTaskA.setName("taskA");
delayTaskA.setNumber(1);
MyDelayTask delayTaskB = new MyDelayTask();
// 延迟1秒后执行
delayTaskB.setTaskInterval(1000L);
delayTaskB.setName("taskB");
delayTaskB.setNumber(2);
nacosDelayTaskExecuteEngine.addTask(MyDelayTask.class, delayTaskA);
Thread.sleep(10000);
nacosDelayTaskExecuteEngine.addTask(MyDelayTask.class, delayTaskB);
Thread.sleep(3000);
}
}
在testTaskMerge方法中,如果两个添加任务之间时间相隔比较久,会出现merge方法失效问题
nacosDelayTaskExecuteEngine.addTask(MyDelayTask.class, delayTaskA);
Thread.sleep(10000); // 中间做一些处理会使merge方法失效
nacosDelayTaskExecuteEngine.addTask(MyDelayTask.class, delayTaskB);
在testTaskLoopMerge方法中,可以利用处理器返回失败,做一个循环延时任务的效果,执行效果如下显示:
在循环执行延时任务A的时候,后面又添加了任务B,A被B合并了,但是保留了A的number属性值。
3.4、NacosDelayTaskExecuteEngine总结
在创建引擎的时候会初始化一个定时任务,在定时任务里面会去取task和processor处理,该引擎中只能添加执行继承了AbstractDelayTask的任务,并且必须指定处理器,如果不指定,则无法处理任务。AbstractDelayTask中有个关键的方法shouldProcess,是能够实现延时的关键。
以上都是个人理解,如有错误恳请指正。