xxl-job-代码创建任务执行本地命令+解决报错

陆沙

已于 2023-09-18 16:08:23 修改

阅读量1.1k

点赞数

于 2023-09-15 10:21:48 首次发布

本文链接：https://blog.csdn.net/pxy7896/article/details/132885570

版权

springboot 同时被 2 个专栏收录

33 篇文章 2 订阅

订阅专栏

docker

15 篇文章 1 订阅

订阅专栏

问题描述

因为环境受限，所以我的mysql和xxl-job-admin是放在docker里的，执行器是放在本地的，要执行的命令也需要本地环境。

本文将记录相关配置、实现过程和遇到的问题及解决方案。

配置

xxl-job-admin配置

xxl-job-admin放在docker容器里跑，可能需要修改的配置如下：

server.port=8080
# 第二个mysql替换localhost，因为我的mysql也是放在docker容器的，名字就是mysql
spring.datasource.url=jdbc:mysql://mysql:3306/xxl_job?&autoReconnect=true&useSSL=false
# username password自己填
spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver

xxl-executor

直接使用xxl-job-executor-sample-springboot，不作修改，在local跑。

Service

service是代码创建和触发任务的核心代码。位置随便放，哪里需要放哪里。

这里的任务是执行一段命令，类似

cd /xxx/xx && python xx.py

因为我需要使用BEAN模式，所以任务参数（executorParam）里填的应该是写这段命令的shell脚本（cmd.sh），如果executor找不到这个shell脚本，会报错。成功创建的任务，在网页上应该是这种状态：
在这里插入图片描述
下面是service代码：


	public String getCookie(String url, String userName, String password) {
		if (this.cookie.length() > 0) {
        	return this.cookie;
        }
		String path = url;
        Map<String, Object> hashMap = new HashMap();
        hashMap.put("userName", userName);
        hashMap.put("password", password);
        HttpResponse response = HttpRequest.post(path).form(hashMap).execute();
        List<HttpCookie> cookies = response.getCookies();
        StringBuilder sb = new StringBuilder();
        for (HttpCookie cookie : cookies) {
            sb.append(cookie.toString());
        }
        this.cookie = sb.toString();
        return this.cookie;
	}


	public void setCookie(String cookie) {
		this.cookie = cookie;
	}


	public void triggerDynamicJob(String url, int id, String cmdStr) {
		String requestUrl = url;
		Map<String, Object> paramMap = new HashMap<>();
		paramMap.put("id", id);
		paramMap.put("executorParam", cmdStr);
		paramMap.put("addressList", "");
		HttpResponse response = HttpRequest.post(requestUrl).form(paramMap).cookie(this.cookie).execute();
		if (HttpStatus.HTTP_OK != response.getStatus()) {
        	return;
        }
        JSONObject jsonObject = new JSONObject(response.body());	
        System.out.println(jsonObject.toString());
	}


	public void delDynamicJob(String url, int id) {
		String requestUrl = url + String.valueOf(id);
		HttpResponse response = HttpRequest.get(requestUrl).cookie(this.cookie).execute();
		if (HttpStatus.HTTP_OK != response.getStatus()) {
        	return;
        }
        JSONObject jsonObject = new JSONObject(response.body());
        //System.out.println(jsonObject.toString());
	}

	
	public int createDynamicJob(String url, String cmdStr) {
		// cmdStr = "print('0')";
        // Prepare request payload
        String requestUrl = url;
        Map<String, Object> paramMap = new HashMap<>();
        // 执行器id
        paramMap.put("jobGroup", 1);
        paramMap.put("jobDesc", "dynamic");
        paramMap.put("scheduleType", "NONE");
        paramMap.put("jobCron", "");
        //paramMap.put("glueType", "GLUE_PYTHON");
        paramMap.put("glueType", "BEAN");
        paramMap.put("author", "pxy");
        paramMap.put("executorRouteStrategy", "FIRST");
        paramMap.put("executorHandler", "commandJobHandler");
        paramMap.put("executorParam", cmdStr);
        paramMap.put("misfireStrategy", "DO_NOTHING");
        paramMap.put("executorBlockStrategy", "SERIAL_EXECUTION");
        paramMap.put("executorTimeout", "0");
        paramMap.put("executorFailRetryCount", "0");
        paramMap.put("glueSource", cmdStr);
        
        HttpResponse response = HttpRequest.post(requestUrl).form(paramMap).cookie(this.cookie).execute();
        //System.out.println("Response: " + response);
        if (HttpStatus.HTTP_OK != response.getStatus()) {
        	return -1;
        }
        JSONObject jsonObject = new JSONObject(response.body());
        int jobId = jsonObject.getInt("content");
        //System.out.println(jobId);
        return jobId;
	}

需要注意的是，代码触发任务时必须要带executorParam。虽然创建时我写了executorParam，在admin网页上手动触发一次，也是能正常执行的，但是代码触发时如果不带executorParam，系统会报错命令为空：

2023-09-14 15:20:55 [com.xxl.job.core.thread.JobThread#run]-[133]-[xxl-job, JobThread-21-1694676055262] 
----------- xxl-job job execute start -----------
----------- Param:
2023-09-14 15:20:55 [com.xxl.job.executor.service.jobhandler.SampleXxlJob#commandJobHandler]-[104]-[xxl-job, JobThread-21-1694676055262] java.io.IOException: Cannot run program "": error=2, 没有那个文件或目录
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
	at com.xxl.job.executor.service.jobhandler.SampleXxlJob.commandJobHandler(SampleXxlJob.java:88)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.xxl.job.core.handler.impl.MethodJobHandler.execute(MethodJobHandler.java:31)
	at com.xxl.job.core.thread.JobThread.run(JobThread.java:166)
Caused by: java.io.IOException: error=2, 没有那个文件或目录
	at java.lang.UNIXProcess.forkAndExec(Native Method)
	at java.lang.UNIXProcess.(UNIXProcess.java:247)
	at java.lang.ProcessImpl.start(ProcessImpl.java:134)
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
	... 7 more

Controller

这里是实际调用Service时的示例代码。

		// 获取cookie. 用户名密码使用默认的
		this.taskService.setCookie(this.taskService.getCookie("http://localhost:8080/xxl-job-admin/login", 
				"admin", "123456"));
		String createUrl = "http://localhost:8080/xxl-job-admin/jobinfo/add";
		// 创建任务
		// 这里homePath是我自定义的一个路径，要求本地可以访问
		// 而且cmd.sh要加上权限，不然会报错
		String cmdStr = homePath + "/cmd.sh";
		int jid = this.taskService.createDynamicJob(createUrl, cmdStr);
		// 要给homePath加权限
		String cmdGrant = "chmod -R 777 " + homePath;
		String[] cmds = new String[] {"sh", "-c", cmdGrant};
		Process pcs = null;
		try {
			pcs = Runtime.getRuntime().exec(cmds);
			BufferedReader stdInput = new BufferedReader(new InputStreamReader(pcs.getInputStream()));
            BufferedReader stdError = new BufferedReader(new InputStreamReader(pcs.getErrorStream()));
            String s = null;
            
            while ((s = stdInput.readLine()) != null) {
                  //log.info(s);
            }
            //log.info("标准错误的输出命令");
            while ((s = stdError.readLine()) != null) {
                  //log.info(s);
            }
			// 会一直等待，所以需要在finally块中杀死进程
			pcs.waitFor();
		} catch (IOException e) {
			// TODO Auto-generated catch block
			//e.printStackTrace();
		} catch (Exception e) {
			//log.error(e.toString());
		}
		finally {
			pcs.destroyForcibly();
		}
		// 触发任务
        this.taskService.triggerDynamicJob("http://localhost:8080/xxl-job-admin/jobinfo/trigger", jid, cmdStr);

成功调度后，调度结果显示成功。如果任务需要的时间较长，那执行结果一开始是null，等任务完成后才会变成成功。
在这里插入图片描述

debug

java.sql.SQLNonTransientConnectionException: Could not create connection to database server

java.sql.SQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110)
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
	...
	at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:86)
	at com.sun.proxy.$Proxy74.findByAddressType(Unknown Source)
	at com.xxl.job.admin.core.thread.JobRegistryHelper$3.run(JobRegistryHelper.java:62)
	at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: com.mysql.cj.exceptions.CJCommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:64)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
	at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
	at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105)
	at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:151)
	at com.mysql.cj.exceptions.ExceptionFactory.createCommunicationsException(ExceptionFactory.java:167)
	at com.mysql.cj.protocol.a.NativeSocketConnection.connect(NativeSocketConnection.java:89)
	at com.mysql.cj.NativeSession.connect(NativeSession.java:120)
	at com.mysql.cj.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:838)
	... 40 common frames omitted
Caused by: java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.Net.pollConnect(Native Method)
	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:660)
	at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:542)
	at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:333)
	at java.base/java.net.Socket.connect(Socket.java:648)
	at com.mysql.cj.protocol.StandardSocketFactory.connect(StandardSocketFactory.java:153)
	at com.mysql.cj.protocol.a.NativeSocketConnection.connect(NativeSocketConnection.java:63)
	... 42 common frames omitted

一开始我的xxl-job-admin不是放在docker的，是放在本地的，结果运行时就报这个错误。我也用这篇（docker-配置mysql+外置数据+连接另一个容器（ip法和network法））里的ip法改了admin的配置，仍然报错，应该还是超时的。最后我只能把admin放docker，所幸本地的执行器正常运行。

xxl-job remoting error(No route to host)

在这里插入图片描述
这个是因为9999端口没有开放。先添加端口号，再重载防火墙即可。

firewall-cmd --zone=public --add-port=9999/tcp --permanent
firewall-cmd --reload

Caused by: java.io.IOException: error=2, 没有那个文件或目录

这个要检查2点：

创建任务时，executorParam或glueSource是不是空的
如果glueType是GLUE_PYTHON，那么glueSource放的是python ide要运行的语句，为空很可能会报错；如果glueType是BEAN，那么在触发任务时要检查executorParam是不是空的，虽然在创建任务时，executorParam并不是必须带的参数，但是BEAN触发时一定要带。
sh文件是否存在、路径是否可以访问
尤其是本文这种BEAN模式执行命令的，千万不要在executorParam上写命令啊！！！那里要写sh的文件路径！！！把命令直接写进sh文件，然后填上路径就行，不然会有下面的报错：

2023-09-13 16:57:41 [com.xxl.job.core.thread.JobThread#run]-[133]-[xxl-job, JobThread-17-1694595461226] 
----------- xxl-job job execute start -----------
----------- Param:cd /home/pxy/sgrna && nohup /home/bioinfo/miniconda3/bin/snakemake  -k  --core 8  1>>snakemake.log 2>&1 &
2023-09-13 16:57:41 [com.xxl.job.executor.service.jobhandler.SampleXxlJob#commandJobHandler]-[104]-[xxl-job, JobThread-17-1694595461226] java.io.IOException: Cannot run program "cd /xxx/xx && nohup /xxxx &": error=2, 没有那个文件或目录
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
	at com.xxl.job.executor.service.jobhandler.SampleXxlJob.commandJobHandler(SampleXxlJob.java:88)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.xxl.job.core.handler.impl.MethodJobHandler.execute(MethodJobHandler.java:31)
	at com.xxl.job.core.thread.JobThread.run(JobThread.java:166)
Caused by: java.io.IOException: error=2, 没有那个文件或目录
	at java.lang.UNIXProcess.forkAndExec(Native Method)
	at java.lang.UNIXProcess.(UNIXProcess.java:247)
	at java.lang.ProcessImpl.start(ProcessImpl.java:134)
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
	... 7 more