Pipeline shell step aborts with -1 : Jenkins-48300

(JENKINS-48300: if on an extremely laggy filesystem, consider -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.HEARTBEAT_CHECK_INTERVAL=86400)

这个问题发生在执行sh 时由于执行时间过长,jenkins 中的外部任务,例如shell 命令,被一个Durable Task 的插件通过heartbeat管理着。当Jenkins agent上发生OOM,100% CPU或者IO阻塞 等问题的时候, heartbeat会偶发性的中断,导致jenkins 的sh 任务异常。

解决方法1:

重构pipeline,使其

1,尽量避免大量并行的复杂step,特别是IO/计算密集型任务

2,尽量避免使用sh 来执行通用型的,IO密集,外部依赖型,复杂型和高耗时的操作. 如可以,尽量使用jenkins plugin来拆解和构建, 如 Artifactory, tee

解决方法2 :

通过增大HEARTBEAT_CHECK_INTERVAL 来提高healthcheck的容错率,可通过下面的代码在pipeline 上下文中来设置

System.setProperty("org.jenkinsci.plugins.durabletask.BourneShellScript.HEARTBEAT_CHECK_INTERVAL", "86400"); # any interval you wish

[2023-05-31 11:07:02] Started by user coding [2023-05-31 11:07:02] Running in Durability level: MAX_SURVIVABILITY [2023-05-31 11:07:04] [Pipeline] Start of Pipeline [2023-05-31 11:07:06] [Pipeline] getContext [2023-05-31 11:07:07] [Pipeline] node [2023-05-31 11:07:07] Running on Jenkins in /root/codingci/tools/jenkins_home/workspace/2553946-cci-31810232-464995 [2023-05-31 11:07:07] [Pipeline] { [2023-05-31 11:07:08] [Pipeline] withEnv [2023-05-31 11:07:08] [Pipeline] { [2023-05-31 11:07:08] [Pipeline] withDockerRegistry [2023-05-31 11:07:08] [Pipeline] { [2023-05-31 11:07:08] [Pipeline] isUnix [2023-05-31 11:07:08] [Pipeline] sh [2023-05-31 11:07:08] + docker inspect -f . public/docker/nodejs:18-2022 [2023-05-31 11:07:08] /root/codingci/tools/jenkins_home/workspace/2553946-cci-31810232-464995@tmp/durable-221f7a67/script.sh: 1: docker: not found [2023-05-31 11:07:08] [Pipeline] isUnix [2023-05-31 11:07:08] [Pipeline] sh [2023-05-31 11:07:09] + docker inspect -f . coding-public-docker.pkg.coding.net/public/docker/nodejs:18-2022 [2023-05-31 11:07:09] /root/codingci/tools/jenkins_home/workspace/2553946-cci-31810232-464995@tmp/durable-4892b310/script.sh: 1: docker: not found [2023-05-31 11:07:09] [Pipeline] isUnix [2023-05-31 11:07:09] [Pipeline] sh [2023-05-31 11:07:09] + docker pull coding-public-docker.pkg.coding.net/public/docker/nodejs:18-2022 [2023-05-31 11:07:09] /root/codingci/tools/jenkins_home/workspace/2553946-cci-31810232-464995@tmp/durable-0770ad1b/script.sh: 1: docker: not found [2023-05-31 11:07:09] [Pipeline] } [2023-05-31 11:07:09] [Pipeline] // withDockerRegistry [2023-05-31 11:07:09] [Pipeline] } [2023-05-31 11:07:09] [Pipeline] // withEnv [2023-05-31 11:07:09] [Pipeline] } [2023-05-31 11:07:09] [Pipeline] // node [2023-05-31 11:07:09] [Pipeline] End of Pipeline [2023-05-31 11:07:09] ERROR: script returned exit code 127 [2023-05-31 11:07:09] Finished: FAILURE
06-01
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值