问题现象
有个流程一启动就失败,在 Job List 和 Flow Log中均无日志输出
在 History 查看执行历史,发现没有Start Time
排查过程
先找到执行该流程的Executor,然后查看Executor日志。可以根据 Execution Id 查找历史日志。也可以 tail -f 日志文件,然后重试该流程。
2023-03-22 11:13:28 INFO ExecutorServlet:127 - User has called action execute on 57925
2023-03-22 11:13:28 INFO FlowPreparer:163 - Project already cached. Skipping download. ProjectVersion{projectId=33, version=2, installedDir=/home/azkaban/azkaban-exec-server/projects/33.2}
2023-03-22 11:13:28 ERROR FlowPreparer:141 - java.nio.file.AccessDeniedException: /home/azkaban/azkaban-exec-server/projects/33.2/___azkaban_project_dir_size_in_bytes___
2023-03-22 11:13:28 ERROR FlowPreparer:115 - Error in setting up project directory: /home/azkaban/azkaban-exec-server/projects, Exception: java.lang.NullPointerException
2023-03-22 11:13:28 ERROR ExecutorServlet:158 - java.lang.NullPointerException
java.lang.RuntimeException: java.lang.NullPointerException
at azkaban.execapp.FlowPreparer.setup(FlowPreparer.java:117)
at azkaban.execapp.FlowRunnerManager.submitFlow(FlowRunnerManager.java:363)
at azkaban.execapp.ExecutorServlet.handleAjaxExecute(ExecutorServlet.java:288)
at azkaban.execapp.ExecutorServlet.handleRequest(ExecutorServlet.java:136)
at azkaban.execapp.ExecutorServlet.doPost(ExecutorServlet.java:93)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.lang.NullPointerException
2023-03-22 11:13:29 INFO FlowRunnerManager:873 - # of executing flows: 0
看到有个 AccessDeniedException
猜测是目录权限问题,/home/azkaban/azkaban-exec-server/projects/33.2
ll /home/azkaban/azkaban-exec-server/projects/33.2
显示为root用户,当前启动executor为普通用户,所以没有权限写入。
应该是之前用root用户启动过导致的。
切换到root 用户 将 azkaban-exec-server 所有目录用户改为当前启动executor的用户
chown -R azkaban:azkaban azkaban-exec-server
验证
重新调度起该流程,则可以正常running了。