oozie-ssh

oozie使用ssh远程执行shell

配置文件

job.properties

# 集群参数

#nameNode地址
nameNode=hdfs://node1:8020
#resourceManager地址
jobTracker=node1:8032
#oozie队列 这个属性一般不做修改
queueName=default

# oozie
#coordinator.xml在hdfs上的路径
oozie.coord.application.path=${nameNode}/user/oozie/apps/shzt
#workflow.xml在hdfs上的路径
workflowAppUri=${nameNode}/user/oozie/apps/shzt
#workflow的名字
workflowName=workflow
#定时任务的开始时间
#start=2018-11-16T17:50+0800
#定时任务的结束时间
end=2020-12-31T23:50+0800
#任务执行的时间间隔,crontab格式的间隔
#星期一： MON
#星期二： TUE
#星期三： WED
#星期四： THUR
#星期五： FRI
#星期六： SAT
#星期日： SUN
#周五不抽取数据
frequency=0 * * * SUN-THUR,SAT
#ssh-user=admin
#ssh-host=node6

coordinator.xml

<coordinator-app name="shzt-coordinator" frequency="${frequency}" start="${start}" end="${end}" timezone="GMT+0800" xmlns="uri:oozie:coordinator:0.4">
    <action>
        <workflow>
            <app-path>${workflowAppUri}</app-path>
            <configuration>
                <property>
                    <name>nameNode</name>
                    <value>${nameNode}</value>
                </property>
                <property>
                    <name>jobTracker</name>
                    <value>${jobTracker}</value>
                </property>
                <property>
                    <name>queueName</name>
                    <value>${queueName}</value>
                </property>
            </configuration>
        </workflow>
    </action>
</coordinator-app>

workflow.xml

<workflow-app xmlns="uri:oozie:workflow:0.5" name="shzt-workflow">

    <start to="start-ssh"/>
    
    <action name="start-ssh" retry-max="3" retry-interval="5">  
        <ssh xmlns="uri:oozie:ssh-action:0.1">
            <host>admin@node6</host>
            <command>sh</command>
            <args>/home/admin/etl_shell/run_ssp_job.sh</args>
            <args>j_ods_ssp_001</args>
            <args>/job/ods/ssp</args>
        </ssh>
        <ok to="end"/>
        <error to="fail"/>
    </action>
    
    <kill name="fail">
        <message>Shell action failed, job id[${wf:id()}], job name[${wf:name()}], error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    
    <end name="end"/>
    
</workflow-app>

启动命令

oozie job -oozie http://node6:11000/oozie/ -config job.properties -run -Dstart=`date "+%Y-%m-%dT%H:%M"+0800`

注意

运行以上代码，出现了如下错误

2018-11-19 14:59:55,913 INFO org.apache.oozie.action.ssh.SshActionExecutor: SERVER[node6] USER[admin] GROUP[-] TOKEN[] APP[shzt-workflow] JOB[0000008-181119143942306-oozie-oozi-W] ACTION[0000008-181119143942306-oozie-oozi-W@start-ssh] Attempting to copy ssh base scripts to remote host [admin@node6]
2018-11-19 14:59:55,996 WARN org.apache.oozie.action.ssh.SshActionExecutor: SERVER[node6] USER[admin] GROUP[-] TOKEN[] APP[shzt-workflow] JOB[0000008-181119143942306-oozie-oozi-W] ACTION[0000008-181119143942306-oozie-oozi-W@start-ssh] Error while executing ssh EXECUTION
2018-11-19 14:59:55,996 WARN org.apache.oozie.command.wf.ActionStartXCommand: SERVER[node6] USER[admin] GROUP[-] TOKEN[] APP[shzt-workflow] JOB[0000008-181119143942306-oozie-oozi-W] ACTION[0000008-181119143942306-oozie-oozi-W@start-ssh] Error starting action [start-ssh]. ErrorType [NON_TRANSIENT], ErrorCode [AUTH_FAILED], Message [AUTH_FAILED: Not able to perform operation [ssh -o PasswordAuthentication=no -o KbdInteractiveDevices=no -o StrictHostKeyChecking=no -o ConnectTimeout=20 admin@node6  mkdir -p oozie-oozi/0000008-181119143942306-oozie-oozi-W/start-ssh--ssh/ ] | ErrorStream: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
]
org.apache.oozie.action.ActionExecutorException: AUTH_FAILED: Not able to perform operation [ssh -o PasswordAuthentication=no -o KbdInteractiveDevices=no -o StrictHostKeyChecking=no -o ConnectTimeout=20 admin@node6  mkdir -p oozie-oozi/0000008-181119143942306-oozie-oozi-W/start-ssh--ssh/ ] | ErrorStream: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

    at org.apache.oozie.action.ssh.SshActionExecutor.execute(SshActionExecutor.java:592)
    at org.apache.oozie.action.ssh.SshActionExecutor.start(SshActionExecutor.java:206)
    at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:232)
    at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
    at org.apache.oozie.command.XCommand.call(XCommand.java:286)
    at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:332)
    at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:261)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Not able to perform operation [ssh -o PasswordAuthentication=no -o KbdInteractiveDevices=no -o StrictHostKeyChecking=no -o ConnectTimeout=20 admin@node6  mkdir -p oozie-oozi/0000008-181119143942306-oozie-oozi-W/start-ssh--ssh/ ] | ErrorStream: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

    at org.apache.oozie.action.ssh.SshActionExecutor.executeCommand(SshActionExecutor.java:342)
    at org.apache.oozie.action.ssh.SshActionExecutor.setupRemote(SshActionExecutor.java:375)
    at org.apache.oozie.action.ssh.SshActionExecutor$1.call(SshActionExecutor.java:208)
    at org.apache.oozie.action.ssh.SshActionExecutor$1.call(SshActionExecutor.java:206)
    at org.apache.oozie.action.ssh.SshActionExecutor.execute(SshActionExecutor.java:550)
    ... 11 more
2018-11-19 14:59:55,996 WARN org.apache.oozie.command.wf.ActionStartXCommand: SERVER[node6] USER[admin] GROUP[-] TOKEN[] APP[shzt-workflow] JOB[0000008-181119143942306-oozie-oozi-W] ACTION[0000008-181119143942306-oozie-oozi-W@start-ssh] Suspending Workflow Job id=0000008-181119143942306-oozie-oozi-W
2018-11-19 15:00:46,043 INFO org.apache.oozie.command.wf.KillXCommand: SERVER[node6] USER[admin] GROUP[-] TOKEN[] APP[shzt-workflow] JOB[0000008-181119143942306-oozie-oozi-W] ACTION[] STARTED WorkflowKillXCommand for jobId=0000008-181119143942306-oozie-oozi-W
2018-11-19 15:00:46,084 INFO org.apache.oozie.command.wf.WorkflowNotificationXCommand: SERVER[node6] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000008-181119143942306-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000008-181119143942306-oozie-oozi-W

很多网友都在说需要配置免密登录，由于我们使用的是admin的账户，所以我配置了两台服务器的admin免密登录，依然报这个错误，然后我估计是帐号不对，然后又去查找资料，终于找到原因和解决方案了。

原因：ssh是没有输入密码导致ssh失败，这里需要配置免密登录,oozie的默认账户是oozie

• 方案一：配置oozie到这台服务器的免密登录，由于我使用的admin,主机是node6，所以需要配置oozie这个帐号到admin@node6的免密登录

• 方案二：修改oozie的账户为有免密登录的用户名,可以在oozie的配置选项卡-高级里面找到oozie的默认账户，然后更改为有免密权限的账户即可

转载于:https://www.cnblogs.com/shenjie2017/articles/9990360.html