shell脚本实现mapreduce任务流

概述

• 作用
• 示例

作用

• 把几个任务组织成一个工作流

示例

• task1 task2 task3  现在 task1 正确完成后执行 task2，task2 正确完成后执行 task3
• 其脚本对应如下

• #!/bin/bash
  function check
  {
          echo $chkid
          if [ $chkid -ne 0 ];then
                  exit
          fi
  }
  ############################################# main ###############################################
  
  # Delete temporary files which are already exits on hdfs that comed form last distcp
  # Parameter:<delete_file_location>
  hadoop fs -rmr /user/hadoop/calculate/demo/*
  chkid=`echo $?`
  check $chkid
  # copy source files form 156 to 130 using distcp command and store this source file on /user/hadoop/calculate/demo/
  # Parameter:<configuration_file_location><soure_file_location><destination_file_location>
  hadoop --config ~/app/conf/ distcp hdfs://xxx:9000/compute/De