azkaban 安装

azkaban

azkaban 是一个用Java开发的开源调度工具workflow,同类型的工具有用Python开发的 airflow 和 luigi.

build

在我的windows上无法build成功，我在CentOS上build成功了

git clone https://github.com/azkaban/azkaban.git
cd azkaban
./gradlew build installDist

run

我的需求比较简单 solo server 就可以满足要求

cd azkaban-solo-server/build/install/azkaban-solo-server
bin/azkaban-solo-start.sh
bin/azkaban-solo-shutdown.sh

config

使用MySQL数据库，MySQL新建用户

CREATE DATABASE azkaban;
CREATE USER 'azkaban'@'%' IDENTIFIED BY 'azkaban';
GRANT ALL PRIVILEGES ON azkaban.* to 'azkaban'@'%';

修改配置文件

#设置为上海时间(东八区)，否则会按美国时间执行
default.timezone.id=Asia/Shanghai

#注释掉默认的H2数据库配置后，配置MySQL数据库
database.type=mysql
mysql.port=3306
mysql.host=192.168.0.1
mysql.database=azkaban
mysql.user=username
mysql.password=password
mysql.numconnections=10


#配置告警邮件
mail.sender=xxx@163.com
mail.host=smtp.163.com
mail.user=mailUsername
mail.password=mailPassword

启动的时候会自动创建数据库

plugin

https://github.com/azkaban/az...

使用ant编译，需要安装ant

sudo yum install ant
sudo yum install ant-junit

sudo apt-get install ant
sudo apt-get install ant-optional

git clone https://github.com/azkaban/azkaban-plugins.git

# step 1
cd azkaban-plugins
cd plugins
cd hadoopsecuritymanager
ant package

# step 2
cd ..
cd crypto
ant package


# step 3
cd ..
cd jobtype
ant package

azkaban-plugins/dist/jobtype/packages/azkaban-jobtype-3.0.0.tar.gz
解压到
azkaban/azkaban-solo-server/build/install/azkaban-solo-server/plugins/jobtypes

commonprivate.properties
common.properties

execute.as.user 设置为false，否者会报错

# configs for jobtype security settings
execute.as.user=false
azkaban.native.lib=

我目前只想运行java job，不想使用hadoop，修改源代码，可以不使用hadoop。否则没有配置hadoop 环境会报错。

--- a/plugins/jobtype/src/azkaban/jobtype/JavaJob.java
+++ b/plugins/jobtype/src/azkaban/jobtype/JavaJob.java
@@ -75,8 +75,8 @@ public class JavaJob extends JavaProcessJob {
     classPath.add(getSourcePathFromClass(JavaProcessJob.class));
     classPath.add(getSourcePathFromClass(SecurityUtils.class));
 
-    classPath.add(HadoopConfigurationInjector.getPath(getJobProps(),
-        getWorkingDirectory()));
+    //classPath.add(HadoopConfigurationInjector.getPath(getJobProps(),
+    //    getWorkingDirectory()));
 
     String loggerPath = getSourcePathFromClass(org.apache.log4j.Logger.class);
     if (!classPath.contains(loggerPath)) {
@@ -157,8 +157,8 @@ public class JavaJob extends JavaProcessJob {
 
   @Override
   public void run() throws Exception {
-    HadoopConfigurationInjector.prepareResourcesToInject(getJobProps(),
-        getWorkingDirectory());
+    //HadoopConfigurationInjector.prepareResourcesToInject(getJobProps(),
+    //    getWorkingDirectory());
     super.run();
   }
 }

--- a/plugins/jobtype/src/azkaban/jobtype/JavaJobRunnerMain.java
+++ b/plugins/jobtype/src/azkaban/jobtype/JavaJobRunnerMain.java
@@ -104,7 +104,7 @@ public class JavaJobRunnerMain {
       }
       _logger.info("Class name " + className);
 
-      HadoopConfigurationInjector.injectResources(new Props(null, props));
+      //HadoopConfigurationInjector.injectResources(new Props(null, props));
 
       // Create the object using proxy
       if (SecurityUtils.shouldProxy(props)) {

重新编译打包就可以