azkaban 安装_azkaban 镜像-CSDN博客

本文链接：https://blog.csdn.net/liangkiller/article/details/102950007

在hadoop领域，常见工作流调度系统有：Oozie,Azkaban,Cascading,Hamake

环境

centos 6.5
JDK 8
azkaban 3.79
gradle-3.5

hostname	ip
vps138	192.168.1.138
vps155	192.168.1.155
vps156	192.168.1.156

下载

$ cd /var/tmp
$ wget https://github.com/azkaban/azkaban/archive/3.79.0.tar.gz
$ tar -zxf 3.79.0.tar.gz
$ cd azkaban-3.79.0

//# 发邮件支持SSL
$ vi ./azkaban-common/src/main/java/azkaban/utils/EmailMessage.java
//# 加到import区域
import java.security.Security;

//# 修改sendEmail()方法
props.put("mail.smtp.ssl.trust", this._mailHost); #在这之后添加
Security.addProvider(new com.sun.net.ssl.internal.ssl.Provider());
final String SSL_FACTORY = "javax.net.ssl.SSLSocketFactory";
props.setProperty("mail.smtp.socketFactory.class", SSL_FACTORY);
props.setProperty("mail.smtp.socketFactory.fallback", "false");
props.setProperty("mail.smtp.socketFactory.port", "465");

编译

//# 安装gradle
$ wget https://services.gradle.org/distributions/gradle-3.5-bin.zip
$ unzip gradle-3.5-bin.zip
$ mv gradle-3.5 /usr/local/gradle
$ echo 'export GRADLE_HOME=/usr/local/gradle' > /etc/profile.d/gradle.sh
$ vi /etc/profile.d/path.sh
export PATH=$PATH:$GRADLE_HOME/bin

$ source /etc/profile
//# 添加国内镜像
$ mkdir -p /root/.gradle
$ vi /root/.gradle/init.gradle
allprojects{
    repositories {
        def ALIYUN_REPOSITORY_URL = 'http://maven.aliyun.com/nexus/content/groups/public'
        def ALIYUN_JCENTER_URL = 'http://maven.aliyun.com/nexus/content/repositories/jcenter'
        all { ArtifactRepository repo ->
            if(repo instanceof MavenArtifactRepository){
                def url = repo.url.toString()
                if (url.startsWith('https://repo1.maven.org/maven2')) {
                    project.logger.lifecycle "Repository ${repo.url} replaced by $ALIYUN_REPOSITORY_URL."
                    remove repo
                }
                if (url.startsWith('https://jcenter.bintray.com/')) {
                    project.logger.lifecycle "Repository ${repo.url} replaced by $ALIYUN_JCENTER_URL."
                    remove repo
                }
            }
        }
        maven {
            url ALIYUN_REPOSITORY_URL
            url ALIYUN_JCENTER_URL
        }
    }
}

$ cd /var/tmp/azkaban-3.79.0
$ gradle wrapper --gradle-distribution-url https://repo.gradle.org/gradle/dist-snapshots/gradle-kotlin-dsl-4.9-20180629133202+0000-all.zip
$ ./gradlew distTar -x test

复制build 后的tar包

编译好的文件都放在各目录的build/distributions/下

$ cp ./*/build/distributions/*.tar.gz /var/tmp
az-hadoop-jobtype-plugin-*.tar.gz
az-hdfs-viewer-*.tar.gz
az-jobsummary-*.tar.gz
azkaban-db-*.tar.gz
azkaban-exec-server-*.tar.gz
azkaban-hadoop-security-plugin-*.tar.gz
azkaban-solo-server-*.tar.gz
azkaban-web-server-*.tar.gz
az-reportal-*.tar.gz

单机模式

不支持单机模式了,会报错

azkaban.use.multiple.executors must be true. Single executor mode is not supported any more

分布式模式

安装azkaban的web,exector服务

web和exector是分开的,多个exector在多台机器上
执行器(executor)通过数据库mysql协调

这里以vps138为web服务器,
vps138,vps155,vps156为exector服务器

$ mkdir -p /usr/local/azkaban

//# 在vps138上安装web server
$ tar -zxf azkaban-web-server-0.1.0-SNAPSHOT.tar.gz
$ mv azkaban-web-server-0.1.0-SNAPSHOT  /usr/local/azkaban/web

//# 在vps138上安装executor server
$ tar -zxf azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz
$ mv azkaban-exec-server-0.1.0-SNAPSHOT /usr/local/azkaban/exec138

//# mysql建表语句,导入msyql
$ tar -zxf azkaban-db-0.1.0-SNAPSHOT.tar.gz

将编译好的azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz传到vps155,vps156;

//# 在vps155上安装executor server
$ tar -zxf azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz
$ mv azkaban-exec-server-0.1.0-SNAPSHOT /usr/local/azkaban/exec155

//# 在vps156上安装executor server
$ tar -zxf azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz
$ mv azkaban-exec-server-0.1.0-SNAPSHOT /usr/local/azkaban/exec156

配置web server

在vps138上配置web server

$ cd /usr/local/azkaban/web
$ mv conf/azkaban.properties conf/azkaban.properties.bak
$ vi conf/azkaban.properties
azkaban.name=BSS
azkaban.label=My Local Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=web/
default.timezone.id=Asia/Shanghai
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=conf/azkaban-users.xml
executor.global.properties=conf/global.properties
azkaban.project.dir=projects
velocity.dev.mode=false
lockdown.create.projects=false
lockdown.upload.projects=false
cache.directory=cache
jetty.connector.stats=true
executor.connector.stats=true
azkaban.jobtype.plugin.dir=plugins/jobtypes

# web界面端口设置
jetty.use.ssl=false
jetty.maxThreads=25
jetty.port=8082

# 数据库配置
database.type=mysql
mysql.port=63751
mysql.host=localhost
mysql.database=azkaban
mysql.user=azkaban
mysql.password=azkaban_password
mysql.numconnections=100

# 多executor调度规则设置
azkaban.use.multiple.executors=true
azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus
azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
azkaban.executorselector.comparator.Memory=1
azkaban.executorselector.comparator.LastDispatched=1
azkaban.executorselector.comparator.CpuUsage=1

web默认用户密码

$ cat conf/azkaban-users.xml
<azkaban-users>
  <user groups="azkaban" password="azkaban" roles="admin" username="azkaban"/>
  <user password="metrics" roles="metrics" username="metrics"/>

  <role name="admin" permissions="ADMIN"/>
  <role name="metrics" permissions="METRICS"/>
</azkaban-users>

配置Executor

在vps138,vps155,vps156上配置

$ cd /usr/local/azkaban/exec
$ vi ./conf/azkaban.properties
executor.maxThreads=50
executor.flow.threads=30
executor.port=12321
azkaban.jobtype.plugin.dir=plugins/jobtypes
executor.global.properties=conf/global.properties

# 数据库配置,在启动executor时会向executors表插入记录
database.type=mysql
mysql.port=63751
mysql.host=localhost
mysql.database=azkaban
mysql.user=azkaban
mysql.password=azkaban_password
mysql.numconnections=100

MySQL安装

参见MySQL 安装并修改密码策略

创建azkaban需要的数据库,用户密码,导入建表语句

mysql> create database azkaban DEFAULT CHARACTER SET utf8;
mysql> grant all on azkaban.* TO 'azkaban'@'%' IDENTIFIED BY 'azkaban_password';
mysql> use azkaban;
mysql> source /var/tmp/azkaban-db-0.1.0-SNAPSHOT/create-all-sql-0.1.0-SNAPSHOT.sql
//# 默认启用executor
mysql> ALTER TABLE `executors` CHANGE `active` `active` TINYINT DEFAULT 1;

启动azkaban服务



//# 会在执行目录下生成日志
# 必须先启动executor-server，否则web-server会报找不到executor的错误
$ cd /usr/local/azkaban/exec138
$ ./bin/start-exec.sh

$ cd /usr/local/azkaban/web
$ ./bin/start-web.sh

停止:
$ ./bin/shutdown-web.sh
$ ./bin/shutdown-exec.sh

WEB界面
http://ip:8082/

输入: azkaban/azkaban

文档地址: Azkaban documentation