大数据学习30：Azkaban3.x 的 two server mode 部署

Azkaban two server mode 部署
接着上次说到的编译，在编译好的目录中找到web 和 executor 。

安装 azkaban-web-server
解压：
/opt/software/azkaban/azkaban/azkaban-web-server/build/distributions
tar -xzvf azkaban-web-server-3.39.0-7-g05c39f7.tar.gz -C /opt/software/azkaban/

安装个mysql ，本机已经安装好了
mysql -uroot -ppassword
#1、创建Azkaban数据库
create database azkaban;
use azkaban 
#创建Azkaban用户
CREATE USER 'azkaban'@'%' IDENTIFIED BY 'azkaban';
#赋权限
GRANT SELECT,INSERT,UPDATE,DELETE ON azkaban.* to 'azkaban'@'%' WITH GRANT OPTION;
也可以赋予更大的权限

#配置这个参数，将上传文件限制改大
/etc/my.cnf
max_allowed_packet=1024M
#重启数据库

#2、初始化数据库
在目录下找到 /opt/software/azkaban/azkaban/azkaban-db/build/sql/create-all-sql-3.39.0-7-g05c39f7.sql
mysql> source /opt/software/azkaban/azkaban/azkaban-db/build/sql/create-all-sql-3.39.0-7-g05c39f7.sql;

#3、安装web
ssl 
#keytool -keystore keystore -alias jetty -genkey -keyalg RSA
[root@hadoop002 azkaban]# keytool -keystore keystore -alias jetty -genkey -keyalg RSA
Enter keystore password:  
Re-enter new password:  #000000
What is your first and last name?
  [Unknown]:  
What is the name of your organizational unit?
  [Unknown]:  
What is the name of your organization?
  [Unknown]:  
What is the name of your City or Locality?
  [Unknown]:  
What is the name of your State or Province?
  [Unknown]:  
What is the two-letter country code for this unit?
  [Unknown]:  
Is CN=Unknown, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown correct?
  [no]:  y

Enter key password for <jetty>
(RETURN if same as keystore password):  
Re-enter new password: 
#生成keystore

#4、配置 Azkaban 参数
[root@hadoop002 azkaban]# cd azkaban-web-server-3.39.0-7-g05c39f7/
[root@hadoop002 azkaban-web-server-3.39.0-7-g05c39f7]# ls 
bin  lib  web

#创建文件
mkdir conf
cd conf/
#将单机的参数文件拷贝过来
cp /opt/software/azkaban/azkaban-solo-server-3.39.0-7-g05c39f7/conf/azkaban.properties  ./

vi azkaban.properties #修改 
database.type=mysql
mysql.port=3306
mysql.host=localhost
mysql.database=azkaban
mysql.user=azkaban
mysql.password=azkaban
mysql.numconnections=100

jetty.use.ssl=true #这里可以写false，这样就会用8888端口
jetty.maxThreads=25
jetty.port=8888


jetty.keystore=/opt/software/azkaban/keystore
jetty.password=000000
jetty.keypassword=000000
jetty.truststore=/opt/software/azkaban/keystore
jetty.trustpassword=000000

#5、启动web server 报错
a) 缺少 conf/log4j.properties 文件 
vi conf/log4j.properties
log4j.rootLogger=INFO,C
log4j.appender.C=org.apache.log4j.ConsoleAppender
log4j.appender.C.Target=System.err
log4j.appender.C.layout=org.apache.log4j.PatternLayout
log4j.appender.C.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n

b)java.sql.SQLException: Access denied for user 'azkaban'@'hadoop002' (using password: YES)
GRANT ALL PRIVILEGES  ON azkaban.* to 'azkaban'@'%' IDENTIFIED BY 'azkaban'  WITH GRANT OPTION;

c)访问http://192.168.137.12:8443/ 失败  如果关闭jetty.use.ssl=true 则可以用http 
要用：https://192.168.137.12:8443/

#6、打开网页输入 wxk 000000
jps 
73056 AzkabanWebServer

#7、a)安装 azkaban-exec-server 解压 
/opt/software/azkaban/azkaban/azkaban-exec-server/build/distributions
tar -xzvf azkaban-exec-server-3.39.0-7-g05c39f7.tar.gz  -C /opt/software/azkaban/

b)#将web 中的conf 拷贝到 解压的文件夹中

c)#注意 azkaban.properties 时区的配置
default.timezone.id=Asia/Shanghai

d)启动bin/azkaban-executor-start.sh

e)验证 jps
74869 AzkabanExecutorServer

#8、运行一个demo，发现报错
Missing required property 'azkaban.native.lib'
在官网中有这样一段话：
Plugin Configurations
Execute-As-User
With a new security enhancement in Azkaban 3.0, Azkaban jobs can now run as the submit user or the user.to.proxy of the flow by default. This ensures that Azkaban takes advantage of the Linux permission security mechanism, and operationally this simplifies resource monitoring and visibility. Set up this behavior by doing the following:-
Execute.as.user is set to true by default. In case needed, it can also be configured to false in azkaban-plugin’s commonprivate.properties
Configure azkaban.native.lib= to the place where you are going to put the compiled execute-as-user.c file (see below)
Generate an executable on the Azkaban box for azkaban-common/src/main/c/execute-as-user.c. it should be named execute-as-user Below is a sample approach
scp ./azkaban-common/src/main/c/execute-as-user.c onto the Azkaban box
run: gcc execute-as-user.c -o execute-as-user
run: chown root execute-as-user (you might need root privilege)
run: chmod 6050 execute-as-user (you might need root privilege)
但是在最新的版本里面没有这个 execute-as-user.c 文件了

解决方法：
a) 先关闭两个服务
b) 在编译好的  azkaban-solo-server 中将 plugins/jobtypes/commonprivate.properties 连带目录直接拷贝到
azkaban-exec-server 目录中，并在 commonprivate.properties 添加一条 azkaban.native.lib=false
commonprivate.properties 中有两条记录：
execute.as.user=false
azkaban.native.lib=false

重启后运行最简单的 "hello world !" 执行成功。

#9、Azkaban 配置短信插件
思路：
http://blog.csdn.net/tracymkgld/article/details/19126087?reload
可以参考这边篇文章。