全流程调度

目录

Azkaban

配置mysql

配置 Executor Server

配置Web Server

Sqoop导出脚本

---

Azkaban

安装azkaban并改名

配置mysql

启动
[doudou@hadoop102 ~]$ mysql -uroot -p123456

	登陆 MySQL，创建 Azkaban 数据库
mysql> create database azkaban;

设置密码有效长度 4 位及以上
set global validate_password_length=4;

创建 azkaban 用户并赋予权限

设置密码有效长度 4 位及以上
mysql> set global validate_password_length=4;

设置密码策略最低级别
mysql> set global validate_password_policy=0;

创建 Azkaban 用户，任何主机都可以访问 Azkaban，密码是 000000
mysql> CREATE USER 'azkaban'@'%' IDENTIFIED BY '000000';

赋予 Azkaban 用户增删改查权限
mysql> GRANT SELECT,INSERT,UPDATE,DELETE ON azkaban.* to 'azkaban'@'%' WITH GRANT OPTION;

 创建azkaban表

mysql> use azkaban;
mysql>	source	/opt/module/azkaban/azkaban-db-3.84.4/create-all- sql-3.84.4.sql
mysql> quit;

更改 MySQL 包大小；防止 Azkaban 连接 MySQL 阻塞

[doudou@hadoop102 ~]$ sudo vim /etc/my.cnf

在[mysqld]下面加一行 max_allowed_packet=1024M
[mysqld] max_allowed_packet=1024M

重启mysql

[doudou@hadoop102 ~]$ sudo systemctl restart mysqld

配置 Executor Server

编辑 azkaban.properties

[atguigu@hadoop102	azkaban]$	vim	/opt/module/azkaban/azkaban- exec/conf/azkaban.properties

修改如下标红的属性

#... default.timezone.id=Asia/Shanghai #... azkaban.webserver.url=http://hadoop102:8081 executor.port=12321 #... database.type=mysql mysql.port=3306 mysql.host=hadoop102 mysql.database=azkaban mysql.user=azkaban mysql.password=000000 mysql.numconnections=100

同步 azkaban-exec 到所有节点

[doudou@hadoop102 conf]$ xsync /opt/module/azkaban/azkaban-exec/

必须进入到/opt/module/azkaban/azkaban-exec 路径，分别在三台机器上，启动executor server

[atguigu@hadoop102 azkaban-exec]$ bin/start-exec.sh 

[doudou@hadoop102 azkaban-exec]$  curl -G "hadoop102:12321/executor?action=activate" && echo
{"status":"success"}
[doudou@hadoop102 azkaban-exec]$  curl -G "hadoop103:12321/executor?action=activate" && echo
{"status":"success"}
[doudou@hadoop102 azkaban-exec]$  curl -G "hadoop104:12321/executor?action=activate" && echo
{"status":"success"}

配置Web Server

编辑 azkaban.properties

修改如下属性

... default.timezone.id=Asia/Shanghai ... database.type=mysql mysql.port=3306 mysql.host=hadoop102 mysql.database=azkaban mysql.user=azkaban mysql.password=000000 mysql.numconnections=100 ... azkaban.executorselector.filters=StaticRemainingFlowSize,CpuStatu s

修改 azkaban-users.xml 文件，添加 atguigu 用户

[doudou@hadoop102 conf]$ vim azkaban-users.xml 

<user password="123456" roles="admin" username="doudou"/>

启动

[doudou@hadoop102 azkaban-web]$ bin/start-web.sh

Sqoop导出脚本

[doudou@hadoop102 bin]$ vim hdfs_to_mysql.sh

#!/bin/bash

hive_db_name=gmall
mysql_db_name=gmall_report

export_data() {
/opt/module/sqoop/bin/sqoop export \
--connect "jdbc:mysql://hadoop102:3306/${mysql_db_name}?useUnicode=true&characterEncoding=utf-8"  \
--username root \
--password 123456 \
--table $1 \
--num-mappers 1 \
--export-dir /warehouse/$hive_db_name/ads/$1 \
--input-fields-terminated-by "\t" \
--update-mode allowinsert \
--update-key $2 \
--input-null-string '\\N'    \
--input-null-non-string '\\N'
}

case $1 in
  "ads_activity_stats" )
    export_data "ads_activity_stats" "dt,activity_id"
  ;;

  "ads_coupon_stats" )
    export_data "ads_coupon_stats" "dt,coupon_id"
  ;;

  "ads_order_by_province" )
    export_data "ads_order_by_province" "dt,recent_days,province_id"
  ;;

  "ads_order_spu_stats" )
    export_data "ads_order_spu_stats" "dt,recent_days,spu_id"
  ;;

  "ads_order_total" )
    export_data "ads_order_total" "dt,recent_days"
  ;;

  "ads_page_path" )
    export_data "ads_page_path" "dt,recent_days,source,target"
  ;;

  "ads_repeat_purchase" )
    export_data "ads_repeat_purchase" "dt,recent_days,tm_id"
  ;;

  "ads_user_action" )
    export_data "ads_user_action" "dt,recent_days"
  ;;

  "ads_user_change" )
    export_data "ads_user_change" "dt"
  ;;

  "ads_user_retention" )
    export_data "ads_user_retention" "create_date,retention_day"
  ;;

  "ads_user_total" )
    export_data "ads_user_total" "dt,recent_days"
  ;;

  "ads_visit_stats" )
    export_data "ads_visit_stats" "dt,recent_days,is_new,channel"
  ;;
  "all" )
    export_data "ads_activity_stats" "dt,activity_id"
    export_data "ads_coupon_stats" "dt,coupon_id"
    export_data "ads_order_by_province" "dt,recent_days,province_id"
    export_data "ads_order_spu_stats" "dt,recent_days,spu_id"
    export_data "ads_order_total" "dt,recent_days"
    export_data "ads_page_path" "dt,recent_days,source,target"
    export_data "ads_repeat_purchase" "dt,recent_days,tm_id"
    export_data "ads_user_action" "dt,recent_days"
    export_data "ads_user_change" "dt"
    export_data "ads_user_retention" "create_date,retention_day"
    export_data "ads_user_total" "dt,recent_days"
    export_data "ads_visit_stats" "dt,recent_days,is_new,channel"
  ;;
esac

[doudou@hadoop102 bin]$ chmod 777 hdfs_to_mysql.sh