离线数仓datax+shell+airflow调度

airflow+shell离线数仓调度

外部配置文件

1.将数据库连接等放入配置文件中

config.py
ROOT_DIR = /home/dag/airflow/dags

config.sh
BUSS_IP="10.10.1.1"
BUSS_PORT="3306"
BUSS_USER="user"
BUSS_PWD="pwd"
BUSS_DB="db"

编写airflow调度脚本

from datetime import datetime
from ariflow import DAG
from ariflow.operators.bash import BashOperator
from config.config import *

with DAG(
	default_args = {
		'owner' : 'user-test',
		'queue' : 'queue1'
		'retries' : 3
	},
	dag_id = 'dag_name1',
	start_time=datetime(2022,04,14),
	schedule_interval="30 00 * * *",
	catchup=False,
	tags=['ods','buss_name'],
	concurrency=3,
	user_defined_macros={
		'BASE_ROOT_DIR' : ROOT_DIR + "/o