Airflow2.0+celery+redis任务调度部署及使用

Airflow任务调度

 

（本文档内容有同事贡献部分，该部分标记为蓝色，对同事表示感谢）

 

目录

一、环境

二、基础参数

三、任务类型

四、使用步骤

五、需要解决的问题（绿色表示已解决）

六、注意事项

---

一、环境

版本：airflow 2.0.0；python 3.6

部署方式：集群部署，运行在anaconda3的虚拟环境 (airflow)

* 节点7 [webserver、schuduler、worker]

* 节点8 [worker]

* 节点9 [worker、schuduler]

官网文档（最新）：http://airflow.apache.org/docs/apache-airflow/stable/start.html

非官方翻译中文文档（1.10.2）：https://airflow.apachecn.org/#/

二、基础参数

default_args = {

'owner': '***',

'start_date': days_ago(1),

'email': ['xxx@qq.com'],

'email_on_failure': True,

'email_on_retry': False,

'retries': 1,

'retry_delay': timedelta(seconds=50),

'pool': 'test',

'priority_weight': 100

}

baseoperator(

:param task_id: a unique, meaningful id for the task

:type task_id: str

:param owner: the owner of the task, using the unix username is recommended

:type owner: str

:param email: the 'to' email address(es) used in email alerts. This can be a

single email or multiple ones. Multiple addresses can be specified as a

comma or semi-colon separated string or by passing a list of strings.

:type email: str or list[str]

:param email_on_retry: Indicates whether email alerts should be sent when a

task is retried

:type email_on_retry: bool

:param email_on_failure: Indicates whether email alerts should be sent when

a task failed

:type email_on_failure: bool

:param retries: the number of retries that should be performed before

failing the task

:type retries: int

:param retry_delay: delay between retries

:type retry_delay: datetime.timedelta

:param retry_exponential_backoff: allow progressive longer waits between

retries by using exponential backoff algorithm on retry delay (delay

will be converted into seconds)

:type retry_exponential_backoff: bool

:param max_retry_delay: maximum delay interval between retries

:type max_retry_delay: datetime.timedelta

:param start_date: The ``start_date`` for the ta