Airflow【实践 01】Airflow官网+自测源代码举例（简化安装+官方及自测python代码

最新推荐文章于 2024-05-12 21:23:34 发布

解佳思提

最新推荐文章于 2024-05-12 21:23:34 发布

阅读量446

点赞数 4

分类专栏：程序员文章标签： python 开发语言

本文链接：https://blog.csdn.net/2301_77033672/article/details/138219345

版权

程序员专栏收录该内容

223 篇文章 1 订阅

订阅专栏

安装任务所需的依赖【代码执行所需要的依赖】

conda install scikit-learn

官方举例文件demo.py放置到dags_folder路径下

文件内容如下：

from datetime import datetime

from airflow import DAG
from airflow.decorators import task
from airflow.operators.bash import BashOperator

# A DAG represents a workflow, a collection of tasks
with DAG(dag_id="demo", start_date=datetime(2022, 1, 1), schedule="0 0 \* \* \*") as dag:

    # Tasks are represented as operators
    hello = BashOperator(task_id="hello", bash_command="echo hello")

    @task()
    def airflow():
        print("airflow")

    # Set dependencies between tasks
    hello >> airflow()

刷新页面即可看到DAG，点击即可执行，效果如下：

3.自测

创建测试文件airflow_test.py并放置到dags_folder目录下

文件内容如下：

from datetime import timedelta
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from airflow.utils.dates import days_ago

# DAG属性定义
default_args = {
    'owner': 'airFlowTest',
    'depends\_on\_past': False,
    'start\_date': days_ago(31),
    # 填入邮箱，方便失败、重试时发送邮件
    'email': ['xxxxx@qq.com'],
    # 失败时发邮件告警
    'email\_on\_failure': True,
    'email\_on\_retry': False,
    # 重试次数
    'retries': 1,
    'retry\_delay': timedelta(minutes=2),
    # 'queue': 'bash\_queue',
    # 'pool': 'backfill',
    # 'priority\_weight': 10,
    # 'end\_date': datetime(2016, 1, 1),bu
    # 'wait\_for\_downstream': False,
    # 'dag': dag,
    # 'sla': timedelta(hours=2),
    # 'execution\_timeout': timedelta(seconds=300),
    # 'on\_failure\_callback': some\_function,
    # 'on\_success\_callback': some\_other\_function,
    # 'on\_retry\_callback': another\_function,
    # 'sla\_miss\_callback': yet\_another\_function,
    'trigger\_rule': 'all\_success'
}

# 定义dag
dag = DAG(
    'air\_test',
    default\_args=default_args,
    description='A simple airflow test',
    schedule\_interval=timedelta(days=1),
)

step_one = BashOperator(
    task\_id='step\_one',
    bash\_command='echo step\_one over! >> /root/airflow/file/airflowtest.log',
    dag=dag,
)

step_two = BashOperator(
    task\_id='step\_two',
    depends\_on\_past=False,
    bash\_command='echo step\_two over! >> /root/airflow/file/airflowtest.log',
    retries=3,
    dag=dag,
)

step_three = BashOperator(
    task\_id='step\_three',
    depends\_on\_past=False,
    bash\_command='echo step\_three over! >> /root/airflow/file/airflowtest.log',
    retries=3,
    dag=dag,
)

step_four = BashOperator(
    task\_id='step\_four',
    depends\_on\_past=False,
    bash\_command='echo step\_four over! >> /root/airflow/file/airflowtest.log',
    retries=3,
    dag=dag,
)

step_one >> step_two >> step_four