docker-compose之airflow搭建

最新推荐文章于 2024-07-18 11:04:42 发布
最新推荐文章于 2024-07-18 11:04:42 发布
阅读量489 收藏 5
点赞数 5
文章标签： docker 容器 运维
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/m0_60622799/article/details/139173910
版权 
# Basic Airflow cluster configuration for CeleryExecutor with Redis and PostgreSQL.
#
# WARNING: This configuration is for local development. Do not use it in a production deployment.
#
# This configuration supports basic configuration using environment variables or an .env file
# The following variables are supported:
#
# AIRFLOW_IMAGE_NAME           - Docker image name used to run Airflow.
#                                Default: apache/airflow:latest-python3.11
# AIRFLOW_UID                  - User ID in Airflow containers
#                                Default: 50000
# AIRFLOW_PROJ_DIR             - Base path to which all the files will be volumed.
#                                Default: .
# Those configurations are useful mostly in case of standalone testing/running Airflow in test/try-out mode
#
# _AIRFLOW_WWW_USER_USERNAME   - Username for the administrator account (if requested).
#                                Default: airflow
# _AIRFLOW_WWW_USER_PASSWORD   - Password for the administrator account (if requested).
#                                Default: airflow
# _PIP_ADDITIONAL_REQUIREMENTS - Additional PIP requirements to add when starting all containers.
#                                Use this option ONLY for quick checks. Installing requirements at container
#                                startup is done EVERY TIME the service is started.
#                                A better way is to build a custom image or extend the official image
#                                as described in https://airflow.apache.org/docs/docker-stack/build.html.
#                                Default: ''
#
# Feel free to modify this file to suit your needs.
---
x-airflow-common:
  &airflow-common
  # In order to add custom dependencies or upgrade provider packages you can use your extended image.
  # Comment the image line, place your Dockerfile in the directory where you placed the docker-compose.yaml
  # and uncomment the "build" line below, Then run `docker-compose build` to build the images.
  #  AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
  #  mysql+pymysql://airflow:airflow123456@127.0.0.1:3306/airflow?use_unicode=true&charset=utf8
  #  AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
  #  AIRFLOW__CORE__EXECUTOR: SequentialExecutor
  #  AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
  image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:latest-python3.11}
  # build: .
  environment:
    &airflow-common-env
    AIRFLOW__CORE__EXECUTOR: CeleryExecutor
    AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
    AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
    AIRFLOW__CELERY__BROKER_URL: redis://:@127.0.0.1:6379/0
    AIRFLOW__CORE__FERNET_KEY: ''
    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
    AIRFLOW__CORE__LOAD_EXAMPLES: 'true'
    AIRFLOW__API__AUTH_BACKENDS: 'airflow.api.auth.backend.basic_auth,airflow.api.auth.backend.session'
    # yamllint disable rule:line-length
    # Use simple http server on scheduler for health checks
    # See https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/check-health.html#scheduler-health-check-server
    # yamllint enable rule:line-length
    AIRFLOW__SCHEDULER__ENABLE_HEALTH_CHECK: 'true'
    # WARNING: Use _PIP_ADDITIONAL_REQUIREMENTS option ONLY for a quick checks
    # for other purpose (development, test and especially production usage) build/extend Airflow image.
    # /usr/local/lib/python3.11/site-packages
    _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}
  volumes:
    - /data/docker-data/airflow/dags:/opt/airflow/dags
    - /data/docker-data/airflow/logs:/opt/airflow/logs
    - /data/docker-data/airflow/conf:/opt/airflow/config
    - /data/docker-data/airflow/plugins:/opt/airflow/plugins
#    - /data/soft/python3.11/lib/python3.11/site-packages:/home/airflow/.local/lib/python3.11/site-packages
  user: "${AIRFLOW_UID:-50000}:0"
  depends_on:
    &airflow-common-depends-on
#    redis:
#      condition: service_healthy
    postgres:
      condition: service_healthy

services:
  postgres:
    image: postgres:13
    environment:
      POSTGRES_USER: airflow
      POSTGRES_PASSWORD: airflow
      POSTGRES_DB: airflow
    ports:
      - "5432:5432"
    volumes:
      - /data/docker-data/airflow/data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "airflow"]
      interval: 10s
      retries: 5
      start_period: 5s
    restart: always

#  redis:
#    image: redis:latest
#    expose:
#      - 6379
#    healthcheck:
#      test: ["CMD", "redis-cli", "ping"]
#      interval: 10s
#      timeout: 30s
#      retries: 50
#      start_period: 30s
#    restart: always
#    privileged: true
#    ports:
#      - 6379:6379
#    environment:
#      TZ: Asia/Shanghai
#    volumes:
#      - /data/docker-data/redis/data:/data
#      - /data/docker-data/redis/conf/redis.conf:/usr/local/redis/conf/redis.conf
#      - /data/docker-data/redis/logs:/logs
#    #    command: ["redis-server","/usr/local/redis/conf/redis.conf"]
#    command: /bin/sh -c "echo 'vm.overcommit_memory = 1' >> /etc/sysctl.conf && redis-server /usr/local/redis/conf/redis.conf "

  airflow-webserver:
    <<: *airflow-common
    command: webserver
    ports:
      - "9999:8080"
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://127.0.0.1:9999/health"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 30s
    restart: always
    depends_on:
      <<: *airflow-common-depends-on
      airflow-init:
        condition: service_completed_successfully

  airflow-scheduler:
    <<: *airflow-common
    command: scheduler
    ports:
      - "9974:8974"
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://127.0.0.1:9974/health"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 30s
    restart: always
    depends_on:
      <<: *airflow-common-depends-on
      airflow-init:
        condition: service_completed_successfully

  airflow-worker:
    <<: *airflow-common
    command: celery worker
    healthcheck:
      # yamllint disable rule:line-length
      test:
        - "CMD-SHELL"
        - 'celery --app airflow.providers.celery.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}" || celery --app airflow.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}"'
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 30s
    environment:
      <<: *airflow-common-env
      # Required to handle warm shutdown of the celery workers properly
      # See https://airflow.apache.org/docs/docker-stack/entrypoint.html#signal-propagation
      DUMB_INIT_SETSID: "0"
    restart: always
    depends_on:
      <<: *airflow-common-depends-on
      airflow-init:
        condition: service_completed_successfully

  airflow-triggerer:
    <<: *airflow-common
    command: triggerer
    healthcheck:
      test: ["CMD-SHELL", 'airflow jobs check --job-type TriggererJob --hostname "$${HOSTNAME}"']
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 30s
    restart: always
    depends_on:
      <<: *airflow-common-depends-on
      airflow-init:
        condition: service_completed_successfully

  airflow-init:
    <<: *airflow-common
    entrypoint: /bin/bash
    # yamllint disable rule:line-length
    command:
      - -c
      - |
        if [[ -z "${AIRFLOW_UID}" ]]; then
          echo
          echo -e "\033[1;33mWARNING!!!: AIRFLOW_UID not set!\e[0m"
          echo "If you are on Linux, you SHOULD follow the instructions below to set "
          echo "AIRFLOW_UID environment variable, otherwise files will be owned by root."
          echo "For other operating systems you can get rid of the warning with manually created .env file:"
          echo "    See: https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html#setting-the-right-airflow-user"
          echo
        fi
        one_meg=1048576
        mem_available=$$(($$(getconf _PHYS_PAGES) * $$(getconf PAGE_SIZE) / one_meg))
        cpus_available=$$(grep -cE 'cpu[0-9]+' /proc/stat)
        disk_available=$$(df / | tail -1 | awk '{print $$4}')
        warning_resources="false"
        if (( mem_available < 4000 )) ; then
          echo
          echo -e "\033[1;33mWARNING!!!: Not enough memory available for Docker.\e[0m"
          echo "At least 4GB of memory required. You have $$(numfmt --to iec $$((mem_available * one_meg)))"
          echo
          warning_resources="true"
        fi
        if (( cpus_available < 2 )); then
          echo
          echo -e "\033[1;33mWARNING!!!: Not enough CPUS available for Docker.\e[0m"
          echo "At least 2 CPUs recommended. You have $${cpus_available}"
          echo
          warning_resources="true"
        fi
        if (( disk_available < one_meg * 10 )); then
          echo
          echo -e "\033[1;33mWARNING!!!: Not enough Disk space available for Docker.\e[0m"
          echo "At least 10 GBs recommended. You have $$(numfmt --to iec $$((disk_available * 1024 )))"
          echo
          warning_resources="true"
        fi
        if [[ $${warning_resources} == "true" ]]; then
          echo
          echo -e "\033[1;33mWARNING!!!: You have not enough resources to run Airflow (see above)!\e[0m"
          echo "Please follow the instructions to increase amount of resources available:"
          echo "   https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html#before-you-begin"
          echo
        fi
        mkdir -p /sources/logs /sources/dags /sources/plugins
        chown -R "${AIRFLOW_UID}:0" /sources/{logs,dags,plugins}
        exec /entrypoint airflow version
    # yamllint enable rule:line-length
    environment:
      <<: *airflow-common-env
      _AIRFLOW_DB_MIGRATE: 'true'
      _AIRFLOW_WWW_USER_CREATE: 'true'
      _AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
      _AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}
      _PIP_ADDITIONAL_REQUIREMENTS: ''
    user: "0:0"
    volumes:
      - /data/docker-data/airflow:/sources

  airflow-cli:
    <<: *airflow-common
    profiles:
      - debug
    environment:
      <<: *airflow-common-env
      CONNECTION_CHECK_MAX_COUNT: "0"
    # Workaround for entrypoint issue. See: https://github.com/apache/airflow/issues/16252
    command:
      - bash
      - -c
      - airflow

  # You can enable flower by adding "--profile flower" option e.g. docker-compose --profile flower up
  # or by explicitly targeted on the command line e.g. docker-compose up flower.
  # See: https://docs.docker.com/compose/profiles/
  flower:
    <<: *airflow-common
    command: celery flower
    profiles:
      - flower
    ports:
      - "9955:5555"
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://127.0.0.1:9955/"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 30s
    restart: always
    depends_on:
      <<: *airflow-common-depends-on
      airflow-init:
        condition: service_completed_successfully








## 方式1----本地安装：
### airflow
# yum install postgresql-devel
# yum install -y gcc make libffi-devel zlib*
#pip3.11 install connexion[swagger-ui]
#pip3.11 install psycopg2 virtualenv kubernetes pandas
#pip3.11 install mysqlclient -i https://pypi.tuna.tsinghua.edu.cn/simple
#pip3.11 install apache-airflow initdb
#
### kill掉之前的线程
#ps -aux | grep airflow | awk -F' '  '{ print $2 }' | xargs kill -9
#ps -aux | grep gunicorn | awk -F' '  '{ print $2 }' | xargs kill -9
#
### airflow的前端页面入口
#http://127.0.0.1:9090
#Login with username: admin  password: 4tGWQnctusgf9vHW

####################针对已经有历史数据重置的场景##########################
# airflow db reset
# airflow db init
# airflow webserver -D
# airflow scheduler -D  #  nohup airflow scheduler >>airflow-scheduler.log 2>& 1 &
# airflow users create --role Admin --username airflow --email admin --firstname admin --lastname admin --password airflow

# 通过命令查看并修改dags的目录：cat airflow.cfg | grep dags_folder
# python脚本编写的DAG只有放在该目录下才能够在web界面正常显示

# 方式2----docker-compose安装: 如果启动不成功，可能需要清除目录/data/docker-data/airflow/data下的数据
# docker-compose -f docker_compose_airflow.yml down
# http://127.0.0.1:9999


#
## 守护进程运行webserver
#$ airflow webserver -D
#
## 守护进程运行调度器
#$ airflow scheduler -D
#
## 守护进程运行调度器
#$ airflow worker -D
#
## 守护进程运行celery worker并指定任务并发数为1
#$ airflow worker -c 1 -D
#
## 暂停任务
#$ airflow pause $dag_id　
#
## 取消暂停，等同于在管理界面打开off按钮
#$ airflow unpause $dag_id
#
## 查看task列表
#$ airflow list_tasks $dag_id
#
## 清空任务实例
#$ airflow clear $dag_id
#
## 运行整个dag文件
#$ airflow trigger_dag $dag_id -r $RUN_ID -e $EXEC_DATE
#
## 运行task
#$ airflow run $dag_id $task_id $execution_date

m0_60622799
关注
  • 5
    点赞
  • 踩
  • 5
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
airflow系列教程(七) 使用docker-compose进行Airflow的容器化部署
大数据指北
03-10 1472
经过前面章节可视化工具搭建airflow容器，虽然有非常直观，但是在真实的内网环境中，往往会没有端口与互联网进行映射，所以非常有必要知道docker-compose的方式进行airflow容器化部署 文章目录前言一、Airflow的镜像包二、docker-compse文件三、部署airflow总结 前言 采用docker-compose的方式进行ariflow的容器化部署，非常方便的一键启动，并且使用local模式运行 一、Airflow的镜像包 airflow1.10.6版本的tar包以及相关文件上
利用Docker安装Airflow
Cmmmmm丶的博客
12-21 682
利用Docker安装Airflow
Airflow的docker-compose.yaml安装
weixin_43651674的博客
01-25 1397
由于airflow的组件多，相互关联性强，特别在使用celery调度方式的情况下，另外加上airflow的版本迭代确实快。所以本地固有安装不是一种良好的维护方式。所以特此推荐使用docker-compose.yaml进行安装，一站式启停。
Airflow安装
学而时习之，不亦说乎？温故而知新，可以为师矣！
12-06 427
前提条件需要安装 Docker 和 Docker-compose，作者用的是 Mac 系统安装的 Airflow。
airflow v2.6.0 Docker Compose 部署
奔跑的蜗牛的博客
06-19 1923
本文主要记载 airflow 容器化部署步骤以及需要注意的事项。
docker部署Airflow(修改URL-path、更换postgres --＞myslq数据库、LDAP登录)
jiangbenchu的博客
03-04 6256
Airflow什么是 Airflow？Airflow 的架构Airflow 解决哪些问题一、docker-compose 安装airflow（postgres）1、创建启动文件airflow-docker-compose.yml.1.1、添加挂载卷，需要修改airflow-docker-compose.yml的位置2、创建本地配置文件airflow.cfg2.1、如果想修改WEB URL地址，需要修改airflow.cfg中以下两个地方3、之后up -d直接启动即可web访问地址：二、存储数据库更换post
【Docker】Airflow 容器化部署
Qinghub‘博客
03-21 1879
Apache Airflow 是一种以有向无环图 (DAG) 形式表达和执行工作流程的工具。它包括用于计划任务、监视任务进度和处理任务依赖性的实用程序。以编程方式编写，安排和监视工作流的平台。 Airflow scheduler程序在遵循指定的依赖项，同时在一组工作线程上执行任务。丰富的命令实用程序使在DAG上执行复杂的调度变的轻而易举。丰富的用户界面使查看生产中正在运行的管道，监视进度以及需要时对问题进行故障排除变的容易。
dockercompose--airflow
大数据_大龙
04-12 500
dockercompose--airflow
Docker尝鲜之Airflow快速安装
热门推荐
走过路过
12-01 1万+
Docker尝鲜之Airflow快速安装 个人初步理解使用Docker应该尽量满足数据和应用隔离，数据尽量有mount在宿主机上；另外，与虚拟机相比，容器有一个很大的差异，它们被设计用来运行单进程，无法很好地模拟一个完整的环境，所以不建议一个容器塞太多的应用——容器的启动是非常廉价的。Docker设计者极力推崇“一个容器一个进程的方式”，所以遵循这样的应用原则就没错了。另外，docker exe
airflow-docker：这是我在Windows 10 WSL2上使用docker-compose进行的Apache Airflow Local开发设置。 它还将包括一些示例DAG和工作流程
02-10
其中包含以下服务的定义： 气流调度器气流网络服务器气流工人airflow-init-db-初始化数据库airflow-init-user-创建气流用户。 花Redis 我必须将原始Yaml中的airflow airflow-init拆分为两个单独的步骤，以使其在...
airflow:docker-compose产生气流
03-19
阿帕奇气流
docker-compose-airflow-authentication:模板通过docker-compose通过用户和密码身份验证在docker中启动气流
04-10
docker-compose-airflow-authentication
bitnami-docker-airflow:适用于Apache Airflow的Bitnami Docker映像
03-05
DRDocker撰写$ curl -LO https://raw.githubusercontent.com/bitnami/bitnami-docker-airflow/master/docker-compose.yml$ docker-compose up 您可以在“部分中找到默认凭据和可用的配置选项。为什么要使用Bitnami ...
bitnami-docker-airflow-worker:适用于Apache Airflow Worker的Bitnami Docker映像
03-06
DRDocker撰写$ curl -LO https://raw.githubusercontent.com/bitnami/bitnami-docker-airflow-worker/master/docker-compose.yml$ docker-compose up 您可以在“部分中找到默认凭据和可用的配置选项。为什么要使用...
airflow 在 docker 中的安装与使用
fwzzzzz的博客
05-21 4589
文章目录airflow 在 docker 中的安装与使用简介airflow 镜像环境变量配置信息dagspip 包的安装挂载目录部署airflow.cfg 配置文件修改示例加载邮件时区登录配置其他配置信息目录结构与文件.envDockerfiledocker-compose.yaml启动查看 dags 信息扩展钉钉机器人报警邮箱报警异常出现AttributeError: module ‘typing‘ has no attribute ‘_ClassVar‘错误airflow.cfg airflow 在 d
【入门Airflow】 使用Docker在本地快速搭建Airflow
mkdir700的博客
12-16 7162
本文主要参考官方文档编写。我将顺着官方文档的流程在本地安装Airflow ，并且附上安装过程的截图。 其实官方文档步骤写得都是比较详细，奈何这英语能力真的拉跨。 不过还是建议大家在使用过程中遇到的任何问题，优先去查看官方文档，有奇效。 ​ 官方文档： https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html 生产准备 这里Airflow官方建议我们不要使用当前这种“快速入门”的方式用在生产环境。本文的这种方式就供我们
利用docker和docker-compose部署airflow
最新发布
qq_42048263的博客
07-18 285
参考https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html。在airflow 控制台中选择Admin下的Connections。配置连接信息，配置好之后点击保存。登录之后界面显示效果如下。内存需要至少有4GB。查看容器运行是否正常。
漏洞复现——Apache Airflow 示例dag中的命令注入（CVE-2020-11978）
m0_65149086的博客
05-26 435
懵逼日常
docker-compose -f docker-compose-cli.yaml up
04-17
docker-compose -f docker-compose-cli.yaml up命令用于启动使用docker-compose工具编排的多个容器。通过指定一个特定的docker-compose文件（在这里是docker-compose-cli.yaml），可以定义和配置多个容器的运行方式和依赖关系。 具体来说，该命令会读取docker-compose-cli.yaml文件中的配置信息，并根据配置启动相应的容器。这些容器可以是不同的服务，例如数据库、Web服务器、消息队列等。通过docker-compose工具，可以方便地定义容器之间的网络连接、数据卷挂载、环境变量设置等。 在执行docker-compose -f docker-compose-cli.yaml up命令后，docker-compose会根据配置文件中的定义，自动创建并启动相应的容器。如果容器已经存在，则会重新启动它们。启动后，可以通过docker命令查看容器的状态、日志等信息。
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值