如何在Kubernetes 上部署Airflow

本文介绍了在Kubernetes上部署Airflow的详细步骤,包括使用Kind创建K8s集群,通过Kubectl管理集群,安装Airflow以及使用Helm简化部署流程。重点讲述了如何使用KubernetesOperator和KubernetesExecutor来调度任务。
摘要由CSDN通过智能技术生成

1 什么是airflow?

  • It’s a data orchestration and scheduling platform
  • it’s a tool to manage your data-flows and data operations
  • allow running your task in a distributed manner
  • manage your task dependency within the dag.

个人理解为一个任务规划平台,利用Dags( directed acyclic graph)规划task的运行计划以及运行方式. 详细的内容参照123,这里不再赘述,拷贝的再全部都不如先把它跑起来,慢慢体会它的优点和缺点。

2 什么是kubernetes?

1 it is an open source platform for managing containerized workloads and services and it operates at the container level rather than at the hardware level
2 it provides you with a framework to run distributed systems resiliently.
3 It takes care of scaling and failover for your application, provides deployment patterns
4 service discovery and load balancing
5 storage orchestration Kubernetes allows you to automatically mount a storage system of your choice, such as local storages, public cloud providers, and more.
6 Automated rollouts and rollbacks
7 Self-healing Kubernetes restarts containers that fail, replaces containers, kills containers that don’t respond to your user-defined health check, and doesn’t advertise them to clients until they are ready to serve.
9 Secret and configuration management Kubernetes lets you store and manage sensitive information, such as passwords, OAuth tokens, and SSH keys.
10 comprises a set of independent, composable control processes that continuously drive the current state towards the provided

Kubernetes (k8s) 是一个开源的容器编排系统,常用于自动化容器化应用的部署、扩展和管理。要将Apache Airflow部署到k8s上,你可以按照以下步骤操作: 1. **安装Kubernetes集群**:确保你有一个运行中的k8s集群,如果没有,可以通过官方文档或服务商提供的服务安装。 2. **创建Docker镜像**:Airflow应该被打包成Docker镜像,包含所有依赖。你可以使用Dockerfile创建一个自定义的镜像。 ```bash # Dockerfile FROM python:3.8-slim-buster COPY requirements.txt ./ RUN pip install --no-cache-dir -r requirements.txt COPY . /app WORKDIR /app EXPOSE 8080 CMD ["airflow", "webserver"] ``` 3. **创建Kubernetes部署文件**(如:`airflow-deployment.yaml`): - 定义Deployment来描述Pod的数量和配置 - 使用Service来暴露应用的端口 ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: airflow-deployment spec: replicas: 1 selector: matchLabels: app: airflow template: metadata: labels: app: airflow spec: containers: - name: airflow image: <your-airflow-docker-image> ports: - containerPort: 8080 --- apiVersion: v1 kind: Service metadata: name: airflow-service spec: selector: app: airflow ports: - protocol: TCP port: 80 targetPort: 8080 type: LoadBalancer ``` 4. **部署到k8s**: 使用`kubectl apply -f airflow-deployment.yaml`命令将配置文件应用到集群。 5. **验证和访问**: 查看Pod状态确认部署成功,然后通过Service的外部IP或域名访问Airflow UI。
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值