Ubuntu 安装 Apache Airflow

先更新系统和安装 pip3

sudo apt update
sudo apt upgrade
sudo apt install python-setuptools
sudo apt install python3-pip

安装基本库

sudo apt install libmysqlclient-dev
sudo apt install libssl-dev
sudo apt install libkrb5-dev

更新本地 pip3 package

注: 要加 ~/.profile export PATH=/home/kazuya/.Local/bin:$PATH

pip3 list --outdated
pip3 list --outdated --format=freeze | grep -v '^\-e' | cut -d = -f 1 | xargs -n1 pip3 install -U

安装 python 虚拟环境

sudo apt install python3-virtualenv

建立 airflow 虚拟环境 和 数据存放 目录

sudo mkdir /opt/airflow
sudo chown -R kazuya:kazuya /opt/airflow

输出 AIRFLOW_HOME

vi ~/.profile

export AIRFLOW_HOME=/opt/airflow

建立 airflow 虚拟环境  

sudo virtualenv /opt/airflow_env

sudo chown -R kazuya:kazuya /opt/airflow_env

起动 python 虚拟环境

cd /opt/airflow_env/bin
source activate

更新虚拟环境 pip3 package 和 安装 apache-airflow

pip3 list --outdated
pip3 list --outdated --format=freeze | grep -v '^\-e' | cut -d = -f 1 | xargs -n1 pip3 install -U

pip3 install apache-airflow
pip3 install typing_extensions

查看 apache-airflow 安装路径 Location: /opt/airflow_env/lib/python3.8/site-packages

$ pip3 show apache-airflow
Name: apache-airflow
Version: 2.1.4
Summary: Programmatically author, schedule and monitor data pipelines
Home-page: https://airflow.apache.org/
Author: Apache Software Foundation
Author-email: dev@airflow.apache.org
License: Apache License 2.0
Location: /opt/airflow_env/lib/python3.8/site-packages
Requires: flask-wtf, dill, python-dateutil, sqlalchemy, sqlalchemy-jsonfield, pendulum, python-nvd3, markdown, apache-airflow-providers-ftp, pygments, docutils, cattrs, pyjwt, importlib-metadata, termcolor, pandas, argcomplete, apache-airflow-providers-imap, numpy, blinker, openapi-spec-validator, setproctitle, httpx, markupsafe, importlib-resources, flask-caching, tabulate, psutil, gunicorn, iso8601, alembic, pyyaml, attrs, croniter, flask-appbuilder, jsonschema, python-daemon, flask, swagger-ui-bundle, lazy-object-proxy, itsdangerous, jinja2, apache-airflow-providers-http, python3-openid, flask-login, cryptography, colorlog, marshmallow-oneofschema, unicodecsv, clickclick, python-slugify, tenacity, inflection, lockfile, rich, apache-airflow-providers-sqlite, werkzeug, graphviz
Required-by:

初始数据库和建立管理账号

airflow db init

airflow users create --username admin --firstname kazuya --lastname kazuya --role Admin --email XXXXXXX@gmail.com
Password

起动 airflow 排程器

airflow scheduler

再开一个 session 和 source activate 虚拟环境 起动 airflow 服务器

airflow webserver -p 8080

 

 

Ctrl-C 停止服务

Ctrl-C to quit webserver
Ctrl-C to quit scheduler

退出 python 虚拟环境

deactivate