airflow和MySQL_airflow快速安装——MySQL

1) 进行pip的更新，否则很多安装会报错

pip install --upgrade pip

pip install --upgrade setuptools

2) 安装开发库

yum install python3-devel

yum install libevent-devel

yum install mysql-devel

3) 创建airflow的MySQL数据库schema及用户

create database apache_airflow;

create user 'airflow'@'%' identified by '123123';

GRANT all privileges on apache_airflow.* TO 'airflow'@'%' IDENTIFIED BY '123123';

FLUSH PRIVILEGES;

set explicit_defaults_for_timestamp = 1; --这一行至关重要

4) 安装airflow

# airflow 需要 home 目录，默认是~/airflow，

# 但是如果你需要，放在其它位置也是可以的

# (可选)

export AIRFLOW_HOME = ~/airflow

# 使用 pip 从 pypi 安装

pip3 install apache-airflow

5) 编辑airflow.cfg配置文件，一般在~/airflow/airflow.cfg

#sql_alchemy_conn = sqlite:data/airflow/airflow.db

sql_alchemy_conn = mysql://airflow:123456@127.0.0.1:3306/apache_airflow

6) 初始化MySQL数据库

> airflow initdb

Traceback (most recent call last):

File "/usr/local/bin/airflow", line 25, in

from airflow.configuration import conf

File "/usr/local/lib/python3.6/site-packages/airflow/__init__.py", line 47, in

settings.initialize()

File "/usr/local/lib/python3.6/site-packages/airflow/settings.py", line 405, in initialize

configure_orm()

File "/usr/local/lib/python3.6/site-packages/airflow/settings.py", line 294, in configure_orm

engine = create_engine(SQL_ALCHEMY_CONN, connect_args=connect_args, **engine_args)

File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/__init__.py", line 500, in create_engine

return strategy.create(*args, **kwargs)

File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 87, in create

dbapi = dialect_cls.dbapi(**dbapi_args)

File "/usr/local/lib64/python3.6/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 118, in dbapi

return __import__("MySQLdb")

ModuleNotFoundError: No module named 'MySQLdb'

说明缺少MySQL组件

> yum install -y mysql-devel

> pip3 install pymysql

> pip3 install mysql

> airflow initdb

DB: mysql://airflow:***@10.1.11.12:3306/shark_airflow

[2020-08-27 09:25:52,760] {db.py:378} INFO - Creating tables

INFO [alembic.runtime.migration] Context impl MySQLImpl.

INFO [alembic.runtime.migration] Will assume non-transactional DDL.

WARNI [airflow.models.crypto] cryptography not found - values will not be stored encrypted.

/usr/local/lib/python3.6/site-packages/airflow/models/dag.py:1342: PendingDeprecationWarning: The requested task could not be added to the DAG because a task with task_id create_tag_template_field_result is already in the DAG. Starting in Airflow 2.0, trying to overwrite a task will raise an exception.

category=PendingDeprecationWarning)

Done.

7) 启动web服务器和调度器

> airflow webserver -p 8080

> airflow scheduler

8) 防止密码明文存储

pip3 install cryptography