AirFlow之安装

文档

• 中文
• 英文

Docker安装

• GitHub
• 自己build

直接安装

• 准备数据库
• 设置基础环境

# 直接安装会报错,需要导入如下设置
export SLUGIFY_USES_TEXT_UNIDECODE=yes

• install

pip install apache-airflow

• 生成airflow.cfg文件
  • 设置airflow的HOME目录，并将其写入当前用户的环境变量中

  export AIRFLOW_HOME=~/airflow
  

  • 执行生成文件

  airflow version
  

• 修改airflow.cfg

  # 数据库连接字符串
  sql_alchemy_conn = sql_alchemy_conn = mysql+pymysql://user_name:passwd@ip:port/db
  
  # executor
  executor = LocalExecutor
  
  # 设置时区
  default_timezone = Asia/Shanghai
  
  # 隐藏官方示例
  load_examples = False
  

• 设置本地时区
  • 修改：~/anaconda3/envs/airflow/lib/python3.7/site-packages/airflow/utils/timezone.py

  # 第27行
  # utc = pendulum.timezone('UTC')
  from airflow import configuration as conf
  
  try:
   tz = conf.get("core", "default_timezone")
   if tz == "system":
     utc = pendulum.local_timezone()
   else:
     utc = pendulum.timezone(tz)
  except Exception:
   pass
  
  #第70行
  # d = dt.datetime.utcnow()
  d = dt.datetime.now()
  

  • 修改：~/anaconda3/envs/airflow/lib/python3.7/site-packages/airflow/utils/sqlalchemy.py

  # 第37行
  # utc = pendulum.timezone('UTC')
  from airflow import configuration as conf
  
  try:
   tz = conf.get("core", "default_timezone")
   if tz == "system":
     utc = pendulum.local_timezone()
   else:
     utc = pendulum.timezone(tz)
  except Exception:
   pass
  

  • 修改：~/anaconda3/envs/airflow/lib/python3.7/site-packages/airflow/www/templates/master.html

  <!-- 第33行 --> 
  <!-- var UTCseconds = (x.getTime() + x.getTimezoneOffset()*60*1000); -->
  var UTCseconds = x.getTime();
  
  <!-- 第37行--> 
  <!-- "timeFormat":"H:i:s %UTC%",--> 
  "timeFormat":"H:i:s",
  

• db初始化

airflow initdb

• 用户认证
  • 安装插件

  pip install "apache-airflow[password]"
  

  • 修改airflow.cfg

  [webserver]
  authenticate = True
  auth_backend = airflow.contrib.auth.backends.password_auth
  

  • 增加用户（Python下执行）

  from airflow import configuration as conf
  
  try:
   tz = conf.get("core", "default_timezone")
   if tz == "system":
    utc = pendulum.local_timezone()
   else:
    utc = pendulum.timezone(tz)
  except Exception:
   pass
  

• 启动服务

# 启动定时器
airflow scheduler
# 启动 web 服务器，默认端口是 8080（可以在airflow.cfg文件中修改）
airflow webserver -p 8081