Airflow安装说明

1、airflow安装

为了后期安装其它软件的需要，在此先安装必要的依赖包(我曾经跨过山和大海，也穿过人山人海...)，重要的事情只说一遍哦
yum -y groupinstall "Development tools"
yum -y install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel xz-devel libffi-devel

# airflow needs a home, ~/airflow is the default,
# but you can lay foundation somewhere else if you prefer
# (optional)
export AIRFLOW_HOME=~/airflow

# install from pypi using pip
pip3 install apache-airflow

# initialize the database
airflow initdb

# start the web server, default port is 8080
airflow webserver -p 8080

# start the scheduler
airflow scheduler

# visit localhost:8080 in the browser and enable the example dag in the home page

 

 

2、设置防火墙（如果不是本地访问airflow）

直接拉取firewall-cmd:
yum install firewalld systemd -y
 
查看服务器上是否安装了firewall
systemctl status firewalld
 
开启防火墙
systemctl start firewalld.service
 
关闭防火墙
systemctl stop firewalld.service
 
设置开机自启
systemctl enable firewalld.service
 
设置关闭开机自启动
systemctl disable firewalld.service
 
在不改变状态的条件下重新加载防火墙
firewall-cmd --reload
 
查看已开放的端口
netstat -anp
 
查询指定8080端口是否开放
firewall-cmd --query-port=8080/tcp
 
yes 表示开放 no表示不开放

3、访问airflow

 

ip:8080

 

 

 

4、配置CeleryExecutor

1）安装MariaDB10.3

vim /etc/yum.repos.d/MariaDB.repo
添加如下内容：
[mariadb]
name = MariaDB
baseurl = http://yum.mariadb.org/10.3/centos7-amd64
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1

yum clean all
yum makecache

Importing the MariaDB GPG Public Key
rpm --import https://yum.mariadb.org/RPM-GPG-KEY-MariaDB

安卓MariaDB10.3
yum install MariaDB-server galera MariaDB-client MariaDB-shared MariaDB-backup MariaDB-common

启动、添加开机自启、检查状态
systemctl start mariadb.service
systemctl enable mariadb.service
systemctl status mariadb.service

mariadb的初始化
/usr/bin/mysql_secure_installation

一般建议按以下进行配置：
Enter current password for root (enter for none): Just press the Enter button
Set root password? [Y/n]: Y
New password: your-MariaDB-root-password
Re-enter new password: your-MariaDB-root-password
Remove anonymous users? [Y/n]: Y
Disallow root login remotely? [Y/n]: n
Remove test database and access to it? [Y/n]: Y
Reload privilege tables now? [Y/n]: Y

登录mysql
mysql -u root -p

至此，MariaDB10.3已安装完毕。

2）因为rabbitmq是用erlang语言开发的，所以先安装erlang

vim /etc/yum.repos.d/rabbitmq-erlang.repo
添加如下内容：
# In /etc/yum.repos.d/rabbitmq-erlang.repo
[rabbitmq-erlang]
name=rabbitmq-erlang
baseurl=https://dl.bintray.com/rabbitmq-erlang/rpm/erlang/22/el/7
gpgcheck=1
gpgkey=https://dl.bintray.com/rabbitmq/Keys/rabbitmq-release-signing-key.asc
repo_gpgcheck=0
enabled=1

安装erlang
yum install erlang

3）安装rabbitmq

vim /etc/yum.repos.d/rabbitmq-server.repo
添加如下内容：
[bintray-rabbitmq-server]
name=bintray-rabbitmq-rpm
baseurl=https://dl.bintray.com/rabbitmq/rpm/rabbitmq-server/v3.8.x/el/7/
gpgcheck=0
repo_gpgcheck=0
enabled=1

安装密钥
rpm --import https://github.com/rabbitmq/signing-keys/releases/download/2.0/rabbitmq-release-signing-key.asc

安装rabbitmq
yum install rabbitmq-server

启动rabbitmq
systemctl start rabbitmq-server.service
查看状态
systemctl status rabbitmq-server.service
设置开机自启
systemctl enable rabbitmq-server.service
设置关闭开机自启动
systemctl disable rabbitmq-server.service

配置rabbitmq（设置用户名root，密码rabbitmq，创建虚拟主机airflow-rabbitmq）
rabbitmqctl add_user root rabbitmq
rabbitmqctl add_vhost airflow-rabbitmq
rabbitmqctl set_user_tags root airflow-rabbitmq
rabbitmqctl set_permissions -p airflow-rabbitmq root ".*" ".*" ".*"
rabbitmq-plugins enable rabbitmq_management

4）安装celery、rabbitmq、mysql组件
pip3 install apache-airflow[celery]
pip3 install apache-airflow[rabbitmq]

pip3 install apache-airflow[mysql]
报错如下：
ERROR: Command errored out with exit status 1:
     command: /usr/local/bin/python3.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-j00jm48a/mysqlclient/setup.py'"'"'; __file__='"'"'/tmp/pip-install-j00jm48a/mysqlclient/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-j00jm48a/mysqlclient/pip-egg-info
         cwd: /tmp/pip-install-j00jm48a/mysqlclient/
    Complete output (10 lines):
    /bin/sh: mysql_config: command not found
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-j00jm48a/mysqlclient/setup.py", line 16, in <module>
        metadata, options = get_config()
      File "/tmp/pip-install-j00jm48a/mysqlclient/setup_posix.py", line 53, in get_config
        libs = mysql_config("libs_r")
      File "/tmp/pip-install-j00jm48a/mysqlclient/setup_posix.py", line 28, in mysql_config
        raise EnvironmentError("%s not found" % (mysql_config.path,))
    OSError: mysql_config not found
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
原因：
缺少以下包之一，
MariaDB-devel指的是包含开发首要的文件和一些静态库。
MariaDB-shared指的是包含一些动态客户端的库。指的是包含开发首要的文件和一些静态库。
MariaDB-shared指的是包含一些动态客户端的库。
解决：
yum install MariaDB-devel

5）CeleryExecutor配置