1、airflow安装
为了后期安装其它软件的需要,在此先安装必要的依赖包(我曾经跨过山和大海,也穿过人山人海...),重要的事情只说一遍哦
yum -y groupinstall "Development tools"
yum -y install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel xz-devel libffi-devel
# airflow needs a home, ~/airflow is the default,
# but you can lay foundation somewhere else if you prefer
# (optional)
export AIRFLOW_HOME=~/airflow
# install from pypi using pip
pip3 install apache-airflow
# initialize the database
airflow initdb
# start the web server, default port is 8080
airflow webserver -p 8080
# start the scheduler
airflow scheduler
# visit localhost:8080 in the browser and enable the example dag in the home page
2、设置防火墙(如果不是本地访问airflow)
直接拉取firewall-cmd:
yum install firewalld systemd -y
查看服务器上是否安装了firewall
systemctl status firewalld
开启防火墙
systemctl start firewalld.service
关闭防火墙
systemctl stop firewalld.service
设置开机自启
systemctl enable firewalld.service
设置关闭开机自启动
systemctl disable firewalld.service
在不改变状态的条件下重新加载防火墙
firewall-cmd --reload
查看已开放的端口
netstat -anp
查询指定8080端口是否开放
firewall-cmd --query-port=8080/tcp
yes 表示开放 no表示不开放
3、访问airflow
ip:8080
4、配置CeleryExecutor
1)安装MariaDB10.3
vim /etc/yum.repos.d/MariaDB.repo
添加如下内容:
[mariadb]
name = MariaDB
baseurl = http://yum.mariadb.org/10.3/centos7-amd64
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1
yum clean all
yum makecache
Importing the MariaDB GPG Public Key
rpm --import https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
安卓MariaDB10.3
yum install MariaDB-server galera MariaDB-client MariaDB-shared MariaDB-backup MariaDB-common
启动、添加开机自启、检查状态
systemctl start mariadb.service
systemctl enable mariadb.service
systemctl status mariadb.service
mariadb的初始化
/usr/bin/mysql_secure_installation
一般建议按以下进行配置:
Enter current password for root (enter for none): Just press the Enter button
Set root password? [Y/n]: Y
New password: your-MariaDB-root-password
Re-enter new password: your-MariaDB-root-password
Remove anonymous users? [Y/n]: Y
Disallow root login remotely? [Y/n]: n
Remove test database and access to it? [Y/n]: Y
Reload privilege tables now? [Y/n]: Y
登录mysql
mysql -u root -p
至此,MariaDB10.3已安装完毕。
2)因为rabbitmq是用erlang语言开发的,所以先安装erlang
vim /etc/yum.repos.d/rabbitmq-erlang.repo
添加如下内容:
# In /etc/yum.repos.d/rabbitmq-erlang.repo
[rabbitmq-erlang]
name=rabbitmq-erlang
baseurl=https://dl.bintray.com/rabbitmq-erlang/rpm/erlang/22/el/7
gpgcheck=1
gpgkey=https://dl.bintray.com/rabbitmq/Keys/rabbitmq-release-signing-key.asc
repo_gpgcheck=0
enabled=1
安装erlang
yum install erlang
3)安装rabbitmq
vim /etc/yum.repos.d/rabbitmq-server.repo
添加如下内容:
[bintray-rabbitmq-server]
name=bintray-rabbitmq-rpm
baseurl=https://dl.bintray.com/rabbitmq/rpm/rabbitmq-server/v3.8.x/el/7/
gpgcheck=0
repo_gpgcheck=0
enabled=1
安装密钥
rpm --import https://github.com/rabbitmq/signing-keys/releases/download/2.0/rabbitmq-release-signing-key.asc
安装rabbitmq
yum install rabbitmq-server
启动rabbitmq
systemctl start rabbitmq-server.service
查看状态
systemctl status rabbitmq-server.service
设置开机自启
systemctl enable rabbitmq-server.service
设置关闭开机自启动
systemctl disable rabbitmq-server.service
配置rabbitmq(设置用户名root,密码rabbitmq,创建虚拟主机airflow-rabbitmq)
rabbitmqctl add_user root rabbitmq
rabbitmqctl add_vhost airflow-rabbitmq
rabbitmqctl set_user_tags root airflow-rabbitmq
rabbitmqctl set_permissions -p airflow-rabbitmq root ".*" ".*" ".*"
rabbitmq-plugins enable rabbitmq_management
4)安装celery、rabbitmq、mysql组件
pip3 install apache-airflow[celery]
pip3 install apache-airflow[rabbitmq]
pip3 install apache-airflow[mysql]
报错如下:
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python3.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-j00jm48a/mysqlclient/setup.py'"'"'; __file__='"'"'/tmp/pip-install-j00jm48a/mysqlclient/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-j00jm48a/mysqlclient/pip-egg-info
cwd: /tmp/pip-install-j00jm48a/mysqlclient/
Complete output (10 lines):
/bin/sh: mysql_config: command not found
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-j00jm48a/mysqlclient/setup.py", line 16, in <module>
metadata, options = get_config()
File "/tmp/pip-install-j00jm48a/mysqlclient/setup_posix.py", line 53, in get_config
libs = mysql_config("libs_r")
File "/tmp/pip-install-j00jm48a/mysqlclient/setup_posix.py", line 28, in mysql_config
raise EnvironmentError("%s not found" % (mysql_config.path,))
OSError: mysql_config not