docker下载、启动、配置apache airflow

        mac下安装apache airflow,会遇到各种麻烦,如python版本不支持、网络下载包慢、各种配置等繁琐问题,幸好docker上有airflow资源。废话不多说,接下来记录我在本地搭建airflow项目的过程。

 

一、启动本地docker

二、访问docker hub上的airflow官网项目地址:https://hub.docker.com/r/apache/airflow

$docker pull apache/airflow

三、在本地命令行工具执行docker pull apache/airflow,执行完之后,等待资源下载完成,之后重启docker容器,即可看到docker已经有airflow镜像。

四、运行airflow,在命令行工具执行以下命令(也可以在docker上对应的镜像镜像图像操作)       

$docker run -d -p 8080:8080 puckel/docker-airflow

五、如第四步没什么问题,即可访问到airflow控制台。

       访问地址:http://localhost:8080/admin/。(端口号在启动命令行里设置)

        注:还没初始化数据库之前,dag下的任务列表是为空的。

六、初始化数据库        

$airflow initdb

这时候应该会报以下错误:

$ airflow initdb
DB: sqlite:usr/local/airflow/airflow.db
[2020-05-21 12:09:20,775] {{db.py:368}} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/airflow/models/crypto.py", line 91, in get_fernet
    for fernet_part in fernet_key.split(',')
  File "/usr/local/lib/python3.7/site-packages/airflow/models/crypto.py", line 91, in <listcomp>
    for fernet_part in fernet_key.split(',')
  File "/usr/local/lib/python3.7/site-packages/cryptography/fernet.py", line 35, in __init__
    key = base64.urlsafe_b64decode(key)
  File "/usr/local/lib/python3.7/base64.py", line 133, in urlsafe_b64decode
    return b64decode(s)
  File "/usr/local/lib/python3.7/base64.py", line 87, in b64decode
    return binascii.a2b_base64(s)
binascii.Error: Incorrect padding

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 37, in <module>
    args.func(args)
  File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 1140, in initdb
    db.initdb(settings.RBAC)
  File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 117, in initdb
    schema='default'))
  File "<string>", line 4, in __init__
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 441, in _initialize_instance
    manager.dispatch.init_failure(self, args, kwargs)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 438, in _initialize_instance
    return manager.original_init(*mixed[1:], **kwargs)
  File "/usr/local/lib/python3.7/site-packages/airflow/models/connection.py", line 127, in __init__
    self.extra = extra
  File "<string>", line 1, in __set__
  File "/usr/local/lib/python3.7/site-packages/airflow/models/connection.py", line 218, in set_extra
    fernet = get_fernet()
  File "/usr/local/lib/python3.7/site-packages/airflow/models/crypto.py", line 95, in get_fernet
    raise AirflowException("Could not create Fernet object: {}".format(ve))
airflow.exceptions.AirflowException: Could not create Fernet object: Incorrect padding

airflow.exceptions.AirflowException: Could not create Fernet object: Incorrect padding

这问题是因为fernet key不正确,解决方法就是在docker命令行下生成fernet key,并配置到airflow.cfg的fernet_key配置项:

$python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"(回车)

>>>print(Fernet.generate_key().decode())"(回车)
>>>这里会得到一个字符串,并拷贝这个字符串

---------------------------------------------------

以上配置完成之后,重新执行:

$airflow initdb

如出现以下信息,则表明数据库初始化成功:

参考:https://www.jianshu.com/p/5349168dd346

 

七、重新刷新airflow控制台之后:

完!

 

  • 0
    点赞
  • 3
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值