Redis Xcom部署到UAT环境-CSDN博客

本文链接：https://blog.csdn.net/terry21/article/details/133383514

目前我们UAT环境的Airflow使用的原生的XCOM机制，使用默认的BaseXcom类。由于之前MWAA的Airflow发生过因为xcom问题导致任务卡死的事故，值得我们花时间研究Redis xcom作为替换方案。

首先看看Airflow官网关于custom xcom的说法

https://airflow.apache.org/docs/apache-airflow/2.7.1/core-concepts/xcoms.html#custom-xcom-backends

我们根据下面github代码作为基础，修改少量代码来作为我们的Redis Xcom方案。

https://gist.github.com/msumit/1bc995c4a21acb257a39c36f0279e3d6

上面github代码主要思想是覆盖BaseXcom类的serialize_value和deserialize_value两个方法，在这两个方法中嵌入redis操作，把xcom的value写到redis上，和从redis取回value，从而达到目的。

我们开发的代码如下：

redis_xcom.py

# This is an example to show how easy to extend the custom XCom backends in Apache Airflow. 
# File: airflow/providers/redis/xcom/redis_xcom.py
# Author: Sumit Maheshwari

import json
import logging
from typing import Any
import uuid

from airflow.configuration import conf
from airflow.models.xcom import BaseXCom
from airflow.providers.redis.hooks.redis import RedisHook

log = logging.getLogger(__name__)
redis_conn_id = conf.get('xcom', 'redis_conn_id', fallback=RedisHook.default_conn_name)
redis_hook = RedisHook(redis_conn_id=redis_conn_id)


class RedisXCom(BaseXCom):

    @classmethod
    def delete(cls, xcom):
        """Delete XCom value from Redis"""
        result = redis_hook.get_conn().delete(xcom.value)
        log.debug("Result of deleting key to Redis %s", result)

    @staticmethod
    def serialize_value(value: Any):
        """Serialize the data as JSON, store into Redis & return the key"""
        val = json.dumps(value).encode('UTF-8')
        key = uuid.uuid4().hex

        log.debug('Setting XCom key %s to Redis', key)
        result = redis_hook.get_conn().set(f"xcom-{key}", val, 60 * 60 * 24 * 30)
        log.debug('Result of publishing to Redis %s', result)
        return BaseXCom.serialize_value(f"xcom-{key}")

    @staticmethod
    def deserialize_value(result: "XCom") -> Any:
        result = redis_hook.get_conn().get(result.value.decode().strip('"'))
        return json.loads(result.decode('UTF-8')) if result is not None \
            else '** XCom not found in Redis **'

上面部署到UAT环境后，发现redis可以正常存储xcom的value