python使用logging库函数记录日志时中文字符被转换为\uxxxx格式

问题描述

在python项目中使用logging库中的函数记录日志，输出到控制台的中文显示为\uxxxx格式

{"process": 17944, "asctime": "2024-03-22 17:08:49,376", "levelname": "INFO", "filename": "test_logging.py", "lineno": 36, "message": "\u8f93\u51fa\u4e2d\u6587\u65e5\u5fd7"}

问题分析

使用json格式进行日志配置，输出日志内容格式化为json字符串。logging配置如下：

import logging
import logging.config

# 定义JSON格式的日志配置
logconfig = {
    'version':1,
    'disable_existing_loggers': False,
    'formatters': {
        "generic": {
            "format": "[%(process)d] [%(asctime)s] %(levelname)s [%(filename)s:%(lineno)s] %(message)s", # 打日志的格式
            "class": "pythonjsonlogger.jsonlogger.JsonFormatter"
        }
    },
    'handlers': {
        "console": {
            "class": "logging.StreamHandler",
            "level": "DEBUG",
            "formatter": "generic",
            "stream": "ext://sys.stdout"
        }
    },
    "root": {
        "level": "DEBUG",
        "handlers": ["console"]
    }
}

# 解析JSON配置并应用配置
logging.config.dictConfig(logconfig)

# 创建一个日志记录器
logger = logging.getLogger(__name__)

JsonFormatter初始化参数如下：

    def __init__(self, *args, **kwargs):
        """
        :param json_default: a function for encoding non-standard objects
            as outlined in https://docs.python.org/3/library/json.html
        :param json_encoder: optional custom encoder
        :param json_serializer: a :meth:`json.dumps`-compatible callable
            that will be used to serialize the log record.
        :param json_indent: an optional :meth:`json.dumps`-compatible numeric value
            that will be used to customize the indent of the output json.
        :param prefix: an optional string prefix added at the beginning of
            the formatted string
        :param rename_fields: an optional dict, used to rename field names in the output.
            Rename message to @message: {'message': '@message'}
        :param static_fields: an optional dict, used to add fields with static values to all logs
        :param json_indent: indent parameter for json.dumps
        :param json_ensure_ascii: ensure_ascii parameter for json.dumps
        :param reserved_attrs: an optional list of fields that will be skipped when
            outputting json log record. Defaults to all log record attributes:
            http://docs.python.org/library/logging.html#logrecord-attributes
        :param timestamp: an optional string/boolean field to add a timestamp when
            outputting the json log record. If string is passed, timestamp will be added
            to log record using string as key. If True boolean is passed, timestamp key
            will be "timestamp". Defaults to False/off.
        """
        self.json_default = self._str_to_fn(kwargs.pop("json_default", None))
        self.json_encoder = self._str_to_fn(kwargs.pop("json_encoder", None))
        self.json_serializer = self._str_to_fn(kwargs.pop("json_serializer", json.dumps))
        self.json_indent = kwargs.pop("json_indent", None)
        self.json_ensure_ascii = kwargs.pop("json_ensure_ascii", True)
        self.prefix = kwargs.pop("prefix", "")
        self.rename_fields = kwargs.pop("rename_fields", {})
        self.static_fields = kwargs.pop("static_fields", {})
        reserved_attrs = kwargs.pop("reserved_attrs", RESERVED_ATTRS)
        self.reserved_attrs = dict(zip(reserved_attrs, reserved_attrs))
        self.timestamp = kwargs.pop("timestamp", False)

        # super(JsonFormatter, self).__init__(*args, **kwargs)
        logging.Formatter.__init__(self, *args, **kwargs)
        if not self.json_encoder and not self.json_default:
            self.json_encoder = JsonEncoder

        self._required_fields = self.parse()
        self._skip_fields = dict(zip(self._required_fields,
                                     self._required_fields))
        self._skip_fields.update(self.reserved_attrs)

在初始化参数中，有一个json_ensure_ascii（ensure_ascii parameter for json.dumps） 参数，该参数的默认值是True，在将json格式的日志转为json字符串时使用。json.dump()将包含中文的json数据转换为json字符串时，如果设置ensure_ascii参数为True，最后得到的json字符串中文会被转换为\uxxxx格。如果在配置formatter时将“json_ensure_ascii”设置为False，则可以避免该问题。

接下来看一下设置formatter的源码：

    def configure_formatter(self, config):
        """Configure a formatter from a dictionary."""
        if '()' in config:
            factory = config['()'] # for use in exception handler
            try:
                result = self.configure_custom(config)
            except TypeError as te:
                if "'format'" not in str(te):
                    raise
                #Name of parameter changed from fmt to format.
                #Retry with old name.
                #This is so that code can be used with older Python versions
                #(e.g. by Django)
                config['fmt'] = config.pop('format')
                config['()'] = factory
                result = self.configure_custom(config)
        else:
            fmt = config.get('format', None)
            dfmt = config.get('datefmt', None)
            style = config.get('style', '%')
            cname = config.get('class', None)

            if not cname:
                c = logging.Formatter
            else:
                c = _resolve(cname)

            # A TypeError would be raised if "validate" key is passed in with a formatter callable
            # that does not accept "validate" as a parameter
            if 'validate' in config:  # if user hasn't mentioned it, the default will be fine
                result = c(fmt, dfmt, style, config['validate'])
            else:
                result = c(fmt, dfmt, style)

        return result

从源码可以看出，系统支持的formatter配置是format、datefmt、style、class以及validate，不支持json_ensure_ascii配置项。想要配置的json_ensure_ascii生效，需要使用自定义配置。自定义配置的formatter解析源码如下：

    def configure_custom(self, config):
        """Configure an object with a user-supplied factory."""
        c = config.pop('()')
        if not callable(c):
            c = self.resolve(c)
        props = config.pop('.', None)
        # Check for valid identifiers
        kwargs = {k: config[k] for k in config if valid_ident(k)}
        result = c(**kwargs)
        if props:
            for name, value in props.items():
                setattr(result, name, value)
        return result

解决方案

修改logging的formatter配置：

'formatters': {
        "generic": {
            "format": "[%(process)d] [%(asctime)s] %(levelname)s [%(filename)s:%(lineno)s] %(message)s", # 打日志的格式
            "()": "pythonjsonlogger.jsonlogger.JsonFormatter",
            "json_ensure_ascii": False
        }
    }

修改后中文日志输出正常：

{"process": 15648, "asctime": "2024-03-22 18:46:17,068", "levelname": "INFO", "filename": "test_logging.py", "lineno": 37, "message": "输出中文日志"}