python多进程日志相关资料

上个月，我们某个系统突然发现日志输出有毛病，后来一看，居然是因为python日志不是进程安全的，后来找到了一个并发包，from cloghandler import ConcurrentRotatingFileHandler就这个东西，但是多天以后，发现，日志还是有毛病，打印的日志编号混乱，而且内容还重复，还缺失。

作为调包侠，我还能做什么，但是是搜索啊？

然后就找到了几个大神对原来可以按时间分割日志的handle类进行修改，来解决问题的，总之咱也看不懂，只能盼望着他们分享的代码不要有什么bug。

ok，首先，我们来看，handle是如何使用的，因为之前我的日志采集是弄好配置文件，然后直接读配置文件的，但是现在明显我没办法这么做了，但是如何创建好一个log，我已经忘记了，

先上别人的代码

import logging
import logging.handlers
# logging初始化工作
logging.basicConfig()
# nor的初始化工作
nor = logging.getLogger("nor")
nor.setLevel(logging.INFO)
# 添加TimedRotatingFileHandler到nor
# 定义一个1分钟换一次log文件的handler
filehandler = logging.handlers.TimedRotatingFileHandler(
        "logging_test2", 'M', 1, 0)
# 设置后缀名称，跟strftime的格式一样
filehandler.suffix = "%Y%m%d-%H%M.log"
nor.addHandler(filehandler)

然后我们再看一段代码

def __init__(self, level, log_path, log_name, when, interval, backupCount):

        if not os.path.exists(log_path):
            try:
                os.makedirs(log_path)
            except Exception:
                exc_type, exc_value, exc_traceback = sys.exc_info()
                print(exc_type, exc_value, exc_traceback)

        format = '%(asctime)s - %(levelname)s - %(message)s'
        formatter = Formatter(format)

        logging.basicConfig(level=level, format=format)

        fileHandler = TimedRotatingFileHandler(filename=log_path + log_name, when=when, interval=interval, backupCount=backupCount)
        fileHandler.setFormatter(formatter)

        # logging.getLogger('') ????root????????????????
        self.log = logging.getLogger('')
        self.log.addHandler(fileHandler) 

用这两串代码，我合成了现在需要使用的代码

def get_logger(name='online'):
    
    # logging.config.dictConfig(log_config.LOGGING_DIC)
    log_path='/home/logs/detection_online/'
    format = '%(asctime)s - %(levelname)s - %(message)s'
    formatter = Formatter(format)

    logging.basicConfig(level='DEBUG', format=format)

    fileHandler = SafeLog(filename=log_path+'run.log', when='H', interval=1,
                                           backupCount=15)
    fileHandler.setFormatter(formatter)
    logger = logging.getLogger(name)
    
    logger.addHandler(fileHandler)
    
    return logger

和以前最大的不同在哪？

就在上面的那个SafeLog处，我们再看这段代码

import os
import time
import multiprocessing
from logging.handlers import TimedRotatingFileHandler
from logging import FileHandler

lock = multiprocessing.Lock()

class SafeLog(TimedRotatingFileHandler):
    def __init__(self, *args, **kwargs):
        super(SafeLog, self).__init__(*args, **kwargs)
        self.suffix_time = ""
        self.origin_basename = self.baseFilename


    def shouldRollover(self, record):
        timeTuple = time.localtime()
        if self.suffix_time != time.strftime(self.suffix, timeTuple) or not os.path.exists(self.origin_basename+'.'+self.suffix_time):
            return 1
        else:
            return 0

    def doRollover(self):
        if self.stream:
            self.stream.close()
            self.stream = None

        currentTimeTuple = time.localtime()
        self.suffix_time = time.strftime(self.suffix, currentTimeTuple)
        self.baseFilename = self.origin_basename + '.' + self.suffix_time

        self.mode = 'a'

        global lock
        with lock:
            if self.backupCount > 0:
                for s in self.getFilesToDelete():
                    os.remove(s)

        if not self.delay:
            self.stream = self._open()

    def getFilesToDelete(self):
        #将源代码的 self.baseFilename 改为 self.origin_basename
        dirName, baseName = os.path.split(self.origin_basename)
        fileNames = os.listdir(dirName)
        result = []
        prefix = baseName + "."
        plen = len(prefix)
        for fileName in fileNames:
            if fileName[:plen] == prefix:
                suffix = fileName[plen:]
                if self.extMatch.match(suffix):
                    result.append(os.path.join(dirName, fileName))
        if len(result) < self.backupCount:
            result = []
        else:
            result.sort()
            result = result[:len(result) - self.backupCount]
        return result

修改python logging源代码，我只能说，我真的看不大懂，代码那么多