问题描述
有些情况下,用多进程处理数据分析的时候,程序要一直运行,但是配置文件的内容不定时的修改,所以为了让数据分析过程在不用重启的情况下能够感知到配置文件发生了变化,引入全局变量来定义配置文件中需要修改的内容,并且用定时调度任务定时的重新读取配置文件进而达到修改全局变量的目的。那么多进程数据处理过程也会在后续的处理中替换修改的内容。
其中实现调度任务采用python的三方包:APSCheduler。具体使用方式见https://cloud.tencent.com/developer/article/2343943
在多进程调用过程中,我并没有在函数内调用全局变量,而是将全局变量当作参数传入。是因为我在函数内调用全局变量时,没有感知到全局变量的变化,即全局变量还是最开始的值。所以采用参数传入的方式。如果您有其他方式,欢迎探讨!
代码实现:
schedule_demo.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-
# ----------------------------------
# @Time : 2023/11/3 14:29
# @File : schedule_demo.py
# Description : 定时调度修改全局变量,多进程处理过程使用全局变量。
# ----------------------------------
import json
import time
from multiprocessing import Pool
from apscheduler.schedulers.background import BackgroundScheduler
def my_function(file='test_num.json'):
with open(file, 'r') as fp:
content = json.load(fp)
return content['num']
global_bar = my_function()
def modify_global_var(file='test_num.json'):
with open(file, 'r') as fp:
content = json.load(fp)
global global_bar
global_bar = content['num']
def schedu_main():
scheduler = BackgroundScheduler()
scheduler.add_job(modify_global_var, 'interval', seconds=2)
scheduler.start()
def work_fun(x, global_bar1):
print(global_bar1)
if x == global_bar1:
return global_bar1, x, x+10
else:
return global_bar1, x, None
def newjobs():
executor = Pool(processes=2)
global global_bar
for msg in range(1000):
time.sleep(2)
print(global_bar, msg)
executor.apply_async(work_fun, (msg, global_bar, ), callback=result_write_file, error_callback=error_callback_func)
executor.close()
def error_callback_func(e):
print("error_callback_func:", e)
def result_write_file(result):
# 将结果写入file中, 只用于测试
with open('./result.txt', 'a+') as f:
f.write("{}--{}--\n".format(time.time(), result))
if __name__ == '__main__':
schedu_main()
newjobs()
配合代码运行的两个测试文件:
test_num.json 【不定时的修改该文件中的数字,模拟配置文件内容修改】
{"num": 23}
result.txt 【简单处理的结果保存到文件中,便于观察】
1699256912.282117--(5, 0, None)--
1699256914.2891016--(5, 1, None)--
1699256916.3010926--(5, 2, None)--
1699256918.3037179--(5, 3, None)--
1699256920.3068454--(5, 4, None)--
1699256922.3107085--(5, 5, 15)--
1699256924.3207312--(8, 6, None)--
1699256926.3276083--(8, 7, None)--
1699256928.3352--(8, 8, 18)--
1699256930.3450694--(8, 9, None)--
1699256932.357837--(10, 10, 20)--
1699256934.3688412--(10, 11, None)--
1699256936.371947--(10, 12, None)--
1699256938.3803031--(13, 13, 23)--
1699256940.39031--(13, 14, None)--
1699256942.3958411--(13, 15, None)--
1699256944.402155--(15, 16, None)--
1699256946.416262--(15, 17, None)--
1699256948.423182--(15, 18, None)--
1699256950.431013--(18, 19, None)--
1699256952.429999--(18, 20, None)--
1699256954.435797--(18, 21, None)--
1699256956.4496002--(23, 22, None)--
1699256958.45329--(23, 23, 33)--
1699256960.46067--(23, 24, None)--