前段时间项目中出了一个问题:多线程定时任务服务中,读入内存的全局constant变量被同进程的其他线程任务篡改(前人挖坑后人跳啊……),导致其他任务获取到错误的constant信息。
解决方案提了两个:
1.进程内全局共用的变量,任务调用前加一层深拷贝,任务调用深拷贝的返回值;
2.自定义不可变类型的dict和list, 从根本上杜绝代码疏忽。
最终采用了方案1,不过个人感觉方案2很有意思,就私下里尝试写了一下,亲测稳定。
该方案实现了:
1.自定义了不可变类型的dict和list,元素的更新/删除/添加 等方法全禁用,只能读。
2.提供了自定义的深拷贝方法:读入内存的全局共享变量本身不允许改变,但有些场景下确实需要利用这些元素做一些处理。方法是:利用递归做深拷贝,从内存中还原出一个全新的dict/list,该list/dict是python自带的普通数据类型,允许所有操作。
代码如下:
ps: 环境为python2.7,python3 的UserDict和UserList的导包路径不同
# coding=utf-8
'''
该模块提供自定义的数据类型,用于一些特殊场景。如:
自定义不可变类型的dict、list,用于声明重要的全局变量、常量,避免写代码疏忽导致的一些问题;
plus:UserDict和UserList模块是python2的官方模块,但python3.7已经迁移到collections中,如果升级python版本,务必做好调整和测试;
'''
from UserDict import UserDict
from UserList import UserList
import copy
class CommonMethods:
@staticmethod
def _copy_loop(items, type):
'''
:param items: 需要copy的对象的enumerate结果;
:param type:目前支持list和dict;
:return:完全可读写的深拷贝结果;
'''
ret = {} if type == 'dict' else [] if type == 'list' else None
if ret is None:
raise Exception('Unexpected item type: %s' % type)
for tag, value in items:
if isinstance(value, UnsetableDict):
cp_value = CommonMethods._copy_loop(value._copy.items(), 'dict')
elif isinstance(value, UnsetableList):
cp_value = CommonMethods._copy_loop(enumerate(value._copy), 'list')
elif isinstance(value, dict):
cp_value = CommonMethods._copy_loop(copy.copy(value).items(), 'dict')
elif isinstance(value, list):
cp_value = CommonMethods._copy_loop(enumerate(copy.copy(value)), 'list')
else:
cp_value = value
if type == 'list':
ret.append(cp_value)
else:
ret[tag] = cp_value
return ret
class UnsetableList(CommonMethods, UserList):
'''
说明:为防止全局变量遭到非预期的改动,这里自定义一个list类,
禁止一切list的修改操作,同时支持list其他所有操作(index、iter、比较运算符等)
'''
def __setitem__(self, i, item):
raise NotAllowedOperation('not allowed to change this list.')
def __delitem__(self, i):
raise NotAllowedOperation('not allowed to change this list.')
@property
def _copy(self): # 浅拷贝,只拷贝一层;
return copy.copy(self.data)
@property
def deep_copy(self):
'''
自定义的深拷贝,递归遍历所有节点,把不可变dict、list转换成正常可操作的普通dict、list,并返回其深拷贝结果;
'''
return self._copy_loop(enumerate(copy.copy(self.data)), 'list')
def append(self, item):
raise NotAllowedOperation('not allowed to change this list.')
def insert(self, i, item):
raise NotAllowedOperation('not allowed to change this list.')
def pop(self, i=-1):
raise NotAllowedOperation('not allowed to change this list.')
def remove(self, item):
raise NotAllowedOperation('not allowed to change this list.')
def reverse(self):
raise NotAllowedOperation('not allowed to change this list.')
def sort(self, *args, **kwds):
raise NotAllowedOperation('not allowed to change this list.')
def extend(self, other):
raise NotAllowedOperation('not allowed to change this list.')
class UnsetableDict(CommonMethods, UserDict):
'''
说明:为防止全局变量遭到非预期的改动,这里自定义一个dict类,禁止一切dict的修改操作,
同时支持dict其他所有操作如items()、keys()、get_key()等;
'''
def __setitem__(self, key, item):
raise NotAllowedOperation('not allowed to change this dict.')
def __delitem__(self, key):
raise NotAllowedOperation('not allowed to change this dict.')
@property
def _copy(self): # 浅拷贝,只拷贝一层;
return copy.copy(self.data)
@property
def deep_copy(self):
'''
自定义的深拷贝,递归遍历所有节点,把不可变dict、list转换成正常可操作的普通dict、list,并返回其深拷贝结果;
'''
return self._copy_loop(copy.copy(self.data).items(), 'dict')
def update(*args, **kwargs):
self = args[0]
if not len(self.data.items()): # 初始实例化dict,允许;
UserDict.update(*args, **kwargs)
else: # 意图update已定义的dict, 拒绝;
raise NotAllowedOperation('not allowed to change this dict.')
def clear(self):
raise NotAllowedOperation('not allowed to change this dict.')
def pop(self, key, *args):
raise NotAllowedOperation('not allowed to change this dict.')
def popitem(self):
raise NotAllowedOperation('not allowed to change this dict.')
@classmethod
def fromkeys(cls, iterable, value=None):
raise NotAllowedOperation('not allowed to change this dict.')
def setdefault(self, key, failobj=None):
raise NotAllowedOperation('not allowed to change this dict.')
UnsetableDict()
*****
class NotAllowedOperation(Exception):
pass
if __name__ == '__main__':
l = UnsetableDict({'a': UnsetableList([UnsetableDict({'c': 1}), 2, 'a', {'ts': [1,UnsetableList([1,2,3])]}])})
a = l.deep_copy
print(a)
a['a'].append({'s': 5})
c = a['a'][0].update({'s': 6})
print(l)
print(a)