python数据优化_性能提升48倍! python redis批量写入大量数据优化过程

1.最原始的版本,直接使用hset,效率很低

写30w条完耗时365秒,这样有两个问题:

相同的key,写入多条应该用hmset代替hset

另外可以用pipeline,避免频繁跟redis服务端交互,大量减少网络io

image.png

image.png

image.png

def get_conn():

r = redis.Redis(host='localhost', port=6379, decode_responses=True)

return r

def test_set_redis():

conn = get_conn()

machineId = 43696000000000

device_no = 88800000

work_in = time.time()

source = "1"

factory_no = "factory"

today = datetime.date.today()

oneday = datetime.timedelta(days=1)

tomorrow = str(today + oneday).replace("-", "")

afterTomorrow = str(today + oneday + oneda