python中list 和 dict 效率的一次实践

业务场景：

    有上亿条的数据入库解析并且入库到sqlserver中去，所以每次优化一秒钟，可能对入库的性能就能提升一天。

    python语句的优化，里边有对list数据去重的代码如下：

   

    object_id_set = []
    remove_objects = []
    for object in objects:
        try:
            if object['object_id'] in object_id_set:
                remove_objects.append(object)
            else:
                object_id_set.append(object['object_id'])
        except Exception as e:
            rh_utils.rh_logger.exception('filter object error: {0}, object: {1}'.format(e, object))

    for remove_object in remove_objects:
        objects.remove(remove_object)

修改点就是 把 object_id_set 变成 dict： {}

   

object_id_set = {}
    remove_objects = []
    for object in objects:
        try:
            if object['object_id'] in object_id_set:
                remove_objects.append(object)
            else:
                object_id_set[object['object_id']] = 1
        except Exception as e:
            remove_objects.append(object)
            rh_utils.rh_logger.exception('filter object error: {0}, object: {1}'.format(e, object))

    for remove_object in remove_objects:
        objects.remove(remove_object)

当 过滤的objects的数目达到1万的时候，时间上面方式的时间居然达到了2s，不可接受。改成下面方法之后变成0.1s的样子。

看来对list进行 in 操作应该是遍历了整个 list的复杂度为 O(n), 而dict大概有类似hash的东西保证查询的复杂度为 O(1).