python存储大量数据的数据结构_python – 存储一组四个(或更多)值的最佳数据结构是什么？...

Python中没有一个内置的数据结构可以完成你想要的任何事情,但是它很容易结合使用它来实现你的目标并且相当有效地完成目标.

例如,假设您的输入是名为employees.csv的逗号分隔值文件中的以下数据,其字段名称定义如第一行所示：

name,age,weight,height

Bob Barker,25,175,6ft 2in

Ted Kingston,28,163,5ft 10in

Mary Manson,27,140,5ft 6in

Sue Sommers,27,132,5ft 8in

Alice Toklas,24,124,5ft 6in

以下是工作代码,说明如何将此数据读取并存储到记录列表中,并自动创建单独的查找表,以查找与每个记录的字段中包含的值相关联的记录.

记录是由namedtuple创建的类的实例,这是一个非常高效的内存,因为每个类都缺少类实例通常包含的__dict__属性.使用它们可以使用点语法(如record.fieldname)按名称访问每个字段.

查找表是defaultdict(list)实例,它们平均提供类似字典的O(1)查找时间,并且还允许多个值与每个值相关联.因此,查找键是要搜索的字段值的值,与之关联的数据将是存储在员工列表中的人员记录的整数索引列表中的值 – 因此它们都是相对的小.

请注意,该类的代码完全是数据驱动的,因为它不包含任何硬编码的字段名称,而是在读入时从csv数据输入文件的第一行中获取.当使用实例时,任何实际当然,retrieve()方法调用必须包含有效的字段名称关键字参数.

更新

修改为在首次读取数据文件时不为每个字段的每个唯一值创建查找表.现在,retrieve()方法仅在需要时创建它们(并保存/缓存结果以供将来使用).也修改为在Python 2.7中工作,包括3.x.

from collections import defaultdict, namedtuple

import csv

class DataBase(object):

def __init__(self, csv_filename, recordname):

# Read data from csv format file into a list of namedtuples.

with open(csv_filename, 'r') as inputfile:

csv_reader = csv.reader(inputfile, delimiter=',')

self.fields = next(csv_reader) # Read header row.

self.Record = namedtuple(recordname, self.fields)

self.records = [self.Record(*row) for row in csv_reader]

self.valid_fieldnames = set(self.fields)

# Create an empty table of lookup tables for each field name that maps

# each unique field value to a list of record-list indices of the ones

# that contain it.

self.lookup_tables = defaultdict(lambda: defaultdict(list))

def retrieve(self, **kwargs):

""" Fetch a list of records with a field name with the value supplied

as a keyword arg (or return None if there aren't any). """

if len(kwargs) != 1: raise ValueError(

'Exactly one fieldname/keyword argument required for function '

'(%s specified)' % ', '.join([repr(k) for k in kwargs.keys()]))

field, value = list(kwargs.items())[0] # Get only keyword arg and value.

if field not in self.valid_fieldnames:

raise ValueError('keyword arg "%s" isn\'t a valid field name' % field)

if field not in self.lookup_tables: # Must create field look up table.

for index, record in enumerate(self.records):

value = getattr(record, field)

self.lookup_tables[field][value].append(index)

matches = [self.records[index]

for index in self.lookup_tables[field].get(value, [])]

return matches if matches else None

if __name__ == '__main__':

empdb = DataBase('employees.csv', 'Person')

print("retrieve(name='Ted Kingston'): {}".format(empdb.retrieve(name='Ted Kingston')))

print("retrieve(age='27'): {}".format(empdb.retrieve(age='27')))

print("retrieve(weight='150'):".format(empdb.retrieve(weight='150')))

try:

print("retrieve(hight='5ft 6in'):".format(empdb.retrieve(hight='5ft 6in')))

except ValueError as e:

print("ValueError('{}') raised as expected".format(e))

else:

raise type('NoExceptionError', (Exception,), {})(

'No exception raised from "retrieve(hight=\'5ft\')" call.')

输出：

retrieve(name='Ted Kingston'): [Person(name='Ted Kingston', age='28', weight='163', height='5ft 10in')]

retrieve(age='27'): [Person(name='Mary Manson', age='27', weight='140', height='5ft 6in'),

Person(name='Sue Sommers', age='27', weight='132', height='5ft 8in')]

retrieve(weight='150'): None

retrieve(hight='5ft 6in'): ValueError('keyword arg "hight" is an invalid fieldname')

raised as expected