python实现快速匹配_python最快捷的方法来合并基于键匹配的字典

I have 2 lists of dictionaries. List A is 34,000 long, list B is 650,000 long. I am essentially inserting all the List B dicts into the List A dicts based on a key match. Currently, I am doing the obvious, but its taking forever (seriously, like a day). There must be a quicker way!

for a in listA:

a['things'] = []

for b in listB:

if a['ID'] == b['ID']:

a['things'].append(b)

解决方案

Here's an approach that may help. I'll leave it to you to fill in the details.

Your code is slow because it is a O(n^2) algorithm, comparing every A against every B.

If you sort each of listA and listB by id first (these are O(nlogn)) operations, then you can iterate easily through the sorted versions of A and B (this will be in linear time).

This approach is common when you have to do external merges on very large data sets. Mihai's answer is better for internal merging, where you simply index everything by id (in memory). If you have the memory to hold these additional structures, and dictionary lookup is constant time, that approach will likely be faster, not to mention simpler. :)

By way of example let's say A had the following ids after sorting

acfgjp

and B had these ids, again after sorting

aaaabbbbcccddeeeefffggiikknnnnppppqqqrrr

The idea is, strangely enough, to keep indexes into A and B (I know that does not sound very Pythonic). At first you are looking at a in A and a in B. So you walk through B adding all the a's to your "things" array for a. Once you exhaust the a's in B, you move up one in A, to c. But the next item in B is b, which is less than c, so you have to skip the b's. Then you arrive at a c in B, so you can start adding into "things" for c. Continue in this fashion until both lists are exhausted. Just one pass. :)