python查看list有无重复_检查python list/numpy ndarray中是否存在重复项的最快方法

以下是我想到的四种方法。在

TL；DR：如果您期望很少(少于1/1000)个重复：def contains_duplicates(X):

return len(np.unique(X)) != len(X)

如果您希望频繁(超过1/1000)重复：

^{pr2}$

第一个方法是从this answer提前退出，它希望返回唯一值，第二个方法与应用于this answer的方法相同。在>>> import numpy as np

>>> X = np.random.normal(0,1,[10000])

>>> def terhorst_early_exit(X):

...: elems = set()

...: for i in X:

...: if i in elems:

...: return True

...: elems.add(i)

...: return False

>>> %timeit terhorst_early_exit(X)

100 loops, best of 3: 10.6 ms per loop

>>> def peterbe_early_exit(X):

...: seen = set()

...: seen_add = seen.add

...: for x in X:

...: if (x in seen or seen_add(x)):

...: return True

...: return False

>>> %timeit peterbe_early_exit(X)

100 loops, best of 3: 9.35 ms per loop

>>> %timeit len(set(X)) != len(X)

100 loops, best of 3: 4.54 ms per loop

>>> %timeit len(np.unique(X)) != len(X)

1000 loops, best of 3: 967 µs per loop

如果从一个普通的Python列表开始，而不是numpy.ndarray，情况会发生变化吗？在>>> X = X.tolist()

>>> %timeit terhorst_early_exit(X)

100 loops, best of 3: 9.34 ms per loop

>>> %timeit peterbe_early_exit(X)

100 loops, best of 3: 8.07 ms per loop

>>> %timeit len(set(X)) != len(X)

100 loops, best of 3: 3.09 ms per loop

>>> %timeit len(np.unique(X)) != len(X)

1000 loops, best of 3: 1.83 ms per loop

编辑：如果我们事先对重复的数量有预期呢？

上面的比较是在假设a)可能没有重复的情况下起作用的，或者b)我们更担心最坏的情况，而不是一般情况。在>>> X = np.random.normal(0, 1, [10000])

>>> for n_duplicates in [1, 10, 100]:

>>> print("{} duplicates".format(n_duplicates))

>>> duplicate_idx = np.random.choice(len(X), n_duplicates, replace=False)

>>> X[duplicate_idx] = 0

>>> print("terhost_early_exit")

>>> %timeit terhorst_early_exit(X)

>>> print("peterbe_early_exit")

>>> %timeit peterbe_early_exit(X)

>>> print("set length")

>>> %timeit len(set(X)) != len(X)

>>> print("numpy unique length")

>>> %timeit len(np.unique(X)) != len(X)

1 duplicates

terhost_early_exit

100 loops, best of 3: 12.3 ms per loop

peterbe_early_exit

100 loops, best of 3: 9.55 ms per loop

set length

100 loops, best of 3: 4.71 ms per loop

numpy unique length

1000 loops, best of 3: 1.31 ms per loop

10 duplicates

terhost_early_exit

1000 loops, best of 3: 1.81 ms per loop

peterbe_early_exit

1000 loops, best of 3: 1.47 ms per loop

set length

100 loops, best of 3: 5.44 ms per loop

numpy unique length

1000 loops, best of 3: 1.37 ms per loop

100 duplicates

terhost_early_exit

10000 loops, best of 3: 111 µs per loop

peterbe_early_exit

10000 loops, best of 3: 99 µs per loop

set length

100 loops, best of 3: 5.16 ms per loop

numpy unique length

1000 loops, best of 3: 1.19 ms per loop

因此，如果您期望很少的重复，那么numpy.unique函数就是最好的选择。随着预期重复数的增加，早期退出方法占主导地位。在