python使用numpy库将一个list中多段文本生成字典并统计词频

目的

将形如 L = [‘I have an apple …’,‘I have an egg…’,‘I don’t like pen…’]的列表进行预处理，并使用numpy.unique函数进行统计，生成提供给tf-idf使用的字典。
ps:也就是除了基础的科学计算库外不用其他库，造轮子行为。

实现

先来稍微复杂一点的实现：

import numpy as np
LA = np.array(['addc cdba ddaa dcaa dcba dcaa dcba dcaa dbaa dbbb cdba dbab dbcb cbba ccab bbbc cacd bbcc abdd accd abcd aadd abdd acdd abdd acdc acbc bbbd bacd aadd abcd abdd accd abdd acdd acdc ccca cdba ddba dcaa cbad bacd aacd abcd abdd abcd abdd abcd abdd acdd accd acdd abdd acdd abdd abcd accd abdd addb bdba dcaa dbbb dbba ddaa cdba dcba dcaa dcba dcaa dcba dcaa dbaa dbab dabc bacd abdd acdd accd abcd abdd acdd addc bdbb ddaa dcba dbaa dcbb dbca dcba ddaa dcaa dbaa dbab dbbb cadb ccca bcac bbbd',
       'cabd bacd abdd acdd accd abdd abcd abdd abcd abdd acdd adcc adcb dcba dcaa dbaa cbac bacd aacd abcd abdd acdd accd abcd abdd bcdb bcac bbbd aacd abcd abdd acdd accd abdd acdd accd abdd abcd acdd acdc bdbc bcbc cbcb ccba cdca ccba dcba dcaa dcba dcaa dbaa dbab dabc bacd abdd acdd acdc accc bcbd bacd badd abdd acdc acbd bbcd badd abdd accd abcd aadd abcd abdd abcd',
       'abcd aacd abdd acdd adcc dcba ddaa dcaa dbab dabc bbdc addb bdba ccab cabd bacd abdd acdd acdc adcb cdaa dcaa dbab cabd badd abdc adcc acbd bbbd bbcd aadd abdd acdd accc accd abcd aadd abdd abcd abdd acdd accc adcc cdba dcaa dbab dbba dcca cdaa ddaa dcba dcaa dcba dcaa dcba dcaa dbaa dbba dcba dcaa dbab dabc bacd aadd abdd acdd addb cdba ddba dcaa dbaa dbab dabb bacd aadd abdd acdd abdd abcd abdd acdd addc bdba',
       'acdd accc adbb ccba dbaa dbab dabc ccdb bcba bcad bacd abdc acbc acbd badc cbbc acbd abcd aadd abdd abcd bbbd bbcd abcd abdd acdd acdc adcb cdaa dcab dabd bacd aacd abcd abdd abcd abdd acdd abdd acdd accc bdcb ccba dcaa dbab dbbb ccda ccca ddaa dcaa dbba dcba dcaa dbaa dcaa dbaa dcaa dbaa dbbb dbba cdba ccba dcba dcaa dbaa dbbb dabc bacd abcd abdd acdd acdc accc cdca ddaa dcaa dbab dabb cabd abdd',
       'bacd aadd abdd acdd abdd abcd abdd acdd acdc adcc accc babd bbdd abcc adbd abcd cacc abbd abcd aadd acdc adbc bccc dabc bbbd abdd acdd adcc accc dcca bbac bbbd bacd abcd abdd abcd abdd acdd accc accb bdab ccbb dbba dbab bccc dbcb ccba cdba dcba dcaa dcab dbca ccca ccba ddba dcba dcaa dcba dcaa dbaa dbab dabb cacd abdd acdd abdd abcd abdd acdc adcb cdba dcba dcaa dbaa dbab dabb cbdb bcbb cccb ccca ddaa dcaa dcba dbaa dcba dcaa dbaa daac cacd aacd abcd',
       'dcaa dcba dcaa dcba dcaa dbab dbbb cacc abcd abdd aacd abdd acdc accc adcc cbcb dcba cdaa dcaa daab dbcb ccca cdab ccbb dacb bbbd bbcd abcd abdd acdd accc abcd abdd acdd accd abcd abdc bcca bdba ddba dcba dcaa dbaa dbab dabb cacd abcd abdd acdd accc accd bbcd bacd abdd abcd abdd abdc adcb adab dcba dbba dcaa ccba dcaa dbab dbbb dcca ddba ddaa dbab dabb dacb adcb adbc cccb dcca ddba'],dtype='<U554')

先初始化为array，为使结果可复现，特地从工程里抽了其中一段数据出来演示。

np.unique(LA[0].split(),return_counts=True)

返回其中一行内的字典和词频如下：

np.unique返回不重复的元素，return_counts返回元素的计数，由于一段文本（LA[0]）中包含有空格，所以我们使用.split()方法分割成一个个字符串，同时LA[0].split()返回了一个列表，存的就是分割完的字符串。

当然这不是我们要的结果，继续：

from itertools import chain
LB = [x.split() for x in LA] #生成嵌套list
LC = list(chain(*LB)) #把各个小list连起来，即chain

对LA进行遍历，大list里面包含了六个小list，再把小list使用迭代工具连起来，最后得到我们的LC：

总共502个词，再像前面一样使用np.unique()：

完成，总共六行代码，然而还可以更简洁。

更短的实现：

np.unique(' '.join(LA).split(),return_counts=True)

加上初始化一共两行代码，堪称完美。