使用TensorFlow将词语转化为数字索引

TensorFlow中内建的类tf.contrib.learn.preprocessing.VocabularyProcessor( max_document_length, min_frequency=0, vocabulary=None, tokenizer_fn=None)可以返回一个“能够将文档中的词汇转化为数字索引文档”的对象。其中，max_document_length表示转换完之后，文档中，每句话的长度，min_frequency=0表示文档中，每个词出现的频次最小数。

from tensorflow.contrib import learn

texts = ['go until jurong point crazy available only in bugis n great world la e buffet cine there got amore wat',
 'ok lar joking wif u oni',
 'free entry in a wkly comp to win fa cup final tkts st may text fa to to receive entry questionstd txt ratetcs apply overs',
 'u dun say so early hor u c already then say',
 'nah i dont think he goes to usf he lives around here though']

texts2 = texts[0:5]
vocab_processor = learn.preprocessing.VocabularyProcessor(20, min_frequency=1)
transformed_texts = np.array([x for x in vocab_processor.transform(texts)])
print(transformed_texts)

## 运行结果：
[[   1    2    3 ...   18   19   20]
 [  21   22   23 ...    0    0    0]
 [  27   28    8 ...   32   41   28]
 ...
 [7687  302    8 ...    0    0    0]
 [ 128 3066  205 ...  166   68   54]
 [3173   64 1156 ...    0    0    0]]