keras计算机内存不足,在keras.preprocessing.text中使用Tokenizer时内存不足

我想用keras建立一个RNN模型来对句子进行分类。

我尝试了以下代码：

docs = []

with open('all_dga.txt', 'r') as f:

for line in f.readlines():

dga_domain, _ = line.split(' ')

docs.append(dga_domain)

t = Tokenizer()

t.fit_on_texts(docs)

encoded_docs = t.texts_to_matrix(docs, mode='count')

print(encoded_docs)

但得到了一个MemoryError。似乎我无法将所有数据加载到内存中。这是输出：

Traceback (most recent call last):

File "test.py", line 11, in

encoded_docs = t.texts_to_matrix(docs, mode='count')

File "/home/yurzho/anaconda3/envs/deepdga/lib/python3.6/site-packages/keras/preprocessing/text.py", line 273, in texts_to_matrix

return self.sequences_to_matrix(sequences, mode=mode)

File "/home/yurzho/anaconda3/envs/deepdga/lib/python3.6/site-packages/keras/preprocessing/text.py", line 303, in sequences_to_matrix

x = np.zeros((len(sequences), num_words))

MemoryError

如果有人熟悉keras，请告诉我如何预处理数据集。

提前致谢！