WORD2VEC-使用GLOVE训练中文词向量

WORD2VEC-使用GLOVE训练中文词向量

准备语料

语料预处理

1. 去停用词
2. 去除所有标点符号 ?!@#$%^&*()~`,./|
3. 重音字
4. 过滤掉频率低的单词
5. 处理数字 每个数字都替换为 #number
6. 单词转小写

如下图：

训练语料

从GitHub下载代码
将语料xxx.txt放入到Glove的主文件夹下。

git clone https://github.com/stanfordnlp/GloVe

修改代码

修改demo.sh脚本

1. 注释下载test.zip的语料脚本
2. 修改训练参数
   2.1 CORPUS=xx.txt 语料文件
   2.2 VOCAB_FILE=vocab.txt 得到的词和词频
   2.3 MEMORY=4.0 内存
   2.4 VOCAB_MIN_COUNT=5 最小词频数
   2.5 VECTOR_SIZE=50 词向量维度
   2.6 MAX_ITER=15 训练迭代次数
   2.7 WINDOW_SIZE=15 上下文窗口数
   2.8 BINARY=2 保存文件类型（2进制）
   2.9 NUM_THREADS=8 线程数

#!/bin/bash
set -e

# Makes programs, downloads sample data, trains a GloVe model, and then evaluates it.
# One optional argument can specify the language used for eval script: matlab, octave or [default] python

make
#if [ ! -e text8 ]; then
#  if hash wget 2>/dev/null; then
#    wget http://mattmahoney.net/dc/text8.zip
#  else
#    curl -O http://mattmahoney.net/dc/text8.zip
#  fi
#  unzip text8.zip
#  rm text8.zip
#fi

CORPUS=target.txt
VOCAB_FILE=vocab.txt
COOCCURRENCE_FILE=cooccurrence.bin
COOCCURRENCE_SHUF_FILE=cooccurrence.shuf.bin
BUILDDIR=build
SAVE_FILE=vectors
VERBOSE=2
MEMORY=4.0
VOCAB_MIN_COUNT=5
VECTOR_SIZE=50
MAX_ITER=15
WINDOW_SIZE=15
BINARY=2
NUM_THREADS=8
X_MAX=10
if hash python 2>/dev/null; then
    PYTHON=python
else
    PYTHON=python3
fi

echo
echo "$ $BUILDDIR/vocab_count -min-count $VOCAB_MIN_COUNT -verbose $VERBOSE < $CORPUS > $VOCAB_FILE"
$BUILDDIR/vocab_count -min-count $VOCAB_MIN_COUNT -verbose $VERBOSE < $CORPUS > $VOCAB_FILE
echo "$ $BUILDDIR/cooccur -memory $MEMORY -vocab-file $VOCAB_FILE -verbose $VERBOSE -window-size $WINDOW_SIZE < $CORPUS > $COOCCURRENCE_FILE"
$BUILDDIR/cooccur -memory $MEMORY -vocab-file $VOCAB_FILE -verbose $VERBOSE -window-size $WINDOW_SIZE < $CORPUS > $COOCCURRENCE_FILE
echo "$ $BUILDDIR/shuffle -memory $MEMORY -verbose $VERBOSE < $COOCCURRENCE_FILE > $COOCCURRENCE_SHUF_FILE"
$BUILDDIR/shuffle -memory $MEMORY -verbose $VERBOSE < $COOCCURRENCE_FILE > $COOCCURRENCE_SHUF_FILE
echo "$ $BUILDDIR/glove -save-file $SAVE_FILE -threads $NUM_THREADS -input-file $COOCCURRENCE_SHUF_FILE -x-max $X_MAX -iter $MAX_ITER -vector-size $VECTOR_SIZE -binary $BINARY -vocab-file $VOCAB_FILE -verbose $VERBOSE"
$BUILDDIR/glove -save-file $SAVE_FILE -threads $NUM_THREADS -input-file $COOCCURRENCE_SHUF_FILE -x-max $X_MAX -iter $MAX_ITER -vector-size $VECTOR_SIZE -binary $BINARY -vocab-file $VOCAB_FILE -verbose $VERBOSE
if [ "$CORPUS" = 'text8' ]; then
   if [ "$1" = 'matlab' ]; then
       matlab -nodisplay -nodesktop -nojvm -nosplash < ./eval/matlab/read_and_evaluate.m 1>&2 
   elif [ "$1" = 'octave' ]; then
       octave < ./eval/octave/read_and_evaluate_octave.m 1>&2
   else
       echo "$ $PYTHON eval/python/evaluate.py"
       $PYTHON eval/python/evaluate.py
   fi
fi

执行 sh

make && sh demo.sh
# 或者后台运行
nohup bash demo.sh >output.txt 2>&1 &

生成的文件
训练后会得到vetors.txt，打开后在第一行加上vacob_size vector_size，这样才能用word2vec的load函数加载成功
vacob_size vector_size可在训练时看到：
如下图：

Glove训练
WordVec训练

欢迎关注公众号：