MITIE训练预料

一、编译

git clone https://github.com/mit-nlp/MITIE.git
cd MITIE

#编译python接口
python setup.py install

#编译原生c++程序
cd MITIE/tools/wordrep
mkdir build
cd build
cmake ..
make

二、使用

1.python使用

from mitie import *

2.训练工具wordrep使用

#结构目录
-MITIE/tools/wordrep/build
 - CMakeCache.txt  
 - CMakeFiles  
 - cmake_install.cmake  
 - dlib_build  
 - Makefile  
 - mitie_build  
 - wordrep


#训练预料
cd ~
mkdir temp
cd temp
mv MITIE/tools/wordrep/build/wordrep .
mkdir zh
cp ~/text.txt ./zh
wordrep -e ./zh

#预料格式
你 可 不要 小看 我 
你 的 身高 是 多少
我 只有 二十厘米 高 ， 现在 还 在 买 半价票 呢 
你 有 多 高
我 个子 不高 呢 ， 只有 二十厘米 ， 出门 要 把 我 抱 在 怀里

#训练完毕之后的目录
/temp
 - substring_set.dat
 - top_word_counts.dat
 - total_word_feature_extractor.dat
 - wordrep
 - zh/
 - substrings.txt
 - top_words.txt
 - word_morph_feature_extractor.dat
 - word_vects.dat

#测试代码
from mitie import *

print ("loading Total Word Feature Extractor...")

twfe = total_word_feature_extractor('zh/total_word_feature_extractor.dat')

# Get fingerprint of feature dictionary
print ("Fingerprint of feature dictionary", twfe.fingerprint)
print ()
# Get number of dimensions of feature vectors
print ("Number of dimensions of feature vectors", twfe.num_dimensions)
print ()
# Get number of words in the dictionary
print ("Number of words in the dictionary", twfe.num_words_in_dictionary)
print ()
# Get list of words in the dictionary
words=twfe.get_words_in_dictionary()
print ("First 10 words in dictionary", words[0:200])
print ()
# Get features for one word
feats = twfe.get_feature_vector("我")
print ("First 5 features of word 'home'", feats[0:])
# The total word feature extractor will generate feature vectors for words not
# in its dictionary as well.  It does this by looking at word morphology.  
feats = twfe.get_feature_vector("_word_not_in_dictionary_")
print ("First 5 features of word '_word_not_in_dictionary_'", feats[0:50])

#输出
Number of dimensions of feature vectors 271
Number of words in the dictionary 2582
.....
...

转载：http://blog.rubenxiao.com/posts/nlp-mitie.html