开源一个文本分析项目

原创 2017年06月12日 18:18:42

Github

https://github.com/sea-boat/TextAnalyzer

TextAnalyzer

a text analizer that can analyze text. so far, it can extract hot words in a text segment by using tf-idf algorithm,at the same time using a score factor to optimize the final score.

also it provides machine learning to make a classification.

Features

extracting hot words from a text.
1. to gather statistics via frequence.
2. to gather statistics via by tf-idf algorithm
3. to gather statistics via a score factor additionally.

synonym can be recognized

SVM Classificator

this analyzer supports to classify text by svm. it involves vectoring the text. we can train the samples and then make a classification by the model.

for convenience,the model,tfidf and vector will be stored.

kmeans clustering && xmeans clustering

this analyzer supports to clustering text by kmeans and xmeans.

vsm clustering

this analyzer supports to clustering text by vsm.

Dependence

https://github.com/sea-boat/IKAnalyzer-Mirror.git

TODO

  • other ml algorithms.
  • emotion analization.

How to use

just simple like this

extracting hot words

  1. indexing a document and get a docId.
long docId = TextIndexer.index(text);
  1. extracting by docId.
 HotWordExtractor extractor = new HotWordExtractor();
 List<Result> list = extractor.extract(0, 20, false);
 if (list != null) for (Result s : list)
    System.out.println(s.getTerm() + " : " + s.getFrequency() + " : " + s.getScore());

a result contains term,frequency and score.

失业证 : 1 : 0.31436604
户口 : 1 : 0.30099702
单位 : 1 : 0.29152703
提取 : 1 : 0.27927202
领取 : 1 : 0.27581802
职工 : 1 : 0.27381304
劳动 : 1 : 0.27370203
关系 : 1 : 0.27080503
本市 : 1 : 0.27080503
终止 : 1 : 0.27080503

SVM classificator

  1. training the samples.
SVMTrainer trainer = new SVMTrainer();
trainer.train();
  1. predicting text classification.
double[] data = trainer.getWordVector(text);
trainer.predict(data);

kmeans clustering && xmeans clustering

List<String> list = DataReader.readContent(KMeansCluster.DATA_FILE);
int[] labels = new KMeansCluster().learn(list);

vsm clustering

List<String> list = DataReader.readContent(VSMCluster.DATA_FILE);
List<String> labels = new VSMCluster().learn(list);

==========广告时间==========

鄙人的新书《Tomcat内核设计剖析》已经在京东预售了,有需要的朋友可以到 https://item.jd.com/12185360.html 进行预定。感谢各位朋友。

=========================

欢迎关注:
这里写图片描述

版权声明:本文为博主原创文章,未经博主允许不得转载。

相关文章推荐

支持中文文本的数据挖掘平台开源项目PyMining发布

本文转自http://www.cnblogs.com/LeftNotEasy/archive/2011/02/27/py_mining_first_release.html 前言   ...

Linux下对NS2项目的仿真分析全过程代码(从tcl脚本-awk文本处理-gnuplot)

1、编写 tcl 脚本文件,得到 out.tr 以及 out.nam 文件;out.tr文件存储的是仿真得到的数据运行代码:创建tcl文件:touch example.tcl (example为文件名...

如何参与一个 GitHub 开源项目?

【说明】本文原载于码农IO(manong.io)官方微信 developerWorks,转载、引用请注明出处及作者。 原文地址:https://guides.github.com/...
内容举报
返回顶部
收藏助手
不良信息举报
您举报文章:深度学习:神经网络中的前向传播和反向传播算法推导
举报原因:
原因补充:

(最多只允许输入30个字)