文本分类:短文本分类

应用场景:我们在和智能产品(如智能音箱,服务机器人,智能电话,车机,手机助手等)对话时。产生的一般都是口语化的短文本。那么如何对短文本进行分类来识别用户意图呢。下面介绍一下集中常用的文本分类技术

1.几种文本分类算法的对比
文本分类实质上是一种分类技术在文本领域的应用。时下最时髦的做法便是用神经网络,比如用CNN进行文本分类。据说会比传统机器学习算法的效果会好。但是训练神经网络需要充足的语料来训练,否则就会出现过拟合的通病。网络中的参数会和真实值相差甚远。因此,短文本如果用CNN这种网络显得有点杀鸡用牛刀的感觉。
在神经网络出现之前,svm(支持向量机)在分类方面称得上是独领风骚。它最重要的一个优点是在小样本训练集上的分类效果会比其他算法好很多。这者特点使它得到了广泛的应用。
2.textGrocery
TextGrocery是一个基于LibLinear和结巴分词的短文本分类工具,特点是高效易用,同时支持中文和英文语料。地址如下https://textgrocery.readthedocs.io/zh/latest/index.html。其中liblinear是libsvm的线性核的改进版本,专门适用于百万数据量的分类。是目前比较好的用于短文本分类的算法

  • 0
    点赞
  • 10
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
496,835 条来自 AG 新闻语料库 4 大类别超过 2000 个新闻源的新闻文章,数据集仅仅援用了标题和描述字段。每个类别分别拥有 30,000 个训练样本及 1900 个测试样本。 README: AG's News Topic Classification Dataset Version 3, Updated 09/09/2015 ORIGIN AG is a collection of more than 1 million news articles. News articles have been gathered from more than 2000 news sources by ComeToMyHead in more than 1 year of activity. ComeToMyHead is an academic news search engine which has been running since July, 2004. The dataset is provided by the academic comunity for research purposes in data mining (clustering, classification, etc), information retrieval (ranking, search, etc), xml, data compression, data streaming, and any other non-commercial activity. For more information, please refer to the link http://www.di.unipi.it/~gulli/AG_corpus_of_news_articles.html . The AG's news topic classification dataset is constructed by Xiang Zhang (xiang.zhang@nyu.edu) from the dataset above. It is used as a text classification benchmark in the following paper: Xiang Zhang, Junbo Zhao, Yann LeCun. Character-level Convolutional Networks for Text Classification. Advances in Neural Information Processing Systems 28 (NIPS 2015). DESCRIPTION The AG's news topic classification dataset is constructed by choosing 4 largest classes from the original corpus. Each class contains 30,000 training samples and 1,900 testing samples. The total number of training samples is 120,000 and testing 7,600. The file classes.txt contains a list of classes corresponding to each label. The files train.csv and test.csv contain all the training samples as comma-sparated values. There are 3 columns in them, corresponding to class index (1 to 4), title and description. The title and description are escaped using double quotes ("), and any internal double quote is escaped by 2 double quotes (""). New lines are escaped by a backslash followed with an "n" character, that is "\n".

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值