NLIP - ML-based 文本分类

最新推荐文章于 2021-12-23 22:52:36 发布

wjsjjss

最新推荐文章于 2021-12-23 22:52:36 发布

阅读量303

点赞数

分类专栏：深度学习

本文链接：https://blog.csdn.net/wjsjjss/article/details/107586123

版权

深度学习专栏收录该内容

23 篇文章 1 订阅

订阅专栏

ML-based 文本分类

基于sklearn的one-hot向量表示示例

from sklearn.feature_extraction.text import CountVectorizer
corpus = [
    'This is the first document.',
    'This document is the second document.',
    'And this is the third one.',
    'Is this the first document?',
]
vectorizer = CountVectorizer()
vectorizer.fit_transform(corpus).toarray()

Count Vectors + RidgeClassifier文本分类

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.linear_model import RidgeClassifier
from sklearn.metrics import f1_score

train_df = pd.read_csv('train_set.csv', sep='\t')

vectorizer = CountVectorizer(max_features=3000)
train_test = vectorizer.fit_transform(train_df['text'])

clf = RidgeClassifier()
clf.fit(train_test[:10000], train_df['label'].values[:10000])

val_pred = clf.predict(train_test[10000:])
print(f1_score(train_df['label'].values[10000:], val_pred, average='macro'))

基于TF-IDF——RidgeClassifier的文本分类

TF(t)= 该词语在当前文档出现的次数 / 当前文档中词语的总数¶
IDF(t)= log_e（文档总数 / 出现该词语的文档总数）

from sklearn.feature_extraction.text import TfidfVectorizer


train_df = pd.read_csv('train_set.csv', sep='\t',)

tfidf = TfidfVectorizer(ngram_range=(1,3), max_features=3000)
train_test = tfidf.fit_transform(train_df['text'])

clf = RidgeClassifier()
clf.fit(train_test[:10000], train_df['label'].values[:10000])

val_pred = clf.predict(train_test[10000:])
print(f1_score(train_df['label'].values[10000:], val_pred, average='macro'))