Task03 基于机器学习的文本分类

Task03 基于机器学习的文本分类

本次学习活动来自Coogle数据科学：30天入门数据竞赛
学习内容来自于：阿里云天池 - 零基础入门NLP - 新闻文本分类

0. 数据准备

# 导包 
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from tqdm import tqdm
import pickle
# 设置输入输出路径
data_path = 'data/'
save_path = 'result/'

# 导包
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import RidgeClassifier
from sklearn import linear_model
from sklearn.metrics import f1_score

# 读取数据文件
train_df = pd.read_csv(data_path+'train_set.csv',sep='\t')

test_df = pd.read_csv(data_path+'test_a.csv')

train_df.head()

	label	text
0	2	2967 6758 339 2021 1854 3731 4109 3792 4149 15...
1	11	4464 486 6352 5619 2465 4802 1452 3137 5778 54...
2	3	7346 4068 5074 3747 5681 6093 1777 2226 7354 6...
3	2	7159 948 4866 2109 5520 2490 211 3956 5520 549...
4	3	3646 3055 3055 2490 4659 6065 3370 5814 2465 5...

1. TFIDF 提取文本特征

词嵌入将不定长的文本转换到定长的空间内，是文本分类的第一步。

Bag of Words（词袋表示），也称为Count Vectors，在sklearn中可以直接CountVectorizer来实现“词袋”转换：

from sklearn.feature_extraction.text import CountVectorizer
corpus = [
    'This is the first document.',
    'This document is the second document.',
    'And this is the third one.',
    'Is this the first document?',
]
vectorizer = CountVectorizer()
vectorizer.fit_transform(corpus).toarray()

TF-IDF

TF-IDF 分数由两部分组成：第一部分是词语频率（Term Frequency），第二部分是逆文档频率（Inverse Document Frequency）。其中计算语料库中文档总数除以含有该词语的文档数量，然后再取对数就是逆文档频率。

• TF(t)= 该词语在当前文档出现的次数 / 当前文档中词语的总数
• IDF(t)= log_e（文档总数 / 出现该词语的文档总数+1）

CountVectorizer是属于常见的特征数值计算类，是一个文本特征提取方法。对于每一个训练文本，它只考虑每种词汇在该训练文本中出现的频率。

CountVectorizer会将文本中的词语转换为词频矩阵，它通过fit_transform函数计算各个词语出现的次数。

train_df['label'].values

array([ 2, 11,  3, ..., 11,  2,  3], dtype=int64)

# 词袋 + 岭回归

# 取出现次数最多的3000个单词作为关键词，使用fit_transform统计词频
vectorizer = CountVectorizer(max_features=3000)
train_test = vectorizer.fit_transform(train_df['text'])

# 取前10000文本个作为训练数据
clf = RidgeClassifier()
clf.fit(train_test[:10000], train_df['label'].values[:10000])

# 使用10000—200000条作为测试数据进行测试，得出f1分数
val_pred = clf.predict(train_test[10000:])
print(f1_score(train_df['label'].values[10000:], val_pred, average='macro'))
# 0.74

0.7488127202457993

2. 使用TFIDF 特征 和 线性模型完成训练和预测

TFIDF的值等于：TF x IDF

# 使用TF-IDF和岭回归完成训练和预测
# TF-IDF +  岭回归

# 取出现次数最多的3000个单词作为关键词，使用fit_transform统计词频,
tfidf = TfidfVectorizer(ngram_range=(1,3), max_features=3000)
# 得出文本TF-IDF特征
train_test = tfidf.fit_transform(train_df['text'])

clf = RidgeClassifier()
clf.fit(train_test[:10000], train_df['label'].values[:10000])

val_pred = clf.predict(train_test[10000:])
print(f1_score(train_df['label'].values[10000:], val_pred, average='macro'))

test_data = pd.read_csv(data_path+'test_a.csv')

# 提取测试数据集的TF-IDF特征
vectorizer = CountVectorizer(max_features=3000)
test_test = vectorizer.fit_transform(test_data['text'])

# 得出预测标签
test_pred_val = clf.predict(test_test)

test_pred_val

array([3, 3, 4, ..., 3, 3, 8], dtype=int64)

test_df = pd.DataFrame(test_pred_val,columns=['label']) 

type(test_df['lable'][0])

numpy.int64

# 保存预测结果
test_df.to_csv(save_path+'submit.csv',index=False,sep=',')

del sample_df

3. 使用TFIDF 特征 和 XGBoost 完成训练和预测

# 取出现次数最多的3000个单词作为关键词，使用fit_transform统计词频,
tfidf = TfidfVectorizer(ngram_range=(1,3), max_features=3000)
# 得出文本TF-IDF特征
train_test = tfidf.fit_transform(train_df['text'])

i = 0
for item in train_test:
    for word in item:
        print(word)
        break
#         i += 1
#         if i>= 1:
#             break
    break

  (0, 2861)	0.021319323290812923
  (0, 2802)	0.019959736004927054
  (0, 2194)	0.021588472538590326
  (0, 891)	0.018603301344074175
  (0, 1247)	0.020608661311821395
  (0, 2038)	0.02240483997783033
  (0, 2996)	0.022130915429207997
  (0, 2923)	0.044051176856073015
  (0, 1183)	0.05097171821741816
  :	:
  (0, 422)	0.027119239565399852
  (0, 197)	0.057173982210201105
  (0, 1414)	0.027833712233284878
  (0, 1301)	0.09802535247134912
  (0, 1387)	0.17425122880337085
  (0, 1150)	0.1704436496531144
  (0, 328)	0.011162664461440474
  (0, 412)	0.032072202918363046
  (0, 2600)	0.09225915260965885
  (0, 791)	0.08049002269211457

# 提取测试数据集的TF-IDF特征
vectorizer = TfidfVectorizer(ngram_range=(1,3), max_features=3000)
test_test = tfidf.fit_transform(test_df['text'])

from xgboost import XGBClassifier

xgm = XGBClassifier()
xgm.fit(train_test, train_df['label'])

D:\ProgramData\Anaconda3\lib\site-packages\xgboost\sklearn.py:1146: UserWarning: The use of label encoder in XGBClassifier is deprecated and will be removed in a future release. To remove this warning, do the following: 1) Pass option use_label_encoder=False when constructing XGBClassifier object; and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1].
  warnings.warn(label_encoder_deprecation_msg, UserWarning)


[18:54:32] WARNING: D:\Build\xgboost\xgboost-1.4.2.git\src\learner.cc:1095: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'multi:softprob' was changed from 'merror' to 'mlogloss'. Explicitly set eval_metric if you'd like to restore the old behavior.


XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
              colsample_bynode=1, colsample_bytree=1, gamma=0, gpu_id=-1,
              importance_type='gain', interaction_constraints='',
              learning_rate=0.300000012, max_delta_step=0, max_depth=6,
              min_child_weight=1, missing=nan, monotone_constraints='()',
              n_estimators=100, n_jobs=8, num_parallel_tree=1,
              objective='multi:softprob', random_state=0, reg_alpha=0,
              reg_lambda=1, scale_pos_weight=None, subsample=1,
              tree_method='exact', validate_parameters=1, verbosity=None)

fit_pred = xgm.predict(test_test)

test_df = pd.DataFrame(fit_pred,columns=['label']) 

# 保存预测结果
test_df.to_csv('result/submit_whole.csv',index=False,sep=',')