Task4 论文种类分类

任务说明

• 学习主题：论文分类（数据建模任务），利用已有数据建模，对新论文进行类别分类；
• 学习内容：使用论文标题完成类别分类；
• 学习成果：学会文本分类的基本方法、TF-IDF等；

数据处理步骤

在原始arxiv论文中论文都有对应的类别，而论文类别是作者填写的。在本次任务中我们可以借助论文的标题和摘要完成：

• 对论文标题和摘要进行处理；
• 对论文类别进行处理；
• 构建文本分类模型；

文本分类思路

• 思路1：TF-IDF+机器学习分类器

直接使用TF-IDF对文本提取特征，使用分类器进行分类，分类器的选择上可以使用SVM、LR、XGboost等

• 思路2：FastText

FastText是入门款的词向量，利用Facebook提供的FastText工具，可以快速构建分类器

• 思路3：WordVec+深度学习分类器

WordVec是进阶款的词向量，并通过构建深度学习分类完成分类。深度学习分类的网络结构可以选择TextCNN、TextRnn或者BiLSTM。

• 思路4：Bert词向量

Bert是高配款的词向量，具有强大的建模学习能力。

具体代码实现以及讲解

为了方便大家入门文本分类，我们选择思路1和思路2给大家讲解。首先完成字段读取：

# 导入所需的package
import seaborn as sns #用于画图
from bs4 import BeautifulSoup #用于爬取arxiv的数据
import re #用于正则表达式，匹配字符串的模式
import requests #用于网络连接，发送网络请求，使用域名获取对应信息
import json #读取数据，我们的数据为json格式的
import pandas as pd #数据处理，数据分析
import numpy as np
import matplotlib.pyplot as plt #画图工具

def readArxivFile(path, columns=['id', 'submitter', 'authors', 'title', 'comments', 'journal-ref', 'doi',
       'report-no', 'categories', 'license', 'abstract', 'versions',
       'update_date', 'authors_parsed'], count=None):
    '''
    定义读取文件的函数
        path: 文件路径
        columns: 需要选择的列
        count: 读取行数
    '''
    
    data  = []
    with open(path, 'r') as f: 
        for idx, line in enumerate(f): 
            if idx == count:
                break
                
            d = json.loads(line)
            d = {
   col : d[col] for col in columns}
            data.append(d)

    data = pd.DataFrame(data)
    return data

data = readArxivFile('./data/arxiv-metadata-oai-2019.json', 
                     ['id', 'title', 'categories', 'abstract'],
                    200000)

data.head()

	id	title	categories	abstract
0	0704.0297	Remnant evolution after a carbon-oxygen white ...	astro-ph	We systematically explore the evolution of t...
1	0704.0342	Cofibrations in the Category of Frolicher Spac...	math.AT	Cofibrations are defined in the category of ...
2	0704.0360	Torsional oscillations of longitudinally inhom...	astro-ph	We explore the effect of an inhomogeneous ma...
3	0704.0525	On the Energy-Momentum Problem in Static Einst...	gr-qc	This paper has been removed by arXiv adminis...
4	0704.0535	The Formation of Globular Cluster Systems in M...	astro-ph	The most massive elliptical galaxies show a ...

为了方便数据的处理，我们可以将标题和摘要拼接一起完成分类。

• 英文text的处理流程
  1. 去掉非英文的标点–当然也depend on英文书写格式 eg: 标准版–i like apple, what about you(标点后面会跟着空格) don’t直接删除标点
  2. 变小写
  3. trim 左右两边的whitespace
  4. 用空格分词: split(’ ')–>string变成单个单词str组成的list
  5. 去停用词
  6. stemming
  7. lemmatization
  8. （optional）将list中的string，用’ '.join重新组合成string

# 这里是分解
data['text'] = data['title'] + data['abstract']

data.text[0] # 里面有很多换行符\n 要替换

'Remnant evolution after a carbon-oxygen white dwarf merger  We systematically explore the evolution of the merger of two carbon-oxygen\n(CO) white dwarfs. The dynamical evolution of a 0.9 Msun + 0.6 Msun CO white\ndwarf merger is followed by a three-dimensional SPH simulation. We use an\nelaborate prescription in which artificial viscosity is essentially absent,\nunless a shock is detected, and a much larger number of SPH particles than\nearlier calculations. Based on this simulation, we suggest that the central\nregion of the merger remnant can, once it has reached quasi-static equilibrium,\nbe approximated as a differentially rotating CO star, which consists of a\nslowly rotating cold core and a rapidly rotating hot envelope surrounded by a\ncentrifugally supported disc. We construct a model of the CO remnant that\nmimics the results of the SPH simulation using a one-dimensional hydrodynamic\nstellar evolution code and then follow its secular evolution. The stellar\nevolution models indicate that the growth of the cold core is controlled by\nneutrino cooling at the interface between the core and the hot envelope, and\nthat carbon ignition in the envelope can be avoided despite high effective\naccretion rates. This result suggests that the assumption of forced accretion\nof cold matter that was adopted in previous studies of the evolution of double\nCO white dwarf merger remnants may not be appropriate. Our results imply that\nat least some products of double CO white dwarfs merger may be considered good\ncandidates for the progenitors of Type Ia supernovae. In this case, the\ncharacteristic time delay between the initial dynamical merger and the eventual\nexplosion would be ~10^5 yr. (Abridged).\n'

data['text'] = data['text'].apply(lambda x: x.replace('\n',' ')) # 对于text这一列的每个值（string）--x：把\n换行符替换成空格str' '

data.text[0]

'Remnant evolution after a carbon-oxygen white dwarf merger  We systematically explore the evolution of the merger of two carbon-oxygen (CO) white dwarfs. The dynamical evolution of a 0.9 Msun + 0.6 Msun CO white dwarf merger is followed by a three-dimensional SPH simulation. We use an elaborate prescription in which artificial viscosity is essentially absent, unless a shock is detected, and a much larger number of SPH particles than earlier calculations. Based on this simulation, we suggest that the central region of the merger remnant can, once it has reached quasi-static equilibrium, be approximated as a differentially rotating CO star, which consists of a slowly rotating cold core and a rapidly rotating hot envelope surrounded by a centrifugally supported disc. We construct a model of the C