使用Python爬取知网信息

最新推荐文章于 2024-10-09 14:48:20 发布
最新推荐文章于 2024-10-09 14:48:20 发布
阅读量2.8k 收藏 41
点赞数 3
文章标签： python 开发语言 后端
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qingshui9/article/details/122148079
版权
该博客介绍了如何使用Python爬虫从知网获取特定年份（1998-2022）的科研论文信息，包括题目、作者、单位、所属期刊、发表时间、下载数、摘要、关键词、专题和分类号。爬虫通过模拟POST请求，解析HTML页面，提取所需数据并保存到CSV文件中。内容涉及网页解析库如lxml和正则表达式，以及使用requests库进行网络请求。
摘要由CSDN通过智能技术生成

使用Python爬取知网信息

import requests
from urllib import request
from lxml import etree
import re
import csv, time
from w3lib.html import remove_tags


def write_data(name):
    #timenow = time.strftime("%Y-%m-%d-%H%M%S", time.localtime())
    file = name
    with open(file, 'a+', encoding='utf-8-sig', newline='') as f:
        writer_f = csv.writer(f)
        writer_f.writerow(['题目','作者','单位','所属期刊','发表时间','下载数','摘要','关键词','专题','分类号'])
    return file
def get_one(year):
    data = {
        'IsSearch': 'true',
        #CO33_1,_2
        'QueryJson': '{"Platform":"","DBCode":"CJFQ","KuaKuCode":"","QNode":{"QGroup":[{"Key":"Subject","Title":"","Logic":4,"Items":[],"ChildItems":[]},{"Key":"ControlGroup","Title":"","Logic":1,"Items":[],"ChildItems":[{"Key":".extend-tit-checklist","Title":"","Logic":1,"Items":[{"Key":0,"Title":"SCI","Logic":2,"Name":"SI","Operate":"=","Value":"Y","ExtendType":14,"ExtendValue":"","Value2":"","BlurType":""},{"Key":0,"Title":"EI","Logic":2,"Name":"EI","Operate":"=","Value":"Y","ExtendType":14,"ExtendValue":"","Value2":"","BlurType":""},{"Key":0,"Title":"北大核心","Logic":2,"Name":"HX","Operate":"=","Value":"Y","ExtendType":14,"ExtendValue":"","Value2":"","BlurType":""},{"Key":0,"Title":"CSSCI","Logic":2,"Name":"CSI","Operate":"=","Value":"Y","ExtendType":14,"ExtendValue":"","Value2":"","BlurType":""},{"Key":0,"Title":"CSCD","Logic":2,"Name":"CSD","Operate":"=","Value":"Y","ExtendType":14,"ExtendValue":"","Value2":"","BlurType":""}],"ChildItems":[]}]},{"Key":"NaviParam","Title":"","Logic":1,"Items":[{"Key":"navi","Title":"","Logic":1,"Name":"专题子栏目代码","Operate":"=","Value":"C033_8?","ExtendType":13,"ExtendValue":"","Value2":"","BlurType":""}],"ChildItems":[]},{"Key":"MutiGroup","Title":"","Logic":1,"Items":[],"ChildItems":[{"Key":"3","Title":"","Logic":1,"Items":[{"Key":"'+year+'","Title":"'+year+'","Logic":2,"Name":"年","Operate":"","Value":"'+year+'","ExtendType":0,"ExtendValue":"","Value2":"","BlurType":""}],"ChildItems":[]}]}]}}',
        'PageName': 'AdvSearch',
        'DBCode': 'CJFQ',
        # 'KuaKuCodes': '',
        'CurPage': '1'
    }
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 Edg/91.0.864.70',
        'Referer': 'https://kns.cnki.net/KNS8/AdvSearch?dbcode=CJFQ',
        #'Cookie' : 'Ecp_ClientId=2210720103601729746; Ecp_IpLoginFail=210720106.117.97.127; cnkiUserKey=3c237cd1-59ab-91cb-22ce-bbfc3ad33332; Ecp_ClientIp=106.117.97.127; RsPerPage=20; ASP.NET_SessionId=fkwgrvclzgkpqsagarxa5mdb; SID_kcms=124120; _pk_ref=["","",1626830042,"https://www.cnki.net/"]; _pk_ses=*; SID_kns_new=kns123112; SID_kns8=123122; CurrSortField=发表时间/(发表时间,'TIME')+desc; CurrSortFieldType=desc; _pk_id=2de25f01-dfc6-40ba-888f-beabd2c0efb8.1626748599.8.1626830093.1626830042.'
    }
    url = 'https://kns.cnki.net/KNS8/Brief/GetGridTableHtml'
    response = requests.post(url,headers=headers,data=data).text
    #print(response)
    pattern = re.compile(
        '<div class=\'pages\'> <span class="total">共(.*?)页</span>.*?</div>',
        re.S
    )
    item = re.findall(pattern,response)
    return item[0]

def get_two(year,i,file):
    data = {
        'IsSearch': 'true',
        'QueryJson': '{"Platform":"","DBCode":"CJFQ","KuaKuCode":"","QNode":{"QGroup":[{"Key":"Subject","Title":"","Logic":4,"Items":[],"ChildItems":[]},{"Key":"ControlGroup","Title":"","Logic":1,"Items":[],"ChildItems":[{"Key":".extend-tit-checklist","Title":"","Logic":1,"Items":[{"Key":0,"Title":"SCI","Logic":2,"Name":"SI","Operate":"=","Value":"Y","ExtendType":14,"ExtendValue":"","Value2":"","BlurType":""},{"Key":0,"Title":"EI","Logic":2,"Name":"EI","Operate":"=","Value":"Y","ExtendType":14,"ExtendValue":"","Value2":"","BlurType":""},{"Key":0,"Title":"北大核心","Logic":2,"Name":"HX","Operate":"=","Value":"Y","ExtendType":14,"ExtendValue":"","Value2":"","BlurType":""},{"Key":0,"Title":"CSSCI","Logic":2,"Name":"CSI","Operate":"=","Value":"Y","ExtendType":14,"ExtendValue":"","Value2":"","BlurType":""},{"Key":0,"Title":"CSCD","Logic":2,"Name":"CSD","Operate":"=","Value":"Y","ExtendType":14,"ExtendValue":"","Value2":"","BlurType":""}],"ChildItems":[]}]},{"Key":"NaviParam","Title":"","Logic":1,"Items":[{"Key":"navi","Title":"","Logic":1,"Name":"专题子栏目代码","Operate":"=","Value":"C033_8?","ExtendType":13,"ExtendValue":"","Value2":"","BlurType":""}],"ChildItems":[]},{"Key":"MutiGroup","Title":"","Logic":1,"Items":[],"ChildItems":[{"Key":"3","Title":"","Logic":1,"Items":[{"Key":"'+year+'","Title":"'+year+'","Logic":2,"Name":"年","Operate":"","Value":"'+year+'","ExtendType":0,"ExtendValue":"","Value2":"","BlurType":""}],"ChildItems":[]}]}]}}',
        'PageName': 'AdvSearch',
        'DBCode': 'CJFQ',
        #'KuaKuCodes': '',
        'CurPage': i
    }
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 Edg/91.0.864.70',
        'Referer': 'https://kns.cnki.net/KNS8/AdvSearch?dbcode=CJFQ',
        # 'Cookie' : ''
    }
    url = 'https://kns.cnki.net/KNS8/Brief/GetGridTableHtml'

    response = requests.post(url, headers=headers, data=data).text
    #print(response)
    pattern = re.compile(
        '<tr.*?<td class="name">.*?<a class="fz14" href=.*?&DbCode=(.*?)&dbname=(.*?)&filename=(.*?)&urlid.*?>.*?</a>.*?</td>.*?<td class="source">(.*?)</td>.*?<td class="date">(.*?)</td>.*?<td class="download">(.*?)</td>.*?</tr>',
        re.S
    )
    items = re.findall(pattern,response)
    for item in items:
        key_words = ' '
        dbcode= item[0]
        dbname = item[1]
        filename = item[2]
        periodical = remove_tags(item[3]).strip() # 所属期刊
        date = str(item[4]).strip()  #发表日期
        download = remove_tags(item[5]).strip()  #下载数量
        if len(download) == 0 :
            download = 0
        #print(date,download)
        url_2 = 'https://kns.cnki.net/kcms/detail/detail.aspx?dbcode=' + dbcode + '&dbname=' + dbname + '&filename=' + filename
        print(url_2)
        # print(subject)
        response_2 = requests.get(url_2)
        text = response_2.content.decode('utf-8')
        pattern_2 = re.compile(
            '<div class="brief">.*?<div class="wx-tit">.*?<h1>(.*?)</h1>.*?<h3.*?>(.*?)</h3>.*?<h3.*?>(.*?)</h3>.*?</div>.*?</div>',
            re.S
        )
        items_2 = re.findall(pattern_2, text)
        for item_2 in items_2:
            subject = remove_tags(item_2[0])  #题目
            author = remove_tags(item_2[1].replace('</span>','|')).replace('1','').replace('2','').replace('3','').replace('4','').replace('5','').replace('6','').replace('7','').replace('8','').replace('','').replace(",",'')
            print(author)
            company = remove_tags(item_2[2].replace('</span>','|')).replace('&nbsp;','').replace('1. ','').replace('2. ','|').replace('3. ','|').replace('4. ','|').replace('5. ','|').replace('6. ','|')#单位
        pattern_3 = re.compile(
            '<div class="row"><span class="rowtit">.*?class="abstract-text">(.*?)</span>.*?</script>',
            re.S
        )
        items_3 = re.findall(pattern_3,text)
        if len(items_3)==0:
            abstract = ' '
        else:
            abstract = items_3[0].replace('&lt;正&gt;','')

        html = etree.HTML(text)
        kkk = html.xpath("//p[@class='keywords']")  #关键词
        ggg = html.xpath("//div[@class='row']/ul/li")
        if len(ggg) == 0:
            classification_Number = ' '
            special = ' '
        else:
            classification_Number = str(ggg[-1].xpath("./p/text()")).replace('\'','').replace('[','').replace(']','').replace(';',' ')#分类号
            special = str(ggg[-2].xpath("./p/text()")).replace('\'','').replace('[','').replace(']','').replace(';',' ') #专题
        if len(kkk) == 0:
            with open(file, 'a+', encoding='utf-8-sig', newline='') as f:
                writer = csv.writer(f)
                writer.writerow([subject,author,company,periodical,date,download,abstract,key_words,special,classification_Number])
        else:
            key_words = str(kkk[0].xpath("./a/text()"))
            key_words = key_words.replace(' ','').replace('\'','').replace(';','').replace('[','').replace(']','').replace('\\','').replace('rn','').replace(',',' ')
            with open(file, 'a+', encoding='utf-8-sig', newline='') as f:
                writer = csv.writer(f)
                writer.writerow([subject,author,company,periodical,date,download,abstract,key_words,special,classification_Number])
name = '铁路运输管理工程.csv'
file = write_data(name)
for year in range(1998,2022):
    year=str(year)
    number = int(get_one(year))
    for i in range(1,number+1):
        get_two(year,i,file)
    time.sleep(1)
print('下载完成')
# year = '2021' 8051
# file = write_data(year)
# number = get_one(year)
# number = int(number)
# for i in range(number):
#     get_two(i,file)
#     time.sleep(1)
# print('下载完成')


#改进：1.按年份自动搜索
#     2.年份进入之后自动统计总页数 然后按页数进行爬取
孤落时辰·
关注
用Python批量获取知网文献信息，轻松解放双手 ！
python03012的博客
11-22 4423
能够很容易定位到摘要的文本，class name：
【爬虫实战】Python爬取知网文献信息
程序员小芽的博客
06-20 4209
Python所有方向路线就是把Python常用的技术点做整理，形成各个领域的知识点汇总，它的用处就在于，你可以按照上面的知识点去找对应的学习资源，保证自己学得较为全面。
使用python脚本抓取知网论文数据demo
最新发布
weixin_60619970的博客
10-09 921
使用python脚本抓取知网论文数据demo
python知网爬虫
04-28
python知网爬虫，根据作者，爬取所有paper信息
python爬取知网论文信息
热门推荐
little-book
03-27 2万+
背景 老师让我爬图书馆所有关于地学的资料，并用深度学习分析。 当时让我爬图书馆，内心是极度崩溃的！！！我还是个孩子啊！！！ 很幸运的找到了一个大佬的代码来自己改改改！ 先放大佬代码连接 Git 接下来就是我自己嚯嚯嚯改的很小白的了。。。 应该是很详细得了，我是纯小白，小白的。学Python很短的时间，用到什么才去学什么。 为了看懂这个代码， 我先看了另外一个大佬的比较一点简单代码...
python爬虫知网实例-python爬取知网
weixin_37988176的博客
10-30 7621
广告关闭腾讯云双11爆品提前享，精选热门产品助力上云，云服务器首年88元起，买的越多返的越多，最高满返5000元！https:github.comgnemougdistribute_crawlercnkispider- 中国知网爬虫。 设置检索条件后，执行srccnkispider.py抓取数据，抓取数据存储在data目录下...如果整站爬全部文本和图片作为离线查询，大概需要10g空间、2小时（1...
python爬虫爬取知网
guoxuying的博客
08-24 8382
python爬虫爬取知网 话不多说，直接上代码！ import requests import re import time import xlrd from xlrd import open_workbook from xlutils.copy import copy class BeautifulPicture(): def get_pic(self): data = xlrd.open_workbook(r'C:\\flim\library_my\new.xls') #
Python爬虫知网
THDFB的博客
08-21 1667
听说Python爬虫知网很火，我今天就做一个。 上代码： import requests import re import time import xlrd from xlrd import open_workbook from xlutils.copy import copy class BeautifulPicture(): def get_pic(self): data = xlrd.open_workbook(r'C:\\flim\library_my\new.xls')
python爬虫论文摘要怎么写_Python爬虫根据关键词爬取知网论文摘要并保存到数据库中【入门必学】...
weixin_39719101的博客
11-20 621
搜索出来的结果和知网上的结果几乎一样，另外以后面试找Python工作，项目经验展示是核心，如果你缺项目练习，去小编的Python交流.裙 ：一久武其而而流一思（数字的谐音）转换下可以找到了，里面很多新教程项目在这个基础上，我简单看了些网页的结构，很容易就能写出爬取得代码（是最基础的，相当不完善，增加其他功能可自行增加）网页的结构还是很清晰的摘要信息也很清晰我使用的是 pymysql 连接的数据库，...
python selenium爬虫_python爬虫——使用selenium爬取知网文献相关信息
weixin_39631519的博客
11-27 845
很多同学已经在奔赴毕业的道路上啦，都要面临一个难题，那就是写论文。不少同学为了写论文熬出了黑眼圈，却仍然一无所获，被论文折磨的死去活来，爱恨交加, 写论文必不可少的步骤就是查资料。古人云:“书读百遍其义自现”, 说不定知网翻多了，你也就知道自己的论文该如何写了。所以小编今天为大家整理了文献获取葵花plus，让你写论文不用愁。写在前面：本文章限于交流讨论，请不要使用文章的代码去攻击别...
毕业论文找文献是个问题，我直接用python把全网文献爬了一遍，这波就很舒服
人生苦短， 还不用Python？
12-17 6833
马上要毕业了兄弟们，毕业论文是个麻烦事，论文要的资料得一条一条去网上查看，那多浪费时间，咱直接写个爬虫，批量下载慢慢看，不舒服？
Python-一个中国知网的爬虫工具给定作者可以获得该作者的所有文献的题录
08-12
一个中国知网的爬虫工具，给定作者，可以获得该作者的所有文献的题录
python爬取知网论文关键词_Python爬虫根据关键词爬取知网论文摘要并保存到数据库中【入门必学】...
weixin_39750195的博客
12-01 1397
搜索出来的结果和知网上的结果几乎一样，另外以后面试找Python工作，项目经验展示是核心，如果你缺项目练习，去小编的Python交流.裙 ：一久武其而而流一思(数字的谐音)转换下可以找到了，里面很多新教程项目在这个基础上，我简单看了些网页的结构，很容易就能写出爬取得代码(是最基础的，相当不完善，增加其他功能可自行增加)网页的结构还是很清晰的摘要信息也很清晰我使用的是 pymysql 连接的数据库，...
python爬虫爬取文献数据
m0_66526403的博客
05-25 4674
文章利用python，安装Selenium工具爬取知网数据
评论 3
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

孤落时辰·

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值