万方论文基本数据爬取

最新推荐文章于 2024-06-11 15:53:02 发布

不爱睡觉不爱起床的哎呦仔

最新推荐文章于 2024-06-11 15:53:02 发布

阅读量1k

点赞数

文章标签：爬虫

本文链接：https://blog.csdn.net/weixin_47431173/article/details/113387761

版权

本文介绍如何爬取万方论文的基本信息，包括代码示例和遇到的问题，如不能爬取的内容和迭代问题的讨论，并提供了解决方案。

摘要由CSDN通过智能技术生成

爬取万方论文数据基本信息

像图中有您是否查看小水电期刊这种用代码1

此类没有小水电期刊那行的用代码2

代码1

#import ```
from selenium import webdriver
from lxml import etree
import time
import re
import pandas as pd
from selenium.webdriver.chrome.options import Options
import matplotlib.pyplot as plt
import numpy as np
from pandas import DataFrame


#变量设置 ```
#http://s.wanfangdata.com.cn/paper?q=  + input（可以是汉字）
url_first = 'http://s.wanfangdata.com.cn/paper?q='
url=url_first+input('请输入您想搜索的关键词：')
#name_xpath = ('%s%d%s' % (first_code, i, last_code_name))
#url = 'http://s.wanfangdata.com.cn/paper?q=%E6%B5%AE%E6%89%98&p=2'

headers={
   
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36'
    }

#无可视化
chrome_options = Options()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-gpu')

#设置浏览器
brower = webdriver.Chrome(executable_path='./chromedriver.exe',options=chrome_options)

#进入网页
brower.get(url)

#找到所有的element，然后用.text 遍历整个列表 进行打印

#all_content_counts = brower.find_element_by_xpath('/html/body/div[3]/div/div[2]/div[2]/div/div[3]/div[2]/div[2]/span[2]').text

#if all_content_counts%20 == 0:
#    page_counts = all_content_counts//20
#else:
#    page_counts = all_content_counts//20+1
#确定页数/最后一页可能未满现不做考虑
#page_counts = all_content_counts//20
#print(page_counts)
all_content_counts_real = brower.find_element_by_xpath('/html/body/div[3]/div/div[2]/div[2]/div/div[3]/div[2]/div[2]/span[2]').text
#/html/body/div[3]/div/div[2]/div[2]/div/div[3]/div[2]/div[2]/span[2]/span[2]
all_content_counts = re.findall('\D*([0-9]*)\D*',all_content_counts_real)[0]
page_counts = int(all_content_counts)//20
if page_counts>=10:
    page_counts=10
else:
    pass
#获得单页数据

first_code ='/html/body/div[3]/div/div[2]/div[2]/div/div[3]/div[2]/div[3]/div['
last_code_name = ']/div/div[1]/div[2]/a'
last_code_ele = ']/div/div[2]'
last_code_abs = ']/div/div[3]'
#for i in range(1,21):#1,21
#    name_xpath = ('%s%d%s' % (first_code,i,last_code_name))
#    ele_xpath = ('%s%d%s' % (first_code,i,last_code_ele))
#    abs_xpath = ('%s%d%s' % (first_code,i,last_code_abs))
#    name = brower.find_element_by_xpath(name_xpath)
#    ele = brower.find_element_by_xpath(ele_xpath)
#    abs = brower.find_element_by_xpath(abs_xpath)
#    name_url = name.get_attribute("href")
#    print(name.text, ele.text, abs.text, name_url)
#brower.find_element_by_class_name('next').click()


#如何迭代多页
#点击下一页
database=[]
for j in range(1,3):#page_counts+1
    if j <= 2:
        for i in range(1, 21):  # 1,21
            name_xpath = ('%s%d%s' % (first_code, i, last_code_name))
            ele_xpath = ('%s%d%s' % (first_code, i, last_code_ele))
            abs_xpath = ('%s%d%s' % (first_code, i, last_code_abs))
            name = brower.find_element_by_xpath(name_xpath)
            ele = brower.find_element_by_xpath(ele_xpath)
            abs = brower.find_element_by_xpath(abs_xpath)
            name_url = name.get_attribute("href")
            print(name.text, ele.text, abs.text, name_url)
            database.append([name.text, ele.text,