德温特专利数据的爬取（selenium\xpath\contains解决了输入框ID老是动态改变的问题）

（一）目标

针对一系列机构名，获取2016-2021年间的每年申请专利数目以及总数、专利家族数、专利授权量、专利被引频次。

（二）方法

1、使用Webdriver模拟人工访问浏览器

步骤为：
（1）定义检索情况：





（2）定位、计算总被引量
从检索结果中定位每个专利的被引量，降序排列后加和所有专利的被引量得到总的被引量

（3）从筛选器中获取每年的申请量：


（3）从筛选器中查看授权/申请状况

2、BeautifulSoup解析页面

这里，有个坑就是“一定要在点击完页面所有操作的按钮之后再进行解析”，这样会避免后面点击的内容解析不出来的情况。

（三）代码实现

from selenium import webdriver
import time
import json
from pprint import pprint
import requests
import redis
import json
import re
import random
from bs4 import BeautifulSoup
import xlwt
work_book = xlwt.Workbook()

driver = webdriver.Chrome()
driver.get(url='https://derwentinnovation.clarivate.com.cn/login/')
time.sleep(2)
driver.find_element_by_xpath('//*[@id="tr-login-username"]').click()
driver.find_element_by_xpath('//*[@id="tr-login-username"]').clear()
driver.find_element_by_xpath('//*[@id="tr-login-username"]').send_keys('pengh@mail.las.ac.cn')
driver.find_element_by_xpath('//*[@id="tr-login-password"]').click()
driver.find_element_by_xpath('//*[@id="tr-login-password"]').clear()
driver.find_element_by_xpath('//*[@id="tr-login-password"]').send_keys('pengh2018#')
time.sleep(2)
driver.find_element_by_xpath('//*[@id="tr-email-form"]/div/div/div/input').click()
time.sleep(10)

#机构名
lists = [
   ...
]


f = open('result.csv','w',encoding='utf-8')
import  csv
wr_f = csv.writer(f)
for li in lists:
    driver.get(url='https://derwentinnovation.clarivate.com.cn/ui/zh/#/home')
    time.sleep(6)
    '//*[@id="mat-input-61"]'
    driver.find_element_by_xpath('/html/body/di-app/div[1]/div[2]/di-app-home/main/section[1]/div/di-app-smart-search/div/div/di-app-search-type/div[1]/mat-card/a').click()
    time.sleep(6)
    print(li)
    app = 0
    grant = 0
    # start querying
    #first quesy

    driver.find_element_by_xpath('//textarea[contains(@placeholder,"Fanuc")]').click()
    driver.find_element_by_xpath('//textarea[contains(@placeholder,"Fanuc")]').clear()
    driver.find_element_by_xpath('//textarea[contains(@placeholder,"Fanuc")]').send_keys(li)
    time.sleep(5)
    #  query button
    driver.find_element_by_xpath('//button[contains(@class,"but-xl ps-button-3 mat-flat-button mat-button-base mat-primary")]').click()

    time.sleep(10)
    # sort
    try:
        driver.find_element_by_xpath('//*[@id="dataTable_scrollHead_Id"]/div/table/thead/tr/th[12]/div').click()
    except:
        pass
    time.sleep(16)
    # year date
    driver.find_element_by_xpath('//*[@id="filter-sticky-bar"]/div/button').click()
    time.sleep(6)
    try:
        driver.find_elements_by_xpath("//mat-icon[.='arrow_right']")[6].click()
    except:
        wr_f.writerow([li,0,0,0,0,0,0,0,0,0,0,0])
        continue
    time.sleep(10)
    # analyze data
    html = driver.page_source.encode('utf-8')
    soup =BeautifulSoup(html,'lxml')

    #get the zhuanli申请

    content_div = soup.find_all("div",{"class":"mat-radio-label-content"})

    # for div in content_div:
    #     for span in div.find_all('span'):
    #         print(span.getText())
    aa = str(content_div[0].find_all('span')[1].getText()).strip().replace(',','')
    aa = int(aa)
    bb = str(content_div[2].find_all('span')[1].getText()).strip().replace(',','')
    if bb !='':
        bb = int(bb)
    print(aa)
    print(bb)
 

    if aa == '0':
        continue
    else:
        # get the patent_apply for each year
        content1 = soup.find_all("span",{"class":"mat-checkbox-label"})
        temp_2015 = 0
        temp_2016 = 0
        temp_2017 = 0
        temp_2018 = 0
        temp_2019 = 0
        temp_2020 = 0

        for con1 in content1[-14:]:
            time.sleep(2)
            temp = str(con1.getText()).strip()

            if (temp[0] == '2' and 6<len(temp)<=10):
                if temp[0:4] == '2015':
                    temp_2015 += int(temp[5:-1])
                elif temp[0:4] == '2016':
                    temp_2016 += int(temp[5:-1])
                elif temp[0:4] == '2017':
                    temp_2017 += int(temp[5:-1])
                elif temp[0:4] == '2018':
                    temp_2018 += int(temp[5:-1])
                elif temp[0:4] == '2019':
                    temp_2019 += int(temp[5:-1])
                elif temp[0:4] == '2020':
                    temp_2020 += int(temp[5:-1])
            elif temp[0] == 'A':
                app = temp[5:-1]
                print(app)
            elif temp[0] == 'G':
                grant = temp[7:-1]
                print(grant)
        print(temp_2015)
        print(temp_2016)
        print(temp_2017)
        print(temp_2018)
        print(temp_2019)
        print(temp_2020)

        #get the sum citation value
        sum = 0
        content2 = soup.find_all("div",{"class":"dataTables_scrollBody"})[0]
        time.sleep(6)
        for tr in content2.table.tbody.find_all('tr'):
            num = tr.find("td",{"class":"sorting_1"}).getText()
            num = str(num).strip()
            num = int(num)
            sum += num
            # print(num)
            if num == 0:
                break
        print(sum)
    wr_f.writerow([li,aa,bb,temp_2015,temp_2016,temp_2017,temp_2018,temp_2019,temp_2020,app,grant,sum])
f.close()
driver.close()

（四）学习到的点

1、当遇到input or button or不论什么元素的ID老是变动的时候，可以用xpath中的contain方式进行定位！亲测有效、好用！
driver.find_element_by_xpath('//textarea[contains(@placeholder,"Fanuc")]').click()
2、用BeautifulSoup解析页面时，要完成所有的点击操作，使得“style:display=none”的元素变得可见，才能解析出完整的内容！