第一种:
根据规律手工构造各个期刊的url
爬取两遍,第一遍爬完,去重(集合),并初始化结果集;第二遍爬取,在结果集里查询并计数
将结果集转化为列表,并按照列表中times关键字对字典(相当于数据集合)整体排序
写入结果到txt文件
from bs4 import BeautifulSoup as Be
import requests as req
import os
BaseUrl = "http://www.52duzhe.com/"
def Do_soup(url):
try:
r = req.get(url,headers={'user-agent':'Mozilla/5.0'})
r.raise_for_status()
r.encoding = r.apparent_encoding
html = r.text
soup = Be(html,'lxml')
return soup
except:
print("获取"+url+"失败")
def Search_each_Per(tag):
aut_set = set()
a_list = []
for i in range(2010,2018):
for j in range(1,25):
if j<10:
ExtraUrl = str(i)+'_0'+str(j)
else:
ExtraUrl = str(i)+'_'+str(j)
if i in [2010,2011,2012]:
if(i == 2012 and j>=14):
url = BaseUrl + ExtraUrl + r"/index.html"
else:
url = BaseUrl + '1_' + ExtraUrl + ".html"
else:
url = BaseUrl + ExtraUrl + r"/index.html"
soup = Do_soup(url) #使用函数
if soup == None:
continue
per_aut_list = soup.find_all('td',class_="author")
if tag==1:
for k in per_aut_list:
aut_set.add(k.string)
print("{}年{}期作者已入库".format(i,j))
else:
for k in per_aut_list:
a_list.append(k.string)
if tag=