前一段时间想看CVPR的论文,想上网找CVPR的论文集,找了半天找不到,然后在微信搜一搜上发现了如下代码,可以一键爬取所有论文,甚喜。原文链接找不到了。只记得文章最开头硕大几个字授人以鱼不如授人以渔。
在下面python代码的同级目录下新建CVPR2023文件夹,然后运行文件即可。
如果需要下载其他年份的论文,只要将
r = requests.get('https://openaccess.thecvf.com/CVPR2023?day=all')
中的2023改为对应年份即可。
import re
import requests
import urllib.request
import os
import socket
# get web context
r = requests.get('https://openaccess.thecvf.com/CVPR2023?day=all')
data = r.text
print(data)
# find all pdf links
link_list = re.findall(r"(?<=href=\").+?pdf(?=\">pdf)", data)
name_list = re.findall(r"(?<=paper.html\">).+(?=</a>)", data)
socket.setdefaulttimeout(30)
cnt = 0
num = len(link_list)
# your local path to download pdf files
localDir = './CVPR2023/'
if not os.path.exists(localDir):
os.makedirs(localDir)
while cnt < num:
url = link_list[cnt]
# seperate file name from url links
file_name = name_list[cnt]
# to avoid some illegal punctuation in file name
file_name = file_name.replace(':', '_')
file_name = file_name.replace('\"', '_')
file_name = file_name.replace('?', '_')
file_name = file_name.replace('/', '_')
file_name = file_name.replace(' ', '_')
file_path = localDir + file_name + '.pdf'
if os.path.exists(file_path):
print('【{}.pdf】 exists,skip downloading.'.format(file_name))
cnt = cnt + 1
continue
else:
# download pdf files
print('[' + str(cnt) + '/' + str(num) + "] Downloading -> " + file_path)
try:
urllib.request.urlretrieve('http://openaccess.thecvf.com/' + url, file_path)
except:
cnt = cnt + 1
continue
cnt = cnt + 1
print("all download finished")