上次我通过两个实战教学展示了如何使用urllib模块(http://blog.csdn.net/mr_blued/article/details/79180017)来构造爬虫,这次告诉大家一个更好的实现爬虫的模块,requests模块。
使用requests模块进行爬虫构造时最好先去了解一下HTTP协议与常见的几种网页请求方式。
闲话少说,我们进入正题。
使用requests模块改进上次的例子中的代码
1.爬取妹子图。(目标网址:http://www.meizitu.com/)
import requests
import os
import re
import time
def url_open(url):
# 以字典的形式添加请求头
header = {
'User-Agent': "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.221 Safari/537.36 SE 2.X MetaSr 1.0"
}
# 使用get方法发送请求获取网页源码
response = requests.get(url, headers=header)
return response
def find_imgs(url):
html = url_open(url).text
p = r'<img src="([^"]+\.jpg)"'
img_addrs = re.findall(p, html)
return img_addrs
def download_mm(folder='OOXX'):
os.mkdir(folder)
os.chdir(folder)
page_num = 1 # 设置为从第一页开始爬取,可以自己改
x = 0 # 自命名图片
img_addrs = [] # 防止图片重复
# 只爬取前两页的图片,可改,同时给图片重命名
while page_num <= 2:
page_url = url + 'a/more_' + str(page_num) + '.html'
addrs = find_imgs(page_url)
print(len(addrs))
# img_addrs = []
for i in addrs:
if i in img_addrs:
continue
else:
img_addrs.append(i)
print(len(img_addrs))
for each in img_addrs:
print(each)
page_num += 1
# x = (len(img_addrs)+1)*(page_num-1)
for each in img_addrs:
filename = str(x) + '.' + each.split('.')[-1]
x += 1
with open(filename, 'wb') as f:
img = url_open(each).content
f.write(img)
# page_num += 1
if __name__ == '__main__':
url = 'http://www.meizitu.com/'
download_mm()
2.爬取百度贴吧图片 (目标网址:https://tieba.baidu.com/p/5085123197)
import requests
import re
import os
def open_url(url):
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.221 Safari/537.36 SE 2.X MetaSr 1.0"}
response = requests.get(url, headers=headers)
return response
def find_img(url):
html = open_url(url).text
p = r'<img class="BDE_Image" src="([^"]+\.jpg)"'
img_addrs = re.findall(p, html)
for each in img_addrs:
print(each)
for each in img_addrs:
file = each.split("/")[-1]
with open(file, "wb") as f:
img = open_url(each).content
f.write(img)
def get_img():
os.mkdir("TieBaTu")
os.chdir("TieBaTu")
find_img(url)
if __name__ == "__main__":
url = 'https://tieba.baidu.com/p/5085123197'
get_img()
总结:
1.熟悉requests模块的方法,以及了解http协议和几种常见的请求方式
2.了解网站的反爬虫策略,并建立相对应的反反爬虫手段
3.知道其他模块的作用。
爬虫项目地址:github