python小白学爬虫
断断续续学习python差不多也有2个月了,摸了不少坑磕磕绊绊的完成了这个爬虫,对我来说还算蛮有意义的,留档纪念下。
爬虫背景
目的:爬取网页前10页的MM图片
网页:http://jandan.net/ooxx(煎蛋网,正经网站)
分析过程:
1、查找当前页面多个图片的URL
2、查找下一页URL
3、循环打开10个页面分别保存每个页面的MM图片
代码如下:
import requests
from bs4 import BeautifulSoup
def img_urls(url):
#查找页面所有图片URL
header = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36'}
res = requests.get(url,headers=header).text
soup = BeautifulSoup(res,'html.parser')
a=soup.find_all('img')
img_urls=['http:'+a[x]['src'] for x in range(len(a))]
return img_urls
def next_url(url):
#查找下一页url
header = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36'}
res = requests.get(url,headers=header).text
soup = BeautifulSoup(res,'html.parser')
page_tag = soup.find('span',class_='current-comment-page')
next_url = 'http:'+page_tag.nextSibling.nextSibling['href']
return next_url
def save_img(url):
#保存图片
header = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36'}
res = requests.get(url,headers=header).content
img_name = url.split('/')[-1]
with open(img_name,'wb') as f:
f.write(res)
def main():
url = 'http://jandan.net/ooxx'
number = 10
#imgs = []
while number:
number-=1
imgs=img_urls(url)
for img_url in imgs:
save_img(img_url)
url = next_url(url)
if __name__=='__main__':
main()
目前水平就这样了,后期慢慢优化