爬虫----代码段总结

图片下载

response=requests.get(url,headers=headers)
if response.status_code==200:
    with open('123.jpg','wb') as f:
        f.write(response.content)
    print('ok')

---

图片下载 —scrapy

pipelines.py:

from scrapy.pipelines.images import ImagesPipeline
from scrapy.utils.project import get_project_settings
import os
#保存图片
class ImagePipeline(ImagesPipeline):
    def get_media_requests(self, item, info):
        image_path = item["nick_image"]
        yield scrapy.Request(image_path)

    #重命名
    def item_completed(self, results, item, info):
    iamge_path = [x["path"] for ok, x in results if ok]
    old_image_name = self.IMAGES_STORE+"/"+iamge_path[0]
    new_image_name = self.IMAGES_STORE+"/"+item['nick_name']+".jpg"
    os.rename(old_image_name,new_image_name)
    item["image_path"] = new_image_name

    return item

在setting中创建image目录: IMAGES_STORE = ‘./Image’

---

scrapy爬虫parse()回调实现下一页

    if self.offset < int(position_numbers):
        self.offset += 10
    new_url = self.url + str(self.offset) + "#a"
    yield scrapy.Request(new_url,callback=self.parse)