scrapy 爬虫学习
小白头发少
这个作者很懒,什么都没留下…
展开
-
scrapy-pipline
pipline Image Pipline(爬取图片) # settings.py IMAGES_STORE = './images' # piplines.py from scrapy import Request from scrapy.exceptions import DropItem from scrapy.piplines.images import ImagesPipline class ImagePipline(ImagesPipline): # 接收spider生成的item,取出原创 2021-09-12 23:21:32 · 83 阅读 · 0 评论 -
scrapy-settings.py
settings.py settings.py BOT_NAME ='TouPIc' # 项目名 SPIDER_MOULES = ['TouPic.spiders'] # 爬虫的位置 NEWSPIDER_MODULE= 'TouPic.spiders' #新建一个爬虫会在的位置 USER_AGENT= '' # 浏览器的表示 ROBOTSTST_OBEY= False # 君子协议 CONCURRENT_REQUESTS = 32 #并发请求 DOWNLOAD_DELAY = 3 #下载延迟 # DO原创 2021-09-12 22:27:52 · 91 阅读 · 0 评论 -
scrapy 爬虫——百度贴吧
scrapy 爬虫——百度贴吧 爬取了百度贴吧凡人修仙传的吧友留言。 settings .py SPIDER_MODULLES = ['stone.spiders'] NEWSPIDER_MODULE = 'stone.spiders' LOG_LEVEL = "WARNING" #去除警告信息 USER_AGENT = "USER_AGENT = 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0;" ROBOTSTXT_原创 2021-07-14 11:09:56 · 267 阅读 · 0 评论