构建项目:
scrapy startproject taobao
pycharm打开项目。
在项目根路径创建一个爬虫:
scrapy genspider 爬虫名称 要爬取的限制域
调试工具:
scrapy shell http://www.taobao.com
#选择<title>标签(也可以class),::取标签值,extract提取数据,first指第一个,等价于extract[0]
response.css('title::text').extract_first()
启动项目:
scrapy crawl quotes
pycharm断点调试:
在根路径下新建文件main.py,内容如下:修改下爬虫名称即可
from scrapy.cmdline import execute
import os
import sys
if __name__ == '__main__':
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
execute(['scrapy', 'crawl', '爬虫name'])
爬虫如下:
爬虫文件:
import scrapy
from ..items import MyscrapyItem
# 定义爬取逻辑
class QuotesSpider(scrapy.Spider):
# 爬虫名称
name = 'quotes'
# 允许的域
allowed_domains = ['lab.scrapyd.cn']
# 初始url集合
start_urls = ['http://lab.scrapyd.cn/page/1/']
# 处理response的方法
def parse(self, response):
# 用css选择器、获取所有的quote节点
quotes = response.css('div.quote')
# 遍历节点集
for quote in quotes:
item = MyscrapyItem()
item["text"] = quote.css('span.text::text').extract_first()
item["author"] = quote.css('span small.author::text').extract_first()
item["tags"] = quote.css('div.tags a.tag::text').extract()
yield item
# 如果存在下一页则继续递归爬取
next_page = response.css('.next a::attr(href)').extract_first()
if next_page is not None:
yield scrapy.Request(next_page, callback=self.parse)
item文件:
# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html
import scrapy
class MyscrapyItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
# 定义爬取的数据字段
text = scrapy.Field()
author = scrapy.Field()
tags = scrapy.Field()
pass
pipelines:
# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html
# useful for handling different item types with a single interface
from itemadapter import ItemAdapter
class MyscrapyPipeline:
def process_item(self, item, spider):
# 处理爬取的结果
print(item)
要激活pipelines还需要修改一下settings.py
将这一段注释释放掉
ITEM_PIPELINES = {
'myscrapy.pipelines.MyscrapyPipeline': 300,
}
注意xpath的使用:
def parse(self, response, **kwargs):
# 使用绝对路径定位标签
elements = response.xpath('//div[@class="ui relaxed divided items explore-repo__list"]//div[@class="item"]')
for element in elements:
# 注意:再次进行xpath的时候是相对路径在需要//前面加上.。是.//而不是//
link = self.allow_domains + element.xpath('.//h3/a/@href').get()
print(link)
pass